Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt

From: Bob Briscoe <[email-masked]>
Date: Tue, 3 Nov 2020 00:05:56 +0000

Michael,

1/ I just posted a draft-13 that has the two AccECN TCP Option orders.

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13

As agreed, it keys the two option orders on two different Option Kinds, 
which I've requested in the IANA section (TBD0 and TBD1). This replaces 
using the initial values of the fields. I've reverted all the 
complicated text about:
* varying the initial field values dependent on option order,
* having to keep consistent order throughout a connection,
* and choosing which order from each end.
All of which is irrelevant now.

It is much simpler (I guess we all knew that). Thank you.

As mentioned, I found my other ToDo list:
2/ We also introduced considerably more guidance on middlebox handling, 
esp. ACK filtering and Segmentation Offload. There was enough extra text 
to be worth breaking that section into subsections.

The guidance on ACK filtering for connections with AccECN negotiated 
updates the (non-)guidance in RFC3449 (BCP69 which said ACK filtering in 
the presence of ECN was a research issue). But there is still no 
guidance for ECN connections when AccECN has not been negotiated.

3/ I also added a SHOULD requirement I had on my ToDo list: Each Data 
Sender SHOULD disable ECT in one direction if all the feedback for three 
or four rounds from the start is CE. That's already done in Linux, and 
in the TCP in iOS and MacOS at least.

Bob

On 28/10/2020 17:17, Bob Briscoe wrote:
> Michael,
>
> On 28/10/2020 15:55, Scharf, Michael wrote:
>> Hi Bob,
>>
>> What is the future plan regarding the option encoding proposed by the 
>> authors?
>>
>> This has been discussed in past meetings and on the list, for 
>> instance: 
>> https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>>
>> I have explained several times my pushback against the encoding 
>> currently described in the document, most notably the requirement to 
>> keep state in the endpoints to decode the format of a TCP option. I 
>> have not changed my mind.
>>
>> I have also mentioned that, for instance, two option codepoints would 
>> IMHO be a better engineering approach ("KISS principle"). That would 
>> be a very simple change to the content to the document. Needless to 
>> say that there are other alternatives for marshalling parameters in 
>> an unambigous way.
>
> [BB] Whoops. In my head I had already shifted to your scheme with 
> two-option kinds. But I see that, in the draft,  I haven't. Whoops - 
> that was a bug in my ToDo list processing. Sorry about that. I'll 
> submit another draft before the deadline with that scheme replacing 
> the current one.
>
> I've just found some other entries in my ToDo list for this draft 
> which I missed as well. I'll include them too.
>
> Sorry again.
>
>
> Bob
>
>>
>> There have been other related comments, e.g. from Ilpo.
>>
>> Best regards
>>
>> Michael (with no hat)
>>
>>> -----Original Message-----
>>> From: tcpm <[email-masked]> On Behalf Of Bob Briscoe
>>> Sent: Wednesday, October 28, 2020 2:04 PM
>>> To: [email-masked]
>>> Cc: Scheffenegger, Richard <[email-masked]>; Mirja
>>> Kuehlewind <[email-masked]>
>>> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-12.txt
>>>
>>> TCPM folks,
>>>
>>> Back in May'20, it was decided to hold back the AccECN (and the ECN++)
>>> drafts for a while until the position on L4S in tsvwg was clearer. 
>>> There
>>> is no formal dependency on the L4S drafts in tsvwg, but it made sense
>>> not to burn codepoints without more certainty.
>>>
>>> I've just refreshed the AccECN and the ECN++ drafts, which had expired.
>>>
>>> I included the following minor changes to AccECN, which you can see via
>>> the diff link below, but in summary:
>>> * moved wrongly categorized "action if unwilling to provide f/b" to the
>>> group of bullets relevant to the receiver
>>> * removed refs to long-expired ecn-fallback draft
>>> * fixed table 1 after xml2rfc changed formatting
>>>
>>> Thanks to those who pointed these out.
>>>
>>> No changes to ECN++ other than update of some refs.
>>>
>>> Cheers
>>>
>>>
>>> Bob
>>>
>>> On 28/10/2020 12:47, [email-masked] wrote:
>>>> A New Internet-Draft is available from the on-line Internet-Drafts 
>>>> directories.
>>>> This draft is a work item of the TCP Maintenance and Minor 
>>>> Extensions WG of
>>> the IETF.
>>>>           Title           : More Accurate ECN Feedback in TCP
>>>>           Authors         : Bob Briscoe
>>>>                             Mirja Kühlewind
>>>>                             Richard Scheffenegger
>>>>     Filename        : draft-ietf-tcpm-accurate-ecn-12.txt
>>>>     Pages           : 58
>>>>     Date            : 2020-10-28
>>>>
>>>> Abstract:
>>>>      Explicit Congestion Notification (ECN) is a mechanism where 
>>>> network
>>>>      nodes can mark IP packets instead of dropping them to indicate
>>>>      incipient congestion to the end-points.  Receivers with an ECN-
>>>>      capable transport protocol feed back this information to the 
>>>> sender.
>>>>      ECN is specified for TCP in such a way that only one feedback 
>>>> signal
>>>>      can be transmitted per Round-Trip Time (RTT).  Recent new TCP
>>>>      mechanisms like Congestion Exposure (ConEx), Data Center TCP 
>>>> (DCTCP)
>>>>      or Low Latency Low Loss Scalable Throughput (L4S) need more 
>>>> accurate
>>>>      ECN feedback information whenever more than one marking is 
>>>> received
>>>>      in one RTT.  This document specifies a scheme to provide more 
>>>> than
>>>>      one feedback signal per RTT in the TCP header.  Given TCP header
>>>>      space is scarce, it allocates a reserved header bit, that was
>>>>      previously used for the ECN-Nonce which has now been declared
>>>>      historic.  It also overloads the two existing ECN flags in the 
>>>> TCP
>>>>      header.  The resulting extra space is exploited to feed back 
>>>> the IP-
>>>>      ECN field received during the 3-way handshake as well. 
>>>> Supplementary
>>>>      feedback information can optionally be provided in a new TCP 
>>>> option,
>>>>      which is never used on the TCP SYN.
>>>>
>>>>
>>>> The IETF datatracker status page for this draft is:
>>>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/
>>>>
>>>> There are also htmlized versions available at:
>>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12
>>>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-12
>>>>
>>>> A diff from the previous version is available at:
>>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-12
>>>>
>>>>
>>>> Please note that it may take a couple of minutes from the time of 
>>>> submission
>>>> until the htmlized version and diff are available at tools.ietf.org.
>>>>
>>>> Internet-Drafts are also available by anonymous FTP at:
>>>> ftp://ftp.ietf.org/internet-drafts/
>>>>
>>>>
>>>> _______________________________________________
>>>> tcpm mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/tcpm
>>> -- 
>>> ________________________________________________________________
>>> Bob Briscoe http://bobbriscoe.net/
>>>                  PRIVILEGED AND CONFIDENTIAL
>>>
>>> _______________________________________________
>>> tcpm mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/tcpm
>> _______________________________________________
>> tcpm mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/tcpm
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Scharf, Michael" <[email-masked]>
Date: Tue, 3 Nov 2020 10:04:59 +0000

> It is much simpler (I guess we all knew that). Thank you.

Yep, and simplicity is IMHO a big plus. This approach is IMHO much better than the previous proposal of keeping state. I'd like to repeat that I would also be fine with other alternative solutions, as long as they are not overly complex.

Anyway, this new text works for me. 

Michael

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 6 Nov 2020 00:53:57 -0800

Hi Bob,

Sorry for the late comments, I have a naive question about the encoding
schemes in the options.
The byte counters will be incremented by every byte the receiver receives.
But, do we really need this level of granularity?
I am thinking if we can byte-shift the counter values to some extent, we
might be able to save option space.

Thanks,
--
Yoshi

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 6 Nov 2020 09:30:01 +0000

Hi Yoshi,

not sure if that really saves that much as you don’t need to send the option with every packet, as such you can manage the overhead dynamically as you can decide how often to send. Also in a scenario where most of the traffic flows only in on direction the AccECN is sent in the other direction.

Mirja

From: tcpm <[email-masked]> on behalf of Yoshifumi Nishida <[email-masked]>
Date: Friday, 6. November 2020 at 09:54
To: Bob Briscoe <[email-masked]>
Cc: "Scheffenegger, Richard" <[email-masked]>, "[email-masked]" <[email-masked]>, Mirja Kuehlewind <[email-masked]>
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt

Hi Bob,

Sorry for the late comments, I have a naive question about the encoding schemes in the options.
The byte counters will be incremented by every byte the receiver receives. But, do we really need this level of granularity?
I am thinking if we can byte-shift the counter values to some extent, we might be able to save option space.

Thanks,
--
Yoshi

On Mon, Nov 2, 2020 at 4:09 PM Bob Briscoe <[email-masked]<mailto:[email-masked]>> wrote:
Michael,

Bob

On 28/10/2020 17:17, Bob Briscoe wrote:
>>> From: tcpm <[email-masked]<mailto:[email-masked]>> On Behalf Of Bob Briscoe
>>> To: [email-masked]<mailto:[email-masked]>
>>> Cc: Scheffenegger, Richard <[email-masked]<mailto:[email-masked]>>; Mirja
>>> Kuehlewind <[email-masked]<mailto:[email-masked]>>
>>> On 28/10/2020 12:47, [email-masked]<mailto:[email-masked]> wrote:
>>>> until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.
>>>>
>>>> [email-masked]<mailto:[email-masked]>
>>> Bob Briscoe http://bobbriscoe.net/<https://protect2.fireeye.com/v1/url?k=94a9aab2-cb32939f-94a9ea29-8692dc8284cb-827b8e584d366b78&q=1&e=b9141f6c-0f70-4414-9212-82cae72b93fc&u=http%3A%2F%2Fbobbriscoe.net%2F>
>>> [email-masked]<mailto:[email-masked]>
>> [email-masked]<mailto:[email-masked]>

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/<https://protect2.fireeye.com/v1/url?k=1641fe24-49dac709-1641bebf-8692dc8284cb-0283475ef4fa8e27&q=1&e=b9141f6c-0f70-4414-9212-82cae72b93fc&u=http%3A%2F%2Fbobbriscoe.net%2F>

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 6 Nov 2020 02:28:06 -0800

Hi Mirja,

It seems to me that we don't have much freedom for the frequency of the
feedback.
The draft says:
      MUST immediately send an ACK once 'n' CE marks have arrived since

      the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
      than 6.

Or, options are not applied to this?

Also, even though traffic flows one direction, I think you might want
to send back SACK options, etc.

Another thought is bulk flow traffic tends to send full MSS packets. I
feel 1 byte granularity might not be very efficient.

In any cases, I just thought saving option space won't be a bad thing
if it's not overly difficult.

Thanks,

--

Yoshi

On Fri, Nov 6, 2020 at 1:30 AM Mirja Kuehlewind <
[email-masked]> wrote:

> Hi Yoshi,
>
>
>
> not sure if that really saves that much as you don’t need to send the
> option with every packet, as such you can manage the overhead dynamically
> as you can decide how often to send. Also in a scenario where most of the
> traffic flows only in on direction the AccECN is sent in the other
> direction.
>
>
>
> Mirja
>
>
>
>
>
>
>
> *From: *tcpm <[email-masked]> on behalf of Yoshifumi Nishida <
> [email-masked]>
> *Date: *Friday, 6. November 2020 at 09:54
> *To: *Bob Briscoe <[email-masked]>
> *Cc: *"Scheffenegger, Richard" <[email-masked]>, "
> [email-masked]" <[email-masked]>, Mirja Kuehlewind <[email-masked]>
> *Subject: *Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt
>
>
>
> Hi Bob,
>
>
>
> <https://protect2.fireeye.com/v1/url?k=94a9aab2-cb32939f-94a9ea29-8692dc8284cb-827b8e584d366b78&q=1&e=b9141f6c-0f70-4414-9212-82cae72b93fc&u=http%3A%2F%2Fbobbriscoe.net%2F>
> <https://protect2.fireeye.com/v1/url?k=1641fe24-49dac709-1641bebf-8692dc8284cb-0283475ef4fa8e27&q=1&e=b9141f6c-0f70-4414-9212-82cae72b93fc&u=http%3A%2F%2Fbobbriscoe.net%2F>
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Nov 2020 13:01:53 +0000

Yoshi,

On 06/11/2020 10:28, Yoshifumi Nishida wrote:
> Hi Mirja,
>
> It seems to me that we don't have much freedom for the frequency of 
> the feedback.
> The draft says:
> the previous ACK, where 'n' SHOULD be 2 and MUST be no greater than 6.

[BB]
Section 3.2.2 about the ACE field, which is in the main TCP header feeds 
back the count of CE-marked *packets* .
Section 3.2.3 is about the AccECN TCP Option, which is optional and 
feeds back a count of *bytes* for each marking.

If there are any places in the draft where this isn't clear, we can fix 
them. But, in this case, the  text right under your quote says:

    These rules for when to send an ACK are designed to be complemented
    by those inSection 3.2.3.3  <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13#section-3.2.3.3>, which concern whether the AccECN TCP
    Option ought to be included on ACKs.

> to send back SACK options, etc.

Last para of the Introduction:

    It is recommended that the AccECN protocol is implemented alongside
    SACK [RFC2018  <https://tools.ietf.org/html/rfc2018>] and ...

Also, interactions with SACK are raised where necessary throughout the 
draft. In particular this one:

       If the smallest allowed AccECN Option would leave insufficient
       space for two SACK blocks on a particular ACK, the Data Receiver
       MUST give precedence to the SACK option (total 18 octets), because
       loss feedback is more critical.

> if it's not overly difficult.

[BB] If efficiency is needed, field(s) (or the whole option) can be omitted.
During the history of this draft we have proposed different ways to 
encode information more efficiently, but WG discussion decided that 
simple omission or inclusion of fields gives optional efficiency with 
minimal complexity.

I'm not saying the act of reducing the granularity of each field would 
be particularly complex.
Rather, complexity comes from the resulting ambiguity. Here's an example:
* The Linux code currently uses the byte counters to help disambiguate 
wrap of the 3-bit packet counter.
* If for instance the byte counters were truncated by 8 bits for 
efficiency, it would add many more ambiguous cases for this code to have 
to take into account.
* Such complexity introduces misunderstanding and bugs.

The design has gone round and round many times. We have tried to 
converge on the best tradeoffs.

Best Regards

Bob

> Thanks,
> --
> Yoshi
>
> On Fri, Nov 6, 2020 at 1:30 AM Mirja Kuehlewind 
> <[email-masked] <mailto:[email-masked]>> 
> wrote:
>
>     Hi Yoshi,
>
>     not sure if that really saves that much as you don’t need to send
>     the option with every packet, as such you can manage the overhead
>     dynamically as you can decide how often to send. Also in a
>     scenario where most of the traffic flows only in on direction the
>     AccECN is sent in the other direction.
>
>     Mirja
>
>     *From: *tcpm <[email-masked]
>     <mailto:[email-masked]>> on behalf of Yoshifumi Nishida
>     <[email-masked] <mailto:[email-masked]>>
>     *To: *Bob Briscoe <[email-masked] <mailto:[email-masked]>>
>     *Cc: *"Scheffenegger, Richard" <[email-masked]
>     <mailto:[email-masked]>>, "[email-masked]
>     <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>,
>     Mirja Kuehlewind <[email-masked] <mailto:[email-masked]>>
>     The byte counters will be incremented by every byte the receiver
>     receives. But, do we really need this level of granularity?
>
>     I am thinking if we can byte-shift the counter values to some
>     extent, we might be able to save option space.
>
>     Thanks,
>
>     --
>
>     Yoshi
>
>         >>> From: tcpm <[email-masked]
>         <mailto:[email-masked]>> On Behalf Of Bob Briscoe
>         >>> To: [email-masked] <mailto:[email-masked]>
>         >>> Cc: Scheffenegger, Richard
>         <[email-masked]
>         <mailto:[email-masked]>>; Mirja
>         >>> Kuehlewind <[email-masked] <mailto:[email-masked]>>
>         >>>> until the htmlized version and diff are available at
>         tools.ietf.org <http://tools.ietf.org>.
>         >>>>
>         >>>> [email-masked] <mailto:[email-masked]>
>         >>> [email-masked] <mailto:[email-masked]>
>         >> [email-masked] <mailto:[email-masked]>
>         [email-masked] <mailto:[email-masked]>

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 6 Nov 2020 14:01:07 +0000

I see this rev introduced a section on ACK Filtering, which I have 
specific comments on. We need to consider the cost of sending ACKs - 
especially anything that results in them being large - however, this 
needs to be done by choosing a default at the receiver - not as an 
adaptation, the adaption if necessary needs to be introduce more ACKs 
when this is beneficial, as in some cases for DAAS, not in trying to 
figure something from delayed or "missing" ACKs.

1) So let me say first that I think the suggestion for updating the 
receiver to thin, and every 2-6 packets seems reasonable.

     “MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
       than 6.”

I do agree that endpoints can do much more to reduce the return path 
capacity that they consume - and that the endpoints have the opportunity 
to encode the feedback information more efficiently than can be done by 
a device (or queue) in the network. From our experiments, n=10 would 
also often be likely OK.

2. Althouigh in some cases a receiver can predict a good "n", I do not 
believe a receiver can typically find out how to choose “n” by observing 
the ACKs on a connection.

Sure, if the path is modelled as a simple bit-congestive path that has a 
queue, then you could monitoring timing/loss of packets and determine a 
rate that tries not to fill a queue. I would not know how common such a 
bottleneck link is.

I do know that many bottleneck links are much more complex and need to 
consider the scheduling transmission of small ACK packets on the return 
path performance. In many radio technologies, the overhead from 
transmission of ACKs is proportionally much higher than for data 
segments. These layer 2s are often constrained in terms of packets (e.g. 
where lower layer framing and transmission scheduling is an important 
consideration to the efficiency/cost of using the service). This has 
often been “fixed” using ACK Thinning or PEPs. Protocols that send fewer 
ACKs/data packet are not impacted.

2. I do not believe a receiver can find out how top to choose “n” by 
observing the ACKs received on a connection.

Sure, if the path is modelled as a simple bit-congestive path that has a 
queue, then you could monitoring timing/loss of packets and determine a 
rate that tries not to fill a queue. I would not know how common such a 
bottleneck link is.

I do know that many bottleneck links are more complex and needs to 
consider the scheduling transmission of small ACK packets on the return 
path performance. In many radio technologies, the overhead from 
transmission of ACKs is proportionally much higher than for data 
segments -  These layer 2s are constrained in terms of packets (e.g. 
where lower layer framing and transmission scheduling is an important 
consideration to the efficiency/cost of using the service). This has 
often been “fixed” using ACK Thinning or PEPs. Protocols that send fewer 
ACKs/data packet are not impacted.

Scheduling (at the IP layer,can also often result in variations in 
timing - something that is hard to predict and I would suggests make 
automated selection of “n” very hard, and the need to thin is often not 
directly visible to a user of the path. Traffic multiplexed with 
protocols that use different ACK policies (perhaps QUIC?) also make this 
sort of detection less reliable.

3) A queue might be competing for scheduling opportunities against other 
IP flows (e.g., WFQ), or competing for transmission opportunities with a 
shared resource (common in radio links).

ACKs can and do take transmission opportunities away from the resource 
pools that would be used by other services that share the same resource 
pool. This can be reduced by some form of per-flow scheduling with 
deeper per-flow queues - but that in itself can induce unwanted coupling 
between flows - limiting the peak rate of one flow, to achieve a higher 
ACK ratio for another flow.

Consider a QUIC flow sharing a return path with a TCP flow, the higher 
rate of larger QUIC ACKs reduces the transmission opportunities for TCP 
ACKs, which in turn suppresses the TCP flow, limiting the overall system 
performance. If the ACKs use the same transmission resource, they could 
take directly away from transmission opportunities in the forward 
direction, but even if they do not - the rate of ACKs retuned impacts 
the sending rate end to end.

4) The new section adds some cross-layer requirements on sub-IP layers, 
which seem to imply more complexity. I’m not comfortable with such a 
proposal.

It seems that the text suggests not implement thinning and that these 
systems implement logic to detect the presence of the option by 
observing SYNs and then configure the link to do something special. I 
don’t think think this is the correct approach - the result will be 
throttling of the connection speed. Is it necessary that the sub-IP 
layer does not implement thinning for connections that use AccECN feedback?

When the text says: “SHOULD preserve the timing of each ACK”
- does that imply that nodes need to discard rather than delay packets 
when link scheduling or transmission scheduling is used. This seems a 
very odd requirement, it might be better too simply strip-out the AccECN 
option or discard all packets that set this.

I can’t see how queuing the ACKs, or deleting ACKs rather than queuing 
them would improve the performance of AccECN TCP connections.

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Nov 2020 21:51:04 +0000

Gorry,

Thanks as always for checking all this... inline tagged [BB]

On 06/11/2020 14:01, Gorry Fairhurst wrote:
> capacity that they consume - and that the endpoints have the 
> opportunity to encode the feedback information more efficiently than 
> can be done by a device (or queue) in the network. From our 
> experiments, n=10 would also often be likely OK.

NB: n is the no. of *CE marks*, which is only the ACK thinning ratio in 
the worst case if there's 100% marking.
IOW, with n=6 the receiver could ACK less than 1 in 10 data packets 
whenever marking probability < 60%.

Altho the receiver's ACK ratio will have to fall to 6 to 1 for the 
duration of a 100% marking sequence, whenever some data packets are not 
marked, the receiver aACK ratio can rise. This *could* allow any 
build-up of ACKs queued on the reverse path to relax, which could 
improve the chances of not overflowing into ACK loss.

It's hard to predict when 100% marking in the forward direction will 
coincide with reverse path congestion, and I'm not saying it won't.
Just pointing out that n does not equal the ACK thinning ratio, it's the 
floor of it.

n=6 is a consequence of only having a 3-bit field to count CE marks. We 
could push the envelope and make the max n=7. But not n=8, otherwise 
wrap after 8 marks would be indistinguishable from zero marks.

> believe a receiver can typically find out how to choose “n” by 
> observing the ACKs on a connection.

[BB] I think you intended to delete one of these sentences. Whatever, 
both of them seem to contain a typo - otherwise they are circular? 
Surely the receiver is sending the ACKs, so either sentence as it stands 
ends with "...I do not believe a receiver can typically decide how often 
to send an ACK by observing how often it is sending ACKs itself." ???

>
> Sure, if the path is modelled as a simple bit-congestive path that has 
> a queue, then you could monitoring timing/loss of packets and 
> determine a rate that tries not to fill a queue. I would not know how 
> common such a bottleneck link is.
>
> consider the scheduling transmission of small ACK packets on the 
> return path performance. In many radio technologies, the overhead from 
> often been “fixed” using ACK Thinning or PEPs. Protocols that send 
> fewer ACKs/data packet are not impacted.

[BB] I agree that the receiver alone cannot predict a good 'n'.
AccECN is a not the complete solution, but we've grasped the opportunity 
to take a step in the right direction (within the chartered scope of 
AccECN). AccECN at least allows the data sender to measure the extent of 
congestion marking on the arriving ACK stream (if there's ECN support at 
the bottleneck). Whereas before AccECN, a random ECN mark on one pure 
ACK in a round couldn't be distinguished from 100% of ACKs marked.

> automated selection of “n” very hard, and the need to thin is often 
> not directly visible to a user of the path. Traffic multiplexed with 
> protocols that use different ACK policies (perhaps QUIC?) also make 
> this sort of detection less reliable.

[BB] Being hard shouldn't prevent us trying.
The remaining piece would be to advance a TCP ACK congestion control 
protocol on the experimental track (RFC5690 
<https://tools.ietf.org/html/rfc5690> is informational).

>
> 3) A queue might be competing for scheduling opportunities against 
> other IP flows (e.g., WFQ), or competing for transmission 
> opportunities with a shared resource (common in radio links).
>
> pools that would be used by other services that share the same 
> resource pool. This can be reduced by some form of per-flow scheduling 
> with deeper per-flow queues - but that in itself can induce unwanted 
> coupling between flows - limiting the peak rate of one flow, to 
> achieve a higher ACK ratio for another flow.
>
> rate of larger QUIC ACKs reduces the transmission opportunities for 
> TCP ACKs, which in turn suppresses the TCP flow, limiting the overall 
> system performance. If the ACKs use the same transmission resource, 
> they could take directly away from transmission opportunities in the 
> forward direction, but even if they do not - the rate of ACKs retuned 
> impacts the sending rate end to end.

[BB] This says two things to me:
* if TCP is going to introduce control over the ACK ratio, QUIC and 
other transports ought to as well {Note 1}.
* and all ACK CC algorithms ought to be designed to ensure some degree 
of equitable capacity sharing {Note 2}, including between the ACK stream 
of flow A and the data stream of flow B transferring in the opposite 
direction.

{Note 1}: I don't think this needs any formal "all or nothing" rule 
where the transport ADs co-ordinate multiple WGs. Enough of the people 
involved are the same for the right thing to happen (I think). But some 
AD oversight wouldn't do any harm.
{Note 2}: There are numerous challenges to face, but I think it's 
paramount to first establish which node is considered responsible for 
solving the problem (the bottleneck or one of the transport endpoints). 
This doesn't imply that the IETF will ever ensure that other nodes don't 
'interfere'. For instance, if the sender is deemed responsible for 
determining the level of ACK filtering, it will still have to make 
allowance for bottlenecks attempting to ACK thin as well.

>
> 4) The new section adds some cross-layer requirements on sub-IP 
> layers, which seem to imply more complexity. I’m not comfortable with 
> such a proposal.
>
> observing SYNs and then configure the link to do something special. 

This was the proposal that I was most interested to hear your views on. 
We can remove it if the WG doesn't want it, but first I want to make 
sure the proposal is fully understood:
* The aim is for the RFC to tell bottleneck implementers (via this text) 
that *only in cases where AccECN has been negotiated* the preferred way 
to improve flow performance is for the link to signal ECN to the 
transport, not to drop the ACKs itself.
* If the link doesn't support ECN, it still does the ACK filtering itself
* Links aren't going to delegate their control unless transports 
actually act on the ECN signals that links emit - and improve 
performance better than links have been. So, it's important that 
transports take on these responsibilities - the data sender takes 
responsibility for determining the ACK rate, asking its correspondent 
receiver does actually thin its own ACKs.

Here's who signals what, and who does what

                link-.
                      \
                       '-[ECN CE]-.
                                   \
                                    '-> receiver
                                     .-
                                    /
             .-[(Acc)ECN feedback]-'
            /
  sender <-'
         -.
           \
            '---[ACK CC ratio]---.
                                  \
                                   '-> receiver(treats as polite 
request, not mandatory)

> I don’t think think this is the correct approach - the result will be 

Why do you say the connection speed would be throttled?
Did the text not explain the proposal well?
Or did you understand the proposal but disagree with it?

> very odd requirement, it might be better too simply strip-out the 
> AccECN option or discard all packets that set this.

Sorry, that may be clumsy wording. It just means "the link SHOULD NOT 
filter ACKs" (if it supports ECN marking).

> them would improve the performance of AccECN TCP connections.

May be the piece where the link's preferred role is to signal to the 
transport to slow down the ACK rate using ECN wasn't clear.

Bob

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 9 Nov 2020 02:50:42 -0800

Hi Bob,

On Fri, Nov 6, 2020 at 5:01 AM Bob Briscoe <[email-masked]> wrote:

> Section 3.2.3 is about the AccECN TCP Option, which is optional and feeds
> back a count of *bytes* for each marking.
>
>    by those in Section 3.2.3.3 <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13#section-3.2.3.3>, which concern whether the AccECN TCP
> Also, even though traffic flows one direction, I think you might want to send back SACK options, etc.
>
>
> I see. Thanks for the clarification.

>
> In any cases, I just thought saving option space won't be a bad thing if it's not overly difficult.
>
>
> During the history of this draft we have proposed different ways to encode
> information more efficiently, but WG discussion decided that simple
> omission or inclusion of fields gives optional efficiency with minimal
> complexity.
>
> I'm not saying the act of reducing the granularity of each field would be
> particularly complex.
> efficiency, it would add many more ambiguous cases for this code to have to
> take into account.
> The design has gone round and round many times. We have tried to converge
> on the best tradeoffs.
>
>

OK. I have some concern that mitigating ambiguity by the using option may
require a certain high frequency,
but if the results from implementations look good, I don't have much things
to say here.

Thanks,
--
Yoshi

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 10 Nov 2020 09:41:35 +0000

See in-line, marked [GF], and I will try to unpick why I don't agree 
with some of this text.

On 06/11/2020 21:51, Bob Briscoe wrote:
> NB: n is the no. of *CE marks*, which is only the ACK thinning ratio 
> in the worst case if there's 100% marking.
> duration of a 100% marking sequence, whenever some data packets are 
> not marked, the receiver aACK ratio can rise. This *could* allow any 
> Just pointing out that n does not equal the ACK thinning ratio, it's 
> the floor of it.
>
> n=6 is a consequence of only having a 3-bit field to count CE marks. 
> We could push the envelope and make the max n=7. But not n=8, 
> otherwise wrap after 8 marks would be indistinguishable from zero marks.
>
[GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6 only when 
there is 100% marking - less with more likely patterns, that seems 
reasonable.

[GF] I also would agree that it is quite OK to increase the ACK-Ratio 
(send more ACKs) in some specific conditions, known to endpoints.

[GF] What concerns me, is that to avoid the need for ACK-Thining and 
related methods within the network, the "normal" condition needs to be 
for a rec eievr to send (far) fewer ACKs.

>>
>> 2. Although in some cases a receiver can predict a good "n", I do not 
> Surely the receiver is sending the ACKs, so either sentence as it 
> stands ends with "...I do not believe a receiver can typically decide 
> how often to send an ACK by observing how often it is sending ACKs 
> itself." ???
>
[GF] So, the topic and solution are a little complex ... let me try to 
write a simpler statement: The receiver alone cannot predict a good ACK 
ratio based on the ACKs the remote endpoint receives. ECN-marking of an 
ACK stream does not provide useful informatiopn, because in this 
specific case, the control packets are small, and we enter the realm of 
packet-congestible links (as in RFC7141), where the overhead of 
transmission time slot scheduling and fitting packets to burst 
transmission plans can be more sigmnificant than the number of bits sent.

>>
>> Sure, if the path is modelled as a simple bit-congestive path that 
>> has a queue, then you could monitoring timing/loss of packets and 
>> return path performance. In many radio technologies, the overhead 
>> from transmission of ACKs is proportionally much higher than for data 
> AccECN is a not the complete solution, but we've grasped the 
> opportunity to take a step in the right direction (within the 
> chartered scope of AccECN). AccECN at least allows the data sender to 
> measure the extent of congestion marking on the arriving ACK stream 
> (if there's ECN support at the bottleneck). Whereas before AccECN, a 
> random ECN mark on one pure ACK in a round couldn't be distinguished 
> from 100% of ACKs marked.
>
[GF] We've relied for decades on ACK-Thinning (or equivalent) in the 
network, where the link KNOWS what the sub-IP layer is doing, and hence 
knows whether an ACK is eating into resources that are scarce; but this 
solution is already problematic for encrypted traffic, and when ACKs 
carry more than ACK information. So, back to my point: To fix this, I 
think we need to reduce the level of feedback at the receiver.

[GF] So ... I see two problems with using ECN marks on the return path 
to vary the ACK Ratio:

(i) I'm not so sure the Layer 2 knows how to mark ACK packets - because 
they are not bit-congestible signals, and  I'd prefer to see ECN marking 
based on queue capacity (bit congestibale). I could see methods that let 
the sub-IP resource allocation ballance the use of forward and return 
transmssion opportunities and use this to CE-mark both directions - and 
see if flows adapt their return path resource consumption to optimise 
their forward data path transmission. It could perform better if it were 
to seperate ACKs (or any short packets? - also sometimes done) from 
"actual" data and then pro-actively CE-mark ACK packets to free-up 
transmission resource for other packets in the uplink or downlink. I'm 
not sure this cross-layer coupling is wise.

(ii) Before a new method is deployed, and the ACK Ratio remains 1:2, 
then the network nodes still do either ACK-Thinning or queue the ACKs.

> <https://tools.ietf.org/html/rfc5690> is informational).

[GF] To me, this is a complicated way to approach the question of how 
much feedback is needed. I actually don't see the justification for 
adding an extra control loop, that will interact with other control 
loops in the network.

>> ACKs can and do take transmission opportunities away from the 
>> resource pools that would be used by other services that share the 
>> same resource pool. This can be reduced by some form of per-flow 
>> scheduling with deeper per-flow queues - but that in itself can 
>> induce unwanted coupling between flows - limiting the peak rate of 
>> one flow, to achieve a higher ACK ratio for another flow.
>>
>> Consider a QUIC flow sharing a return path with a TCP flow, the 
>> higher rate of larger QUIC ACKs reduces the transmission 
>> opportunities for TCP ACKs, which in turn suppresses the TCP flow, 
>> limiting the overall system performance. If the ACKs use the same 
>> transmission resource, they could take directly away from 
>> transmission opportunities in the forward direction, but even if they 
>> do not - the rate of ACKs retuned impacts the sending rate end to end.
>
> of equitable capacity sharing {Note 2}, including between the ACK 
> stream of flow A and the data stream of flow B transferring in the 
> opposite direction.
>
> involved are the same for the right thing to happen (I think). But 
> some AD oversight wouldn't do any harm.
> solving the problem (the bottleneck or one of the transport 
> endpoints). This doesn't imply that the IETF will ever ensure that 
> other nodes don't 'interfere'. For instance, if the sender is deemed 
> responsible for determining the level of ACK filtering, it will still 
> have to make allowance for bottlenecks attempting to ACK thin as well.
>
>
>>
> This was the proposal that I was most interested to hear your views 
> on. We can remove it if the WG doesn't want it, but first I want to 
> make sure the proposal is fully understood:
> * The aim is for the RFC to tell bottleneck implementers (via this 
> text) that *only in cases where AccECN has been negotiated* the 
> preferred way to improve flow performance is for the link to signal 
> ECN to the transport, not to drop the ACKs itself.

[GF] In principle could be interesting - if this were a separate signal 
for "ACKs" - we don't have this signal. We have a signal for all IP traffic.

>                                   '-> receiver(treats as polite 
[GF] That's a useful diagram to help understand!!! The control loops for 
layer 2 channel sharing oftem operate on a similar timescale.

[GF] Let's be sure we start from the same problem:

* Current TCP Specs suggest the ACK Ratio is 1:2. I could say that means 
that 33% of transmission opportunities are used solely for feedback. 
Before we had sophisticated transports, tha was fine, but nowadays this 
is a very high proportion by modern standards, and given the way the 
ACK-Clock no longer has to be for each segment or pair of segments ... 
reality moved away from that when stretch-ACKs became common.

* Many senders (e.g, using offload) might have an ACK Ratio of 1:4, but 
not all.

* Some paths "Thin" ... by how much depends.  In many cases, an ACK 
Ratio 1:2, causes excessive resource usage that predjudices performance, 
an ACK Ratio of 1:4 is still 20% of transmission opportunities. An 
ACK-Ratio of 1:10, would mean 10% of transmission opportunities are used 
for feedback (that's an order of magnitude less control than data 
segments, and is a reasonable operating point). My suggestion is that 
MOST of the time, transports can live with an ACK Ratio 1:10, and in 
many cases this is enough to avoid ACK Thinning measures being triggered.

* A transport can adapt feedback, for sure,  and there will be cases 
where lower rate (e.g. 1:1 or 1:2 initially) or higher rates (1:10,1:20, 
etc as rates increase and rate-based methods are used) will be benefit 
the transport - but changing the default operating point from 1:2 to 
1:10 is likely already significantly reducing the need (and triggering) 
of ACK thinning.

- So, maybe we should do this, rather than encourage a new generation of 
smart ECN-based ACK modification schemes. If we mitigate the feedback 
traffic, we can then focus on deploying ECN for capacity detection.

> Why do you say the connection speed would be throttled?
[GF] For example: A sudden increasse in reverse path traffic will need 
to be buffered until the radio transmission schedule changes. That will 
delay feedback to the sender (advertised rwnd/cwnd for TCP), and hence 
the forward rate will then throttle determined by the bottleneck ACK rate.
> Did the text not explain the proposal well?

Well, I think the problem is complex, which is why this email is already 
long.

>> - does that imply that nodes need to discard rather than delay 
>> packets when link scheduling or transmission scheduling is used. This 
>> seems a very odd requirement, it might be better too simply strip-out 
>> the AccECN option or discard all packets that set this.
>
>> I can’t see how queuing the ACKs, or deleting ACKs rather than 
>> queuing them would improve the performance of AccECN TCP connections.
>
> Bob Briscoehttp://bobbriscoe.net/

Summary ... because this thread has lots of "discussion" or a 
complicated interaction, and you asked about the AccECN draft text:

+ I agree that AccECN can be used with a sender that uses a more 
appropiate target ACK Ratio - that is good! I agree this is worth saying.

+ ACK Thining, and variants - including uses of compression - is a 
deployed mitigation, and that changing the ACK Ratio can eliminate this 
practice and the need to implement this.

+ I agree that if the path is a simple bottleneck, then ECN-marking 
should be encouraged. However, not appropiate without much more 
consideration for a more complex L2 where the marking is 
packet-congestible. A link predomonantly carrying small ACKs seems to 
lead into the research issues in RFC7141.

+ I would disagree with any encouragement to use cross-layer methods 
with use ECN-marking to seek to adjust ACK feedback traffic and I am 
concerned that using ECN to modify the ACK Ratio might create new layers 
of feedback loop that will aggrevate the problem.

I think the current text goes to far.

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 12 Nov 2020 02:40:52 +0000

Gorry, [this thread is getting even longer, but I think we might soon 
start to violently agree.]
Inline tagged [BB2]...

On 10/11/2020 09:41, Gorry Fairhurst wrote:
>>> adaptation, the adaption if necessary needs to be introduce more 
>>> ACKs when this is beneficial, as in some cases for DAAS, not in 
>>> trying to figure something from delayed or "missing" ACKs.
>>>

[BB2] "What concerns me is that, ... the normal condition needs to be 
for a receiver to send (far) fewer ACKs".
That's ambiguous to me... it might mean:
a) your concern is that the implication of what I'm saying will need far 
fewer ACKs (which you don't want).

But given what you say later, I'm pretty sure you meant:
b) your concern is that you want the normal condition to be fewer ACKs 
(i.e. you want the opposite of (a))

>>>
>>> 2. Although in some cases a receiver can predict a good "n", I do 
>>> not believe a receiver can typically find out how to choose “n” by 
> write a simpler statement: The receiver alone cannot predict a good 
> ACK ratio based on the ACKs the remote endpoint receives. ECN-marking 
> of an ACK stream does not provide useful informatiopn, because in this 
> specific case, the control packets are small, and we enter the realm 
> of packet-congestible links (as in RFC7141), where the overhead of 

[BB2] If a link becomes packet-congested, ECN can (and should) still be 
used to indicate such congestion. The implementation has to be designed 
so that the ECN-marking process isn't compromised by packet processing 
overload. But that's perfectly possible for a good implementer.

>>> determine a rate that tries not to fill a queue. I would not know 
>>> how common such a bottleneck link is.
>>>
>>> from transmission of ACKs is proportionally much higher than for 
>>> data segments -  These layer 2s are constrained in terms of packets 
>>> (e.g. where lower layer framing and transmission scheduling is an 
>>> important consideration to the efficiency/cost of using the 
>>> service). This has often been “fixed” using ACK Thinning or PEPs. 
>>> Protocols that send fewer ACKs/data packet are not impacted.
>>
> network, where the link KNOWS what the sub-IP layer is doing, and 
> hence knows whether an ACK is eating into resources that are scarce; 
> but this solution is already problematic for encrypted traffic, and 
> when ACKs carry more than ACK information. So, back to my point: To 
> fix this, I think we need to reduce the level of feedback at the receiver.
>

[BB2] Certainly as we get more packets per RTT, we shouldn't need to 
have more ACKs per RTT. Better still, we ought to pack more information 
into each ACK.

However, do you have enough evidence to say reducing the absolute level 
of feedback is /all/ that is necessary? That's quite a strong assertion. 
It presumes that there is not a trade-off. It says, if we achieve some 
lower level of feedback then, in all the cases when the reverse path is 
uncongested, it would never be advantageous to increase the ACK frequency.

> (i) I'm not so sure the Layer 2 knows how to mark ACK packets - 
> because they are not bit-congestible signals, and  I'd prefer to see 
> ECN marking based on queue capacity (bit congestibale).
>

[BB2] I don't see why ECN is hard or inappropriate for 
packet-congestion. Is there any experience that says that? And I don't 
know why you'd prefer to see ECN for bit congestion. Is there some 
reasoning behind that?

> I could see methods that let the sub-IP resource allocation ballance 
> the use of forward and return transmssion opportunities and use this 
> to CE-mark both directions - and see if flows adapt their return path 
> resource consumption to optimise their forward data path transmission. 
> It could perform better if it were to seperate ACKs (or any short 
> packets? - also sometimes done) from "actual" data and then 
> pro-actively CE-mark ACK packets to free-up transmission resource for 
> other packets in the uplink or downlink. I'm not sure this cross-layer 
> coupling is wise.
>

[BB2] Why complicate matters? A link doesn't know which flow in one 
direction relates to which flow in the other (the reverse path of some 
flows might not even go through this link). And a link doesn't know the 
relative importance of ACKs and data, which is specific to different 
congestion controllers. So how can a link balance both directions well? 
What's wrong with the link blindly signalling congestion on packets 
(irrespective of whether data or ACKs), in each direction independently. 
And let the end systems find which degree of self-controlled 
ACK-thinning maximizes their performance.

What is important is to come to consensus on whether the link is or is 
not the best place to control reverse path congestion.  As above, there 
is scope for transports to just reduce the regular ACK load, full stop. 
Beyond that, to find a good trade-off between too much and too few ACKs, 
I certainly think we can agree that neither the transport nor the link 
can do that well on its own. Communication between them seems essential. 
Attempting to communicate from host to network seems like a non-starter, 
given the information the transport would have to convey would be quite 
complex. Congestion signalling from the link to the transport seems 
sufficient (to me). And using ECN seems likely to be significantly 
better than using loss.

> then the network nodes still do either ACK-Thinning or queue the ACKs.

[BB2] As above, I'm sure there's scope for reducing ACK frequency 
without (significantly?) affecting performance. And you've been doing 
this research, so you should know. And I guess, maybe, that might make 
most link-based ACK filtering unnecessary. But I don't think we need to 
decide now that this will be enough, and ACK CC will not be necessary 
(see later).

> adding an extra control loop, that will interact with other control 

[BB2] Yes, I understand that you favour the transport picking a higher 
ACK ratio for all paths. But surely the empirical evidence from 
link-based ACK filtering shows that it adapts the ACK ratio to 
conditions, and doesn't just apply a higher ACK ratio all the time. But 
I guess you would argue that's because links are fighting with 
transports that weren't specifically designed to cope with a higher ACK 
ratio, so not filtering at all is better when it's not necessary.

>>> transmission opportunities in the forward direction, but even if 
>>> they do not - the rate of ACKs retuned impacts the sending rate end 
>>> to end.

[BB2] A question:
Has the approach ever been tried where the link just does dumb 
congestion notification (without regard to type or size of packets), and 
the end system congestion controls optimize data transfer performance 
and the tradeoff with ACK ratio? Whether analytically, or empirically?

>> * and all ACK CC algorithms ought to be designed to ensure some 
>> degree of equitable capacity sharing {Note 2}, including between the 
>> ACK stream of flow A and the data stream of flow B transferring in 
>> the opposite direction.
>>
>> where the transport ADs co-ordinate multiple WGs. Enough of the 
>> people involved are the same for the right thing to happen (I think). 
>> But some AD oversight wouldn't do any harm.
>>> layers, which seem to imply more complexity. I’m not comfortable 
>>> with such a proposal.
>>>
>>> It seems that the text suggests not implement thinning and that 
>>> these systems implement logic to detect the presence of the option 
>>> by observing SYNs and then configure the link to do something special. 
>>
> [GF] In principle could be interesting - if this were a separate 
> signal for "ACKs" - we don't have this signal. We have a signal for 
> all IP traffic.
>

[BB2] I don't think the signal should be different for ACKs and data. 
That's unnecessarily complicated for the link, and makes the transport's 
job harder if different links each choose a different balance (it's hard 
enough even if all links use the same balance).

>> responsibility for determining the ACK rate, asking [I meant 
>> 'assuming'] its correspondent receiver does actually thin its own ACKs.
> [GF] That's a useful diagram to help understand!!! The control loops 
> for layer 2 channel sharing oftem operate on a similar timescale.
>
> * Current TCP Specs suggest the ACK Ratio is 1:2. I could say that 
> means that 33% of transmission opportunities are used solely for 
> feedback.
>

[BB2] Well, that depends how much aggregation is used to pack packets 
into each xmt opportunity. But your figures here and below are still 
meaningful relative to each other, even if aggregation shifts them all 
downwards in absolute terms.

> Before we had sophisticated transports, tha was fine, but nowadays 
> this is a very high proportion by modern standards, and given the way 
> the ACK-Clock no longer has to be for each segment or pair of segments 
> ... reality moved away from that when stretch-ACKs became common.
>
> * Many senders (e.g, using offload) might have an ACK Ratio of 1:4, 
> but not all.
>
> Ratio 1:2, causes excessive resource usage that predjudices 
> performance, an ACK Ratio of 1:4 is still 20% of transmission 
> opportunities. An ACK-Ratio of 1:10, would mean 10% of transmission 
> opportunities are used for feedback (that's an order of magnitude less 
> control than data segments, and is a reasonable operating point). My 
> suggestion is that MOST of the time, transports can live with an ACK 
> Ratio 1:10, and in many cases this is enough to avoid ACK Thinning 
> measures being triggered.
>
> where lower rate (e.g. 1:1 or 1:2 initially) or higher rates 
> (1:10,1:20, etc as rates increase and rate-based methods are used) 
> will be benefit the transport - but changing the default operating 
> point from 1:2 to 1:10 is likely already significantly reducing the 
> need (and triggering) of ACK thinning.
>

[BB2] The point tho' is that the opportunity for ACK thinning has arisen 
as packet rates have increased, because you only really need a certain 
number of ACKs per RTT to maintain control. E.g if you need 8 ACKs per 
RTT then if the window is 16 you need ACK ratio = 2, but if the window 
is 320, ACK ratio = 40 should work OK. However, just because the average 
flow rate is continually increasing, doesn't mean flows don't still need 
to go slow before they go fast.

If a link enforces an ACK ratio, it doesn't know which flows are 
currently getting started, which flows have small RTTs, etc. etc. Flows 
know that.

> - So, maybe we should do this, rather than encourage a new generation 
> of smart ECN-based ACK modification schemes. If we mitigate the 
> feedback traffic, we can then focus on deploying ECN for capacity 
> detection.
>

[BB2] I'm fine starting out with transports doing more ACK thinning 
themselves.
But it's no use if links still filter the ACKs of these 'good' flows on 
top of the flows filtering themselves.

QUIC has encryption to prevent that (at least until networks find 
heuristics to do QUIC ACK thinning). TCP does not.

So it does not harm to require links not to filter AccECN-capable 
packets (perhaps we can widen that to all ECN-capable packets).
And it also does no harm to recommend that links ECN mark instead. If 
they did, they would of course ECN-mark all ECN-capable packets, of course.

Then:
* if your strategy of cutting down the absolute ACK ratio is sufficient, 
we don't need to introduce ACK CC, but we still have more ECN deployment 
anyway.
* And if your strategy isn't enough, we have the signals for the basis 
of robust ACK CC.

>>> I don’t think think this is the correct approach - the result will 
>>> be throttling of the connection speed. Is it necessary that the 
>>> sub-IP layer does not implement thinning for connections that use 
>>> AccECN feedback?
>>
> to be buffered until the radio transmission schedule changes. That 
> will delay feedback to the sender (advertised rwnd/cwnd for TCP), and 
> hence the forward rate will then throttle determined by the bottleneck 
> ACK rate.

[BB2] Well, actually the ACKs leaving the head of the buffer should be 
congestion-marked for the backlog behind them when they dequeue, as well 
as for their sojourn time (due to the buffer in front of them when they 
enqueued). I can point you at a paper on this if interested.
But, whatever, this is a better way to get the receiver to thin the ACK 
stream than the network discarding packets from it (better = less 
damaging to performance).

And, bear in mind that the first step is for transports to reduce ACK 
ratio overall, so we need a way for the link to distinguish the old 
transports that probably haven't reduced their ACK ratio, from newer 
ones that probably have.

Encouraging the link to ECN mark (all traffic incl ACKs) is worthwhile 
whether or not it's used for ACK CC.

And we only resort to ACK CC if necessary - but the signals are there if 
we need to develop the algorithms.

> Well, I think the problem is complex, which is why this email is 
> already long.
>
>>> packets when link scheduling or transmission scheduling is used. 
>>> This seems a very odd requirement, it might be better too simply 
>>> strip-out the AccECN option or discard all packets that set this.
>>

Bob

> deployed mitigation, and that changing the ACK Ratio can eliminate 
> this practice and the need to implement this.
>
> concerned that using ECN to modify the ACK Ratio might create new 
> layers of feedback loop that will aggrevate the problem.
>