Subject: Re: [tsvwg] start of WGLC on L4S drafts: DualQ AQM

From: Bob Briscoe <[email-masked]>
Date: Tue, 2 Nov 2021 16:36:03 +0000

Gorry,

I've now worked through all your comments. See [BB2]

On 25/10/2021 10:35, Gorry Fairhurst wrote:
> On 23/10/2021 17:18, Bob Briscoe wrote:
>> Gorry,
>>
>> Thanks for your (as always) diligent review. Unless I've responded 
>> below with [BB], consider them accepted as written...
>>
>> On 12/08/2021 11:30, Gorry Fairhurst wrote:
>>> I have reviewed DualQ Coupled AQMs for  L4S
>>>
>>> I re-reviewed: draft-ietf-tsvwg-aqm-dualq-coupled-16 in the WGLC. As 
>>> an inidvidual, I support publication as a PS, and have some comments 
>>> below on -16, that
>>>
>>> 1)
>>> The abstract is unnecessarily long. It does not need to provide this 
>>> level of background information. I’d suggest the editors shorten this.
>>
>> [BB] I've compressed it down to the size of the first of the two 
>> paras. I haven't run this past my co-authors, so you might see some 
>> (hopefully minor) changes in the version that gets submitted.
>>
>> This specification defines a framework for coupling the Active Queue 
>> Management (AQM) algorithms in two queues intended for flows with 
>> different responses to congestion. This provides a way for the 
>> Internet to transition from the scaling problems of standard TCP 
>> Reno-friendly ('Classic') congestion controls to the family of 
>> 'Scalable' congestion controls. These achieve consistently very Low 
>> queuing Latency, very Low congestion Loss and Scaling of per-flow 
>> throughput (L4S) by using Explicit Congestion Notification (ECN) in a 
>> modified way. Until the Coupled DualQ, these L4S senders could only 
>> be deployed where a clean-slate environment could be arranged, such 
>> as in private data centres. The coupling acts like a semi-permeable 
>> membrane; isolating the sub-millisecond average queuing delay and 
>> zero congestion loss of L4S from Classic latency and loss; but 
>> pooling the capacity between any combination of Scalable and Classic 
>> flows with roughly equivalent throughput per flow. The DualQ achieves 
>> this indirectly, without having to inspect transport layer flow 
>> identifiers and without compromising the performance of the Classic 
>> traffic. The solution has low complexity and requires no 
>> configuration for the public Internet.
>>
> I don't think the abstract needs to claim what is achieved, only what 
> is sought by the spec. Claiming what is achieved would require 
> evidence, and claiming a solution is not necessary. Instead I think an 
> RFC seeks to publish a spec that intends to deliver an expected 
> outcome. From my perspective, I'd feel happier with changes such as:
>
> /achieve/are designed to achieve/
>
> /The solution has low/The DualQ AQM design has low/
>

[BB2] Done.

>>> ---
>>> 2) The 'Low Loss" part of the name denotes
>>> - This has mixed quote marks. In other places, double quotes are used.
>>> ---
>>> 3) There appears to be some repetition of topics in section 1.4 
>>> between this and the L4S arch specifications. This just gives more 
>>> to read. Please do consider removing duplicate examples/discussion.
>>
>> [BB] The "Problem Outline" section was rewritten over Drafts 09 & 10 
>> (both July 2019) to be specific to the Problem the DualQ solves, and 
>> other material was moved to l4s-arch.
>>
>> The Scope section was also rewritten. It has a couple of paras 
>> putting the DualQ in the wider context of scalable congestion 
>> controls, then refers to l4s-arch for details. The rest of the scope 
>> section is specific to the DualQ. Yes, it does pick some pieces out 
>> of l4s-arch (e.g. most likely deployment scenario for the dualQ). But 
>> the aim was to give implementers who are only interested in the DualQ 
>> what they need.
>>
>> This might not be the way you would have chosen to decide which 
>> material to include or exclude, but I hope you'll understand that 
>> this means I really don't want to go through the exercise of 
>> rewriting these sections again,... except to address specific concerns.
>>
> Understood, to avoid multiple advice, I'll defer to the Shepherd and 
> AD for advice here, if any is needed.
>>> ---
>>> 4) Typo - additional space:
>>> “[RFC5681] .”
>>> ---
>>> 5) Unpublished Individual ID as a Ref explaining a “MUST”:
>>> “The two queues can be part of a larger
>>>    queuing hierarchy [I-D.briscoe-tsvwg-l4s-diffserv].”
>>> - Since this relates a to a requirement, I do not think this is 
>>> appropiate. The reference provided is out of date, and does not 
>>> appear to be progressing. Could this reference be removed?
>>
>> [BB] You're right the normative section about Functional Requirements 
>> is not the right place for this. I've deleted the whole second sentence.
>>
>> Nonetheless, an implementer will need to have an idea how this DualQ 
>> fits into their existing queuing structures, so this sentence is 
>> worth keeping somewhere. I've stuck it at the end of §2.4 "Overall 
>> DualQ Coupled AQM Structure". I've also kept the ref, 'cos there is 
>> no other place an implementer could go for this information. But it's 
>> not in a normative section any more, and I've added some fluff to 
>> make it clearer that this reference is just early ideas:
>>     "The two queues could optionally be part of a larger queuing 
>> hierarchy, such as the initial example ideas in 
>> [I-D.briscoe-tsvwg-l4s-diffserv]."
>
> OK
>
>>
>> BTW, as I understand the chairs' position on the l4s-diffserv draft, 
>> it is 'on hold' while the WG focuses on the main L4S drafts. That 
>> means there will be no request for adoption (yet), but it also hasn't 
>> been refused adoption.
> Yes - this should only be used as an example.
>>
>>> ---
>>> 6) Unpublished Individual ID as a Ref:
>>> “and Voice-Admit
>>>    service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]),
>>> - I’d prefer to leave this without a REF that to refer to an 
>>> individual draft submission.
>>
>> [BB] I've replaced all 4 refs in this para with one ref to §5.4.1 of 
>> ecn-l4s-id. We can then confine this discussion of which refs are 
>> appropriate to that doc.
>>
>> CURRENT:
>> For instance addresses of specific applications or hosts (see 
>> [I-D.ietf-tsvwg-ecn-l4s-id 
>> <https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-ecn-l4s-id>]), 
>> specific Diffserv codepoints such as EF (Expedited Forwarding) and 
>> Voice-Admit service classes (see [I-D.briscoe-tsvwg-l4s-diffserv 
>> <https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.briscoe-tsvwg-l4s-diffserv>]), 
>> the Non-Queue-Building (NQB) per-hop behaviour [I-D.ietf-tsvwg-nqb 
>> <https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-nqb>] 
>> or certain protocols (e.g. ARP, DNS).
>>
>> PROPOSED:
>> For instance addresses of specific applications or hosts; specific 
>> Diffserv codepoints such as EF (Expedited Forwarding), Voice-Admit or 
>> the Non-Queue-Building (NQB) per-hop behaviour; or certain protocols 
>> (e.g. ARP, DNS) (see Section 5.4.1 of [I-D.ietf-tsvwg-ecn-l4s-id 
>> <https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-ecn-l4s-id>]). 
>>
> OK with me.
>>
>>> ---
>>> 7) NiT: “The scheduler SHOULD be work-conserving.”
>>> - This “SHOULD” introduces a design decision: Please add a one 
>>> sentence explanation what happens if it is not.
>>
>> [BB]
>> PROPOSED:
>> The scheduler SHOULD be work-conserving, or otherwise close to work 
>> conserving, given Classic service will often rely on borrowing from 
>> the L4S service.
>>
> I'm not sure the additional explanation needs to be in the RFC2119 
> clause, it might actually be clearer as a separate following sentence.
>
> The senetence seems to be going in the correct direction, but is 
> "borrowing" clear in this context?
>

[BB2] I'm pretty sure borrowing is a commonly used term  in the 
scheduler world, altho I can only find it in the context of CBQ. So I'll 
avoid it, and after 'work-conserving' I've started a new sentence 
reworded as follows:

     "This is because Classic traffic needs to be able to efficiently 
fill any space left by L4S traffic even though the scheduler would 
otherwise allocate it to L4S."

>>> ---
>>> 8) Dead reference (used multiple times)
>>> "if the packet is Not-ECT, the appropriate action depends on
>>>          whether some other function is protecting the L queue from
>>>          misbehaving flows (e.g. per-flow queue
>>>          protection [I-D.briscoe-docsis-q-protection] or latency
>>>          policing)"
>>> "Therefore per-flow policing
>>>    (e.g. [I-D.briscoe-docsis-q-protection]) needs to be separable 
>>> from a
>>>    basic AQM, as an option under policy control."
>>> "However, the addition of DOCSIS per-flow
>>>    Queue Protection [I-D.briscoe-docsis-q-protection] turns this into
>>>    'delay on saturation"
>>> - The tracker shows draft-briscoe-docsis-q-protection has expired 
>>> and is no longer active. What is the current status of this ID?
>>
>> [BB] The ISE has indicated it is OK in principle to proceed to 
>> publication. I have a couple of edits to it and I just haven't 
>> prioritized them. Sorry.
>>
> That's OK.
>>> ---
>>> 9) RFC2119 usage:
>>> " The following
>>>    parameters MAY be operator-configurable, e.g. to tune for non-
>>>    Internet settings:"
>>> - To me this does not seem like a standards interoperability 
>>> statement. Do we need RFC2119 language, would it be OK to just say:
>>> " The following
>>>    parameters can be operator-configurable, e.g. to tune for non-
>>>    Internet settings:"
>>
>> [BB] I thought this was one of the few occasions where MAY is 
>> appropriate. But obviously you don't. If an implementer doesn't 
>> follow this MAY, their device won't be able to interoperate well with 
>> congestion controls in non-Internet environments. But whether this is 
>> a concern depends on the market their device is intended to address.
>> To me that's nearly word-for-word what RFC2119 says about the 
>> intended use of MAY.
>>
>> I'll leave it for now, and see what you say in response.
>>
> It would be nice if the sentence before says the default configuration 
> parameters need to be safe for Internet deployment - I didn't check, 
> but you'll know.

[BB2] I'm not sure whether you want me to do anything here. The sentence 
before seems OK to me - it's restating the zero-config recommendation of 
RFC 7567:
     "By default, a DualQ Coupled AQM SHOULD NOT need any configuration 
for use at a bottleneck on the public Internet [RFC7567]."

>>> ---
>>> 10)RFC2119 usage:
>>> "The maximum queue
>>>       delay per queue per interval MAY also be recorded."
>>> - I'd sa surely a system can record anything? --  So this does not 
>>> seem to need RFC2119 language, would it be OK to just say "can"... 
>>> I'd also be interested to see a sentence that helps me understand 
>>> *why* this might be useful as a log entry.
>>
>> [BB] I've added "..., to aid diagnosis of faults and anomalous events"
>> Again, this is really about interop - not data plane, but management 
>> plane - interop with typical monitoring systems. So there's the same 
>> argument for a 'MAY' as above.
>> Again, I'll leave in the MAY and see if you disagree.
>>
> Perfect.
>>> ---
>>> 11) NiT:
>>> "Curvy RED can be used as is when
>>>    the RTT range to be supported is limited, otherwise an adaptation
>>>    mechanism is required."
>>> /required/needed/ - for avoidance of doubt?
>>
>> [BB] I'm impressed that you've found this in the depths of an appendix!
>> Agree with 'needed' in this first case, but...
>>
> OK
>>> ---
>>> 12) NiT:
>>> "Priority of L4S is required to be conditional to avoid total
>>>    starvation of Classic by heavy L4S traffic."
>>> /required/needed/ - for avoidance of doubt?
>>
>> [BB] Here, 'required' is actually correct. So I've added a pointer to 
>> the requirement:
>>     "Priority of L4S is required to be conditional (see Section 
>> 2.5.1) to avoid..."
>>
>> Thank you very much.
>>
> Aha, that's good.

[BB2] Thank you again,

Bob

>>
>> Bob
>>
> Seems like you might have what you need from me to make the next revision.
>
> Gorry
>
>>> --------------
>>>>
>>>> On 29.07.21, 18:18, "tsvwg on behalf of Wesley 
>>>> Eddy"<[email-masked] on behalf of [email-masked]>  wrote:
>>>>
>>>>     This message is starting a combined working group last call on 
>>>> 3 of the
>>>>     L4S drafts:
>>>>
>>>>     - 
>>>> Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/ 
>>>>
>>>>
>>>>     - DualQ:
>>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
>>>>
>>>>     - ECN 
>>>> ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>>>>
>>>>     The WGLC will last through 4 weeks from today, and then we'll 
>>>> see what
>>>>     to do next.  Please submit any comments you have on these to 
>>>> the TSVWG
>>>>     list in that timeframe.
>>>>
>>>>     The chairs are considering a possible virtual interim following 
>>>> the
>>>>     close in order to work through feedback received.
>>>>
>>>>     The work on the L4S operational guidance draft is continuing in
>>>>     parallel, but that draft is not being last called yet.
>>>>
>>>>
>>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 2 Nov 2021 17:03:22 +0000

Thanks Bob. I suspect this addresses All my current Comments on this draft. 

Gorry

> On 2 Nov 2021, at 16:36, Bob Briscoe <[email-masked]> wrote:
> 
>  Gorry,
> 
>>> Thanks for your (as always) diligent review. Unless I've responded below with [BB], consider them accepted as written...
>>> 
>>>> I re-reviewed: draft-ietf-tsvwg-aqm-dualq-coupled-16 in the WGLC. As an inidvidual, I support publication as a PS, and have some comments below on -16, that 
>>>> 
>>>> 1) 
>>>> The abstract is unnecessarily long. It does not need to provide this level of background information. I’d suggest the editors shorten this. 
>>> 
>>> [BB] I've compressed it down to the size of the first of the two paras. I haven't run this past my co-authors, so you might see some (hopefully minor) changes in the version that gets submitted.
>>> 
>>> This specification defines a framework for coupling the Active Queue Management (AQM) algorithms in two queues intended for flows with different responses to congestion. This provides a way for the Internet to transition from the scaling problems of standard TCP Reno-friendly ('Classic') congestion controls to the family of 'Scalable' congestion controls. These achieve consistently very Low queuing Latency, very Low congestion Loss and Scaling of per-flow throughput (L4S) by using Explicit Congestion Notification (ECN) in a modified way. Until the Coupled DualQ, these L4S senders could only be deployed where a clean-slate environment could be arranged, such as in private data centres. The coupling acts like a semi-permeable membrane; isolating the sub-millisecond average queuing delay and zero congestion loss of L4S from Classic latency and loss; but pooling the capacity between any combination of Scalable and Classic flows with roughly equivalent throughput per flow. The DualQ achieves this indirectly, without having to inspect transport layer flow identifiers and without compromising the performance of the Classic traffic. The solution has low complexity and requires no configuration for the public Internet.
>>> 
>> I don't think the abstract needs to claim what is achieved, only what is sought by the spec. Claiming what is achieved would require evidence, and claiming a solution is not necessary.  Instead I think an RFC seeks to publish a spec that intends to deliver an expected outcome. From my perspective, I'd feel happier with changes such as:
>> 
>>>> 3) There appears to be some repetition of topics in section 1.4 between this and the L4S arch specifications. This just gives more to read. Please do consider removing duplicate examples/discussion. 
>>> 
>>> [BB] The "Problem Outline" section was rewritten over Drafts 09 & 10 (both July 2019) to be specific to the Problem the DualQ solves, and other material was moved to l4s-arch.
>>> 
>>> The Scope section was also rewritten. It has a couple of paras putting the DualQ in the wider context of scalable congestion controls, then refers to l4s-arch for details. The rest of the scope section is specific to the DualQ. Yes, it does pick some pieces out of l4s-arch (e.g. most likely deployment scenario for the dualQ). But the aim was to give implementers who are only interested in the DualQ what they need.
>>> 
>>> This might not be the way you would have chosen to decide which material to include or exclude, but I hope you'll understand that this means I really don't want to go through the exercise of rewriting these sections again,... except to address specific concerns. 
>>> 
>> Understood, to avoid multiple advice, I'll defer to the Shepherd and AD for advice here, if any is needed.
>>>> --- 
>>>> - Since this relates a to a requirement, I do not think this is appropiate. The reference provided is out of date, and does not appear to be progressing. Could this reference be removed? 
>>> 
>>> [BB] You're right the normative section about Functional Requirements is not the right place for this. I've deleted the whole second sentence.
>>> 
>>> Nonetheless, an implementer will need to have an idea how this DualQ fits into their existing queuing structures, so this sentence is worth keeping somewhere. I've stuck it at the end of §2.4 "Overall DualQ Coupled AQM Structure". I've also kept the ref, 'cos there is no other place an implementer could go for this information. But it's not in a normative section any more, and I've added some fluff to make it clearer that this reference is just early ideas:
>>>     "The two queues could optionally be part of a larger queuing hierarchy, such as the initial example ideas in [I-D.briscoe-tsvwg-l4s-diffserv]."
>> OK
>> 
>>> 
>>> BTW, as I understand the chairs' position on the l4s-diffserv draft, it is 'on hold' while the WG focuses on the main L4S drafts. That means there will be no request for adoption (yet), but it also hasn't been refused adoption.
>>>> - I’d prefer to leave this without a REF that to refer to an individual draft submission. 
>>> 
>>> [BB] I've replaced all 4 refs in this para with one ref to §5.4.1 of ecn-l4s-id. We can then confine this discussion of which refs are appropriate to that doc.
>>> 
>>> CURRENT:
>>> For instance addresses of specific applications or hosts (see [I-D.ietf-tsvwg-ecn-l4s-id]), specific Diffserv codepoints such as EF (Expedited Forwarding) and Voice-Admit service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]), the Non-Queue-Building (NQB) per-hop behaviour [I-D.ietf-tsvwg-nqb] or certain protocols (e.g. ARP, DNS). 
>>> 
>>> PROPOSED:
>>> For instance addresses of specific applications or hosts; specific Diffserv codepoints such as EF (Expedited Forwarding), Voice-Admit or the Non-Queue-Building (NQB) per-hop behaviour; or certain protocols (e.g. ARP, DNS) (see Section 5.4.1 of [I-D.ietf-tsvwg-ecn-l4s-id]). 
>> OK with me.
>>> 
>>>> --- 
>>>> - This “SHOULD” introduces a design decision: Please add a one sentence explanation what happens if it is not. 
>>> 
>>> [BB]
>>> PROPOSED:
>>> The scheduler SHOULD be work-conserving, or otherwise close to work conserving, given Classic service will often rely on borrowing from the L4S service.
>>> 
>> I'm not sure the additional explanation needs to be in the RFC2119 clause, it might actually be clearer as a separate following sentence. 
>> 
>> The senetence seems to be going in the correct direction, but is "borrowing" clear in this context?
>> 
> 
> [BB2] I'm pretty sure borrowing is a commonly used term  in the scheduler world, altho I can only find it in the context of CBQ. So I'll avoid it, and after 'work-conserving' I've started a new sentence reworded as follows:
> 
>     "This is because Classic traffic needs to be able to efficiently fill any space left by L4S traffic even though the scheduler would otherwise allocate it to L4S."
> 
>>>> --- 
>>>>    (e.g. [I-D.briscoe-docsis-q-protection]) needs to be separable from a 
>>>> - The tracker shows draft-briscoe-docsis-q-protection has expired and is no longer active. What is the current status of this ID? 
>>> 
>>> [BB] The ISE has indicated it is OK in principle to proceed to publication. I have a couple of edits to it and I just haven't prioritized them. Sorry.
>>> 
>> That's OK. 
>>>> --- 
>>>> 9) RFC2119 usage: 
>>>> " The following 
>>>> - To me this does not seem like a standards interoperability statement. Do we need RFC2119 language, would it be OK to just say: 
>>>> " The following 
>>> [BB] I thought this was one of the few occasions where MAY is appropriate. But obviously you don't. If an implementer doesn't follow this MAY, their device won't be able to interoperate well with congestion controls in non-Internet environments. But whether this is a concern depends on the market their device is intended to address. 
>>> To me that's nearly word-for-word what RFC2119 says about the intended use of MAY.
>>> 
>> It would be nice if the sentence before says the default configuration parameters need to be safe for Internet deployment - I didn't check, but you'll know.
> 
> [BB2] I'm not sure whether you want me to do anything here. The sentence before seems OK to me - it's restating the zero-config recommendation of RFC 7567:
>     "By default, a DualQ Coupled AQM SHOULD NOT need any configuration for use at a bottleneck on the public Internet [RFC7567]."
> 
> 
>>>> --- 
>>>> 10)RFC2119 usage: 
>>>> "The maximum queue 
>>>> - I'd sa surely a system can record anything? --  So this does not seem to need RFC2119 language, would it be OK to just say "can"... I'd also be interested to see a sentence that helps me understand *why* this might be useful as a log entry. 
>>> 
>>> Again, this is really about interop - not data plane, but management plane - interop with typical monitoring systems. So there's the same argument for a 'MAY' as above. 
>>> [BB] Here, 'required' is actually correct. So I've added a pointer to the requirement:
>>>     "Priority of L4S is required to be conditional (see Section 2.5.1) to avoid..."
>>> 
>>> Thank you very much.
>>> 
>> Aha, that's good.
> 
> [BB2] Thank you again,
> 
> 
> Bob
> 
> 
>>> 
>>> Bob
>>> 
>>>>> On 29.07.21, 18:18, "tsvwg on behalf of Wesley Eddy"<[email-masked] on behalf of [email-masked]>  wrote: 
>>>>> 
>>>>>     This message is starting a combined working group last call on 3 of the 
>>>>>     L4S drafts: 
>>>>> 
>>>>>     - Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/ 
>>>>> 
>>>>>     - DualQ: 
>>>>>     - ECN ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/ 
>>>>> 
>>>>>     The WGLC will last through 4 weeks from today, and then we'll see what 
>>>>>     to do next.  Please submit any comments you have on these to the TSVWG 
>>>>>     list in that timeframe. 
>>>>> 
>>>>>     The chairs are considering a possible virtual interim following the

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 5 Nov 2021 12:30:55 +0100

Hi Gorry,

thanks.

> On Oct 28, 2021, at 17:42, Gorry Fairhurst <[email-masked]> wrote:
> 
> I'll quote some words from RFC805.

[SM] https://www.rfc-editor.org/rfc/rfc8085, "RFC 805: Computer Mail Meeting Notes" seems a bit of a stretch ;)

> 
> This notes that transport need to "employ suitable mechanisms to prevent congestion collapse and
> establish a degree of fairness".
> 
> It also makes this recommendation:
>    "Bulk-transfer applications that choose not to implement TFRC or TCP-
>   like windowing SHOULD implement a congestion control scheme that
>   results in bandwidth (capacity) use that competes fairly with TCP
>   within an order of magnitude."

[SM] Question: am I right that we are talking base2-orders of magnitude or base10? I presume base2.

But the core of my question was more about whether a systemic bias of noticeable magnitude (say because L4S flow consistently get higher throughput) is to be considered differently from a random bias (where the fairness between flows depends on the exact conditions and the "winner" is not reliably predictable).

Now, I am also puzzled, if this one order of magnitude fairness is a BCP, why are we considering RFC-ing DualQ which at short RTTs robustly and  reliably introduces imbalances on the order of 1:15 (as in the draft) or 1:10 as in the implementation for Linux, both of which are clearly >> 1:2, and the argument about the added queueing delay for the classic flow does not hold water, because it is DualQ that inflicts that damage on classic flows. (Due to the unfortunate choice of 15ms for the classic delay target and the unfortunate choice of the weights for the weighted priority scheduler, if we agree on the one order of magnitude goal, these two parameters need to be re-visited and the draft needs to spell out this 1:2 fairness goal explicitly as guidance for implementers of a dual queue design).

Regards
	Sebastian

P.S.: If it appears that I am repeating the same arguments over and over again, this is because sadly the same arguments/objections still apply.

> 
> - Which at least to me seems like a starting point for any conversation.
> 
> Gorry
> 
> On 28/10/2021 15:42, Sebastian Moeller wrote:
>> Hi list,
>> 
>> one question below, prefixed [SM].
>> 
>> 
>>> On Oct 26, 2021, at 01:12, Bob Briscoe <[email-masked]> wrote:
>>> 
>>> Pete,
>>> 
>>> Replying to your points inline [BB] on each draft one at a time, starting with dualq-coupled-16...
>>> 
>>> On 24/08/2021 21:27, Pete Heist wrote:
>>>> # Comments on dualq-coupled-16
>>>> 
>>>> 
>>>>> dualq-coupled Abstract:
>>>>> 
>>>>> Analytical study and implementation testing of the coupled aqm have
>>>>> shown that scalable and classic flows competing under similar
>>>>> conditions run at roughly the same rate.
>>>>> 
>>>> Testing DualPI2 with the defaults showed about a 2x advantage in
>>>> throughput for L4S flows vs non-L4S flows, at the same RTT (see #2
>>>> above). "Roughly the same rate" isn't quantified, but the results seem
>>>> different from that.
>>>> 
>>> [BB] I've changed it to "roughly equivalent", which I think is the briefest way of summarizing loads of results for words in an abstract.
>>> 
>>> 2:1 cannot cause a significantly negative impact [RFC5033] and is well within the "order of magnitude" required in the most recent RFC on this subject [RFC8085] (altho' that was about bulk UDP transfers, I can't see how fairness would depend on which transport protocol is in use).
>> 	[SM] Is there consensus, that a systematic rate-bias of 2:1 is equivalent? I was under the impression that if rates randomly vary in the 2:1 to 1:2 range that would be equivalent, but a robust and reliable advantage of one type over the other would not qualify as equivalent. What am I missing here?
>> 
>> 
>> Regards
>> 	Sebastian
>> 
>>>>> dualq-coupled Abstract:
>>>>> 
>>>>> When tested in a residential broadband setting, DCTCP also achieves
>>>>> sub-millisecond average queuing delay and zero congestion loss under
>>>>> a wide range of mixes of DCTCP and `Classic' broadband Internet
>>>>> traffic, without compromising the performance of the Classic traffic.
>>>>> The solution has low complexity and requires no configuration for the
>>>>> public Internet.
>>>>> 
>>>> Bursty traffic or link layers can raise the limits on low queueing
>>>> delay:
>>>> 
>>>> 
>>>> https://github.com/heistp/l4s-tests/#between-flow-induced-delay
>>> [BB] Gorry asked me to cut down the abstract, so I've removed all the stuff about testing in a residential broadband setting anyway.
>>> See the mail I just sent him for the new wording. However, I can see now that changing it is probably going to lead to further haggling over different wording...
>>> 
>>> Nonetheless, let's move from wording to your technical point itself. Before I was hired by CableLabs to work on Low Latency DOCSIS I wrote a tech report about why bursty effects like this might happen and how to deal with it if they were a problem [SigQ-Dyn]. It's about how sojourn time doesn't measure a burst even once the whole burst is queued. You'll find this is already referenced from the dualq draft. We implemented the alternative metric ('scaled sojourn time'), but didn't use it in the end because we couldn't find a good reason for it being necessary. Since that time, I've thought of a simpler way of implementing an improved variant of the metric, which I wrote into the next rev of the draft recently.
>>> 
>>> But before considering replacing sojourn time we need to bear in mind that there's only one run of one experiment that looks odd. So we need to set ourselves up with a lot more solid evidence. If there is a performance degradation in more typical scenarios, we need to know whether it's big enough to worry about. If it is, the modified metric should improve the performance again.
>>> 
>>> [SigQ-Dyn] Briscoe, B., "Rapid Signalling of Queue Dynamics", Technical Report TR-BB-2017-001 arXiv:1904.07044 [cs.NI], September 2017, <https://arxiv.org/abs/1904.07044>.
>>> 
>>>>> dualq-coupled Section 1.1:
>>>>> 
>>>>> In contrast, dualq coupled aqms addresses the root cause of the
>>>>> latency problem --- they are an enabler for the smooth low latency
>>>>> scalable behaviour of scalable congestion controls, so that every
>>>>> packet in every flow can enjoy very low latency, then there is no
>>>>> need to isolate each flow into a separate queue.
>>>>> 
>>>> I would like to be sure that this is true, not just with bursty traffic
>>>> and link layers, but also cross-flow traffic (tests needed on the
>>>> latter).
>>>> 
>>> [BB] Which part do you mean by 'this'? "...every flow can enjoy..."? or "No need to isolate..."?
>>> 
>>> I've edited the former to "...can potentially enjoy...".
>>> 
>>>> 
>>>> 
>>>>> dualq-coupled Section 1.4:
>>>>> 
>>>>> Thousands of tests have been conducted in a typical fixed residential
>>>>> broadband setting.  experiments used a range of base round trip
>>>>> delays up to 100ms and link rates up to 200 mb/s between the data
>>>>> centre and home network, with varying amounts of background traffic
>>>>> in both queues.
>>>>> 
>>>> It sounds like these tests were performed on a certain type of link
>>>> (residential cable?), in a certain RTT range (below 100ms).
>>>> 
>>> [BB] The draft says "below 100ms". And it gives the ref to the paper with the spec of the tests - it says they were over DSL, then emulated with an ethernet testbed for higher speeds than the DSL could achieve, after comparing the validity of the emulation.
>>> 
>>>> Bursty
>>>> links and higher RTTs should also be tested if L4S is to be used for
>>>> general purpose congestion control (see #3 above). We should end up
>>>> with a system that works well under most all conditions, otherwise it's
>>>> not clear how users and admins will know to change CCAs from one
>>>> activity to the next.
>>>> 
>>> See my posting on the list to Sebastian on 31 Aug 21:
>>> "[tsvwg] tests of Prague utilization with bursty competition in L and C [more burst sources]"
>>> https://mailarchive.ietf.org/arch/msg/tsvwg/LfcOHNupIusPHyjVV4vFWFwS8bI/
>>> 
>>> There's a lot in that conversation, and you'll see some results of it appear in the ecn-l4s-id draft shortly.
>>> At the risk of over-simplifying, two key sentences are:
>>> * "Updated link designs are not going to appear any time soon. In the mean time, the parameters of any flaky links and the L4S target Q delay can be set to levels that trade off between utilization and delay."
>>> * "Whichever technology enables reduced queuing delay variability, the world would want to change to reduce burst delays as well. "
>>> 
>>> 
>>>>> dualq-coupled Section 4.1.3:
>>>>> 
>>>>> Experiments with the DualPI2 AQM (Appendix A) have shown that
>>>>> introducing 'drop on saturation' at 100% L4S marking addresses this
>>>>> problem with unresponsive ECN as well as addressing the saturation
>>>>> problem.  It leaves only a small range of congestion levels where
>>>>> unresponsive traffic gains any advantage from using the ECN
>>>>> capability, and the advantage is hardly detectable [DualQ-Test].
>>>>> 
>>>> I might be misunderstanding the text, but two-flow tests of traffic
>>>> that is unresponsive to CE show that, as with RFC3168 ECN, there is an
>>>> advantage to setting ECT(0) or ECT(1), then not responding to CE,
>>>> that's easy to detect:
>>>> 
>>> [BB] That section is about unresponsive traffic gaining advantage from using the ECN capability.
>>> I think you're talking about gaining advantage from being unresponsive, which is not in doubt.
>>> 
>>> To be clearer, I'll add "... unresponsive traffic gains any advantage from using the ECN capability (relative to being unresponsive without ECN)"
>>> 
>>>> https://sce.dnsmgr.net/results/ect1-2020-05-20-s11-ce-unresponsive/l4s-s11-ce-unresponsive/
>>> [BB] What am I looking at here? Sorry, I'm not familiar with the framing of these tests.
>>> 
>>> ___________
>>> [BB] Whatever, thank you v much for your continued testing, reviewing and comments.
>>> 
>>> Cheers
>>> 
>>> 
>>> Bob
>>> 
>>> --

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 5 Nov 2021 11:35:57 +0000

I thought the discussion was about RFC8085 - sorry for the typo.

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 5 Nov 2021 13:01:24 +0000

Gorry,

On 05/11/2021 11:35, Gorry Fairhurst wrote:
>> [SM] https://www.rfc-editor.org/rfc/rfc8085, "RFC 805: Computer Mail 
>> Meeting Notes" seems a bit of a stretch ;)
>>
>>

[BB] Gorry, you were an author - can you enlighten us on any ML 
conversation that led to this choice of words, which might help 
determine whether it was binary or decimal?

>>
>> But the core of my question was more about whether a systemic bias of 
>> noticeable magnitude (say because L4S flow consistently get higher 
>> throughput) is to be considered differently from a random bias (where 
>> the fairness between flows depends on the exact conditions and the 
>> "winner" is not reliably predictable).

[BB] I don't think the RFCs say anything about that distinction. If they 
did, they would have to say on what timescale the random bias was 
measured over, and that would be far more specific than this subject 
warrants, given it's all finger-in-the-air numbers and vague wording in 
the first place.

It's a fair question though, because the original concept of 
TCP-friendliness was introduced because TFRC took longer to reach 
equality with Reno than Reno did (= temporary bias). Nonetheless, since 
then, the question has shifted to how much more aggressive a different 
CC might be (= systematic bias).

BTW, my position on rate fairness is that it's the wrong metric, but I'm 
going along with this conversation on the basis that some other people 
use it as a metric.

Bob

>>
>>
>> Now, I am also puzzled, if this one order of magnitude fairness is a 
>> BCP, why are we considering RFC-ing DualQ which at short RTTs 
>> robustly and  reliably introduces imbalances on the order of 1:15 (as 
>> in the draft) or 1:10 as in the implementation for Linux, both of 
>> which are clearly >> 1:2, and the argument about the added queueing 
>> delay for the classic flow does not hold water, because it is DualQ 
>> that inflicts that damage on classic flows. (Due to the unfortunate 
>> choice of 15ms for the classic delay target and the unfortunate 
>> choice of the weights for the weighted priority scheduler, if we 
>> agree on the one order of magnitude goal, these two parameters need 
>> to be re-visited and the draft needs to spell out this 1:2 fairness 
>> goal explicitly as guidance for implementers of a dual queue design).
>>
>> Regards
>>     Sebastian
>>
>> P.S.: If it appears that I am repeating the same arguments over and 
>> over again, this is because sadly the same arguments/objections still 
>> apply.
>>
>>
>>
>>>>> [BB] I've changed it to "roughly equivalent", which I think is the 
>>>>> briefest way of summarizing loads of results for words in an 
>>>>> abstract.
>>>>>
>>>>> 2:1 cannot cause a significantly negative impact [RFC5033] and is 
>>>>> well within the "order of magnitude" required in the most recent 
>>>>> RFC on this subject [RFC8085] (altho' that was about bulk UDP 
>>>>> transfers, I can't see how fairness would depend on which 
>>>>> transport protocol is in use).
>>>>     [SM] Is there consensus, that a systematic rate-bias of 2:1 is 
>>>> equivalent? I was under the impression that if rates randomly vary 
>>>> in the 2:1 to 1:2 range that would be equivalent, but a robust and 
>>>> reliable advantage of one type over the other would not qualify as 
>>>> equivalent. What am I missing here?
>>>>
>>>>
>>>> Regards
>>>>     Sebastian
>>>>
>>>>>>> dualq-coupled Abstract:
>>>>>>>
>>>>> [BB] Gorry asked me to cut down the abstract, so I've removed all 
>>>>> the stuff about testing in a residential broadband setting anyway.
>>>>> See the mail I just sent him for the new wording. However, I can 
>>>>> see now that changing it is probably going to lead to further 
>>>>> haggling over different wording...
>>>>>
>>>>> Nonetheless, let's move from wording to your technical point 
>>>>> itself. Before I was hired by CableLabs to work on Low Latency 
>>>>> DOCSIS I wrote a tech report about why bursty effects like this 
>>>>> might happen and how to deal with it if they were a problem 
>>>>> [SigQ-Dyn]. It's about how sojourn time doesn't measure a burst 
>>>>> even once the whole burst is queued. You'll find this is already 
>>>>> referenced from the dualq draft. We implemented the alternative 
>>>>> metric ('scaled sojourn time'), but didn't use it in the end 
>>>>> because we couldn't find a good reason for it being necessary. 
>>>>> Since that time, I've thought of a simpler way of implementing an 
>>>>> improved variant of the metric, which I wrote into the next rev of 
>>>>> the draft recently.
>>>>>
>>>>> But before considering replacing sojourn time we need to bear in 
>>>>> mind that there's only one run of one experiment that looks odd. 
>>>>> So we need to set ourselves up with a lot more solid evidence. If 
>>>>> there is a performance degradation in more typical scenarios, we 
>>>>> need to know whether it's big enough to worry about. If it is, the 
>>>>> modified metric should improve the performance again.
>>>>>
>>>>> [SigQ-Dyn] Briscoe, B., "Rapid Signalling of Queue Dynamics", 
>>>>> Technical Report TR-BB-2017-001 arXiv:1904.07044 [cs.NI], 
>>>>> September 2017, <https://arxiv.org/abs/1904.07044>.
>>>>>
>>>>> [BB] The draft says "below 100ms". And it gives the ref to the 
>>>>> paper with the spec of the tests - it says they were over DSL, 
>>>>> then emulated with an ethernet testbed for higher speeds than the 
>>>>> DSL could achieve, after comparing the validity of the emulation.
>>>>>
>>>>>> Bursty
>>>>> There's a lot in that conversation, and you'll see some results of 
>>>>> it appear in the ecn-l4s-id draft shortly.
>>>>> * "Updated link designs are not going to appear any time soon. In 
>>>>> the mean time, the parameters of any flaky links and the L4S 
>>>>> target Q delay can be set to levels that trade off between 
>>>>> utilization and delay."
>>>>> * "Whichever technology enables reduced queuing delay variability, 
>>>>> the world would want to change to reduce burst delays as well. "
>>>>>
>>>>>
>>>>> To be clearer, I'll add "... unresponsive traffic gains any 
>>>>> advantage from using the ECN capability (relative to being 
>>>>> unresponsive without ECN)"
>>>>>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 7 Nov 2021 16:40:13 +0000

Jake,

Thank you for your new reviews. Pls see [BB] inline...

On 21/08/2021 21:11, Holland, Jake wrote:
> Overall:
> The documents are in much better shape than the last time I reviewed
> them, thanks for all the improvements.
>
> I'm reviewing l4s-id and l4s-arch but didn't have time to get to
> dualq-aqm yet.  But I wanted to make sure these got posted, and
> might not have more time before the deadline.
>
> ----
>
> draft-ietf-tsvwg-ecn-l4s-id-19:
> There's a few major issues.  These really should be fixed before
> sending it to the IESG, IMO, but with these fixed I'd be happy to
> see it shipped.
>
> Major:
>
> 1. This should be proposed standard, not experimental.
>
> [snip]

[BB] See separate thread.

>
> 2. This text from Section 4.3 should be strengthened to recommend
> detection of paths that might be classic-marking L4S traffic (shared
> queue or not):
>
>     o  In uncontrolled environments, monitoring MUST be implemented to
>        support detection of problems with an ECN-capable AQM at the path
>        bottleneck that appears not to support L4S and might be in a
>        shared queue.
>
> Even in fq, the standing queue that will be built by a scalable cc with
> classic marking is contrary to "1-2ms latency at 99th percentile" goal
> of L4S, even leaving aside the harm to colliding flows (though of course
> the harm to colliding flows, including from hash collisions in fq, is
> also another reason to make this change).

[BB] If we make this requirement so that monitoring MUST report all 
Classic ECN AQMs, operators could just have a sea of data that doesn't 
say whether there are any of the single-queue AQMs out there - much more 
likley to be a potential problem than FQ AQMs. The key word in this 
requirement is 'problems'.

Also note that the word 'might' in 'might be in a shared queue' allows 
for the case of a flow-queue that is exhibiting problems at the time it 
is monitored (e.g. during real time monitoring when within a VPN). BTW, 
we created a matrix of cases, and checked all the possible cases when we 
wordsmithed this requirement.

Also, detecting a flow-queue that is ECN marking but appears to exceed 
expected L4S delay is already in the flow's self-interest. So I wouldn't 
worry about trying to legislate over-precisely about this case. For 
instance, of the two OS developers that I have had detailed 
conversations with about this, both are working on the basis that it's 
worth taking sthg like the max of the responses to increasing delay and 
to ECN marks, which will automatically lead to a Classic-like response 
in an FQ-CoDel queue, as well as in other cases like suddenly reduced 
capacity.

> 3. This text from Section 4.3 should be strengthened to require that
> sending L4S traffic in uncontrolled environments does not happen when
> classic marking of L4S traffic is detected for a shared queue, and at
> least recommend that it not happen even for fq:
>
>     o  In uncontrolled environments, monitoring MUST be implemented to
>        support detection of problems with an ECN-capable AQM at the path
>        bottleneck that appears not to support L4S and might be in a
>        shared queue.  Such monitoring SHOULD be applied to live traffic
>        that is using Scalable congestion control.  Alternatively,
>        monitoring need not be applied to live traffic, if monitoring has
>        been arranged to cover the paths that live traffic takes through
>        uncontrolled environments.
>
> The current text requires monitoring, but only gives a single SHOULD for
> live traffic, plus non-normatively permits one alternative.  This allows
> operators to monitor but not cut off (so this requirement as currently
> written would be satisfied by write-only logging for instance, with the
> SHOULD easily dismissable with an "implementation complexity" hand-wave
> while still following the spec).
>
> Suggested alternative, feel free to edit:
>
>     o  In uncontrolled environments, L4S traffic MUST NOT be sent without
>        monitoring to detect marking of L4S traffic by non-L4S bottlenecks.
>        This monitoring can for example be performed on live traffic, or
>        can rely on monitoring that covers the paths live traffic takes
>        through uncontrolled environments.  Where non-L4S bottlenecks are
>        observed marking L4S traffic, L4S sending MUST be disabled if the
>        bottleneck is a shared queue, and SHOULD be disabled if it is FQ.

[BB] Aside: Your middle sentence has (intentionally?) lost the 'SHOULD' 
preference for monitoring live, rather than out of band.

I think the next para in the draft (not given in your quote) with its 
mandatory final sentence is stronger than your alternative.

       The detection function SHOULD be capable of making the congestion
       control adapt its ECN-marking response to coexist safely with
       Classic congestion controls such as standard Reno [RFC5681  <https://datatracker.ietf.org/doc/html/rfc5681>], as
       required by [RFC5033  <https://datatracker.ietf.org/doc/html/rfc5033>].  Alternatively, if adaptation is not
       implemented and problems with such an AQM are detected, the
       scalable congestion control MUST be replaced by a Classic
       congestion control.

The differences are:
* Replacement is a 'MUST' not a 'SHOULD', but conditional on 'problems'
* It doesn't distinguish between shared queue and FQ.

We didn't think it was reasonable to give the sender a requirement that 
depends on what type of queue it encounters, because the sender doesn't 
know the type for sure. That's why we made this conditional on 
'problems' instead. The last sentence of this passage from the 
referenced Appendix A.1.5 explains why:

    CDN servers placed within an access ISP's
    network can be considered as a single controlled environment, but any
    onward networks served by the access network, including all the
    attached customer networks, would be unlikely to fall under the same
    degree of coordinated control.  Monitoring is expressed as a 'MUST'
    for these uncontrolled segments of paths (e.g.  beyond the access ISP
    in a home network), because there is a possibility that there might
    be a shared queue Classic ECN AQM in that segment.  Nonetheless, the
    intent is to only require occasional monitoring of these uncontrolled
    regions, and not to burden CDN operators if monitoring never uncovers
    any potential problems, given it is anyway in the CDN's own interests
    not to degrade the service of its own customers.

In summary I think your suggestion of conditioning on the type of queue 
is likely to be infeasible and therefore prone to being ignored.
This is not easy but, on balance, I don't think I've seen anything here 
that is tighter than the existing wording.

>
> 4. Although l4s-arch claims that l4s-id satisfies the RFC 4774
> requirements, it's hard to tell whether it does so.  Specifically:
>
> 4.a. From section 7 of RFC 4774:
>     Specifications of alternate ECN semantics must clearly state how they
>     address the issues raised in this document, particularly the issues
>     discussed in Section 2.
>
> I don't see how issues 2 and 3 from section 2 are covered in l4s-id.
>  From section 4 of RFC 4774:
>     (2) How does the possible presence of old routers affect the
>         performance of the alternate-ECN connections?
>
>     (3) How does the possible presence of old routers affect the
>         coexistence of the alternate-ECN traffic with competing traffic
>         on the path?
>
>     When alternate semantics are defined for the ECN field, it is
>     necessary to ensure that there are no problems caused by old routers
>     along the path that don't understand the alternate ECN semantics.
>
> 4.b. Section 4 goes on to describe 3 options for how alternate ECN
> semantics should be treated.  I don't see a claim in the L4S docs
> specifying which of the 3 options the L4S spec for alternate ECN semantics
> matches, but it implicitly appears to be trying to say it's option 3
> (unsafe) and asserting that the detection and adaptive response satisfies
> the requirement for isolation on option 3, I think?
>
> Maybe there are sections in l4s-id that intend to cover these points,
> and there just needs to be text listing what they are, but I don't think
> the link is obvious enough to satisfy the "clearly state" requirement
> from RFC 4774.  So it would be very helpful to add a list of references
> to sections that are intended to address these RFC 4774 requirements.

[BB] After guidance from the chairs, the co-authors have been working 
offlist on a fairly lengthy new subsection in ecn-l4s-id about where the 
RFC4774 requirements are and are not satisfied, and how that is to be 
dealt with. Rather than pasting it all here (it's long) you should see 
it later today once the secretariat allow the draft through (or 
otherwise when the servers re-open in the morning). And I'm sure the 
chairs will give everyone time to read it over, and review it.

>
>
> Nits:
> - the list in section 7.1 has a weird formatting problem for the sub-
>    list:
>
>     o  Did use of L4S over the Internet result in improvements to the
>        following metrics:
>
>     o
>
>        *  queue delay (mean and 99th percentile) under various loads;

[BB] Thx - (had already noticed this one myself as well).

Continued...

>
> ----
>
> draft-ietf-tsvwg-l4s-arch-10:
> Summary: Almost ready
>
> +1 to Gorry's comments here, especially regarding the use of "all traffic":
> https://mailarchive.ietf.org/arch/msg/tsvwg/vMMsQpXs65lk1E7NpV5RlmpyqdI/

[BB] Accepted - see response to Gorry/Alex.

>
> Minor:
> 1. l4s-arch section 1:
>     It has been demonstrated that, once access network bit rates reach
>     levels now common in the developed world, increasing capacity offers
>     diminishing returns if latency (delay) is not addressed.
> - This needs a reference.

[BB] I've added:

   [Dukkipati15]
               Dukkipati, N. and N. McKeown, "Why Flow-Completion Time is
               the Right Metric for Congestion Control", ACM CCR
               36(1):59--62, January 2006,
               <https://dl.acm.org/doi/10.1145/1111322.1111336>.
   [Rajiullah15]
               Rajiullah, M., "Towards a Low Latency Internet:
               Understanding and Solutions", Masters Thesis; Karlstad
               Uni, Dept of Maths & CS 2015:41, 2015, <https://www.diva-
               portal.org/smash/get/diva2:846109/FULLTEXT01.pdf>.

> Editorial:
> 1. l4s-arch 4.2 section a:
> - This is a confusing wall of text.  I think it would be better to
>    give a much briefer summary here with a reference.  Exposition
>    like "the obvious part" and "the less obvious part" are a minus
>    here--I don't think the obviousness claims made here generalize
>    well.

[BB] When it only referenced the DualQ draft and only gave a short 
summary, we were asked to summarize more fully how the DualQ works here 
(sigh!).

So we've kept the explanation here, but de-confused and de-walled it a bit:
* Removed the 'obvious/non-obvious' wording.
* Within the largest block: Introduced sub-bullets for the two parts of 
the semi-permeable membrane, each starting "Latency isolation:" and 
"Bandwidth pooling:". Then continued the part about the scheduler with 2 
further bullets, again starting each with "for latency isolation" and 
"for bandwidth pooling".
* Cut out some redundancy.
* I considered shifting out the last para of the DualQ bullet, which is 
about the DualQ document, but I decided there was no better place for it.

It's not brilliant, but I think it's better.

>
> 2. All the uses of underlining for emphasis are a minus.  The
>    places where it seems necessary or useful are a good hint that
>    the text on its own is not adequately capturing the intended
>    meaning.
>    Leaning on this kind of toned emphasis makes assumptions about
>    connotations that don't hold very well even for native English
>    speakers and break down entirely for non-native speakers, so they
>    are generally out of place in a technical document that will need
>    to be correctly interpreted by an international audience with many
>    non-native speakers, IMO.

[BB] I agree, use of emphasis is excessive. I've taken out /some/ but 
not /all/ ;)

The ones left are:
* "on /average/" (because the comparison with the other P99 is 3 lines 
of dense numbers away)
* "without the /need/ for per-flow operations" (to help the reader not 
to just run over the word without thinking)
* "without /requiring/ inspection" (---ditto---)

Hopefully, this is now down to a non-irritating level for those who 
don't like it, and now conforms to the guidance from the only source I 
could quickly find on international technical writing style:

    "Do not use italics for

      * mere emphasis. (Italics are acceptable if emphasis might
        otherwise be lost; in general, however, use syntax to provide
        emphasis.)"

American Psychological Association (APA, 2009, pp. 104-106)
via 
https://writing.stackexchange.com/questions/10750/when-should-i-use-italics-in-scientific-writing

(BTW, underscores in ASCII can mean either italics or underlining. In 
the HTML rendering, these words come out in italics - which was the 
intention)

>
> ----
>
> I'm going to be too pressed for time to do a more detailed review,
> but I wanted to get the above comments in.
>
> As a final aside, I'd like to see this happen.  The only goal I'm
> pursuing at this point wrt this work is avoiding preventable harm
> from pushing this out in a way that's likely to cause confusion
> when and if problems are encountered in production.  (In particular,
> I have given up on efforts to improve the signaling design, since
> the authors have rejected all such suggestions and this work needs
> to get moved out of the wg one way or another.)

[BB] On signalling design, I'd like to point out that it's not just the 
authors - the chairs have judged that the WG wants to go this way.

Whatever, thank you - for your reviews, and for your continuing 
perseverance and diligence.

Cheers

Bob

>
> Best,
> Jake
>
>
> On 07-29, 9:18 AM, "Wesley Eddy" <[email-masked]> wrote:
>
> - Architecture: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>
> - DualQ:
> - ECN ID: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 7 Nov 2021 17:27:12 +0000

Pete,

Thank you again for all the work you've put in, particularly the 
experiments, but also putting in the time to review all theree drafts.
Replying to your points inline [BB] on each draft one at a time, this 
time on l4s-arch-10...

On 24/08/2021 21:27, Pete Heist wrote:
> # Comments on l4s-arch-10
>
>> l4s-arch Abstract:
>>
>> These mechanisms aim to ensure that the latency and throughput
>> performance using an L4S-compliant congestion controller is usually
>> much better (and never worse) than the performance would have been
>> using a 'Classic' congestion controller, and that competing flows
>> continuing to use 'Classic' controllers are typically not impacted by
>> the presence of L4S.
> The goal "and never worse" was not reached, as demonstrated by the
> throughput performance with bursty traffic and links, relative to the
> performance for conventional CCAs and AQMs (see #3 above).
>
> Classic CCAs may be impacted by the presence of L4S (see #1 and #2
> above).

[BB] These are the aims for the L4S experiment. We shall see whether 
they can be met if this experiment is allowed to proceed on the Internet.
But I think you're right that 'never' is rather too ambitious, so I've 
toned that down to 'rarely'.

The list of open questions in my editor's copy of ecn-l4s-id now 
includes the following (the first was already there, but I've added a 
ref to your experiments):

    *  How much do bursty links in the Internet affect L4S performance
       (see "Underutilization with Bursty Links" in [Heist21]), and how
       much limitation of burstiness from upstream links was needed and/or
       was realized - both at senders and at links, especially radio
       links (see bullet #7 in Section 4.3 and Section 5.6)?

    *  Is the initial experiment with mis-marked bursty traffic at high
       RTT (see "Underutilization with Bursty Traffic" in [Heist21])
       indicative of similar problems at lower RTTs and, if so, how
       effective is the suggested remedy in Appendix A.1 of
       [I-D.ietf-tsvwg-aqm-dualq-coupled] (or possible other remedies)?

On bursty upstream links, yes it's predictable that the reference 
implementation doesn't meet this goal when the bottleneck link is 
configured with a shallow threshold and there is a bursty upstream link. 
This general issue is well-known - I doubt it is specific to L4S. It's 
one of the challenges of trying to reduce queuing delay at one link at a 
time, while not being able to improve all the other links at the same time.
     (A personal aim of mine is to turn the 'potential worst-case' 
nature of this problem into a 'current worst-case' problem. I mean that, 
at the moment you have to relax the target delay of a AQM to cater for 
the /potential/ worst-case upstream link burstiness. I would like to be 
able to sense the /actual/ worst case upstream link and _very slowly_ 
adapt the target configuration to it. But it's not so simple.)

Nonetheless, there's a lot that can be done towards system-wide 
improvements even without actors having to coordinate. As I said in my 
response to your similar comment on the DualQ draft, see my posting on 
the list to Sebastian on 31 Aug 21: "[tsvwg] tests of Prague utilization 
with bursty competition in L and C [more burst sources]"
https://mailarchive.ietf.org/arch/msg/tsvwg/LfcOHNupIusPHyjVV4vFWFwS8bI/
     In that email there is sthg like the text that will appear in a new 
section 5.6 in ecn-l4s-id about upstream bursty links.
     For instance, dealing with the burstiness of all WiFi interfaces on 
an Access Point at the same time as deploying an L4S AQM on the upstream 
WAN interface.

>> l4s-arch Section 1:
>>
>> Therefore, the goal is an Internet service with very Low queueing
>> Latency, very Low Loss and Scalable throughput (L4S).  Very low
>> queuing latency means less than 1 millisecond (ms) on average and
>> less than about 2 ms at the 99th percentile.
> The latency goal only appears attainable on certain link layers at
> present, under certain conditions.

[BB] Recall that these are goals for the L4S Experiments ahead of us 
(not just behind us). I can't see that there's any evidence /yet/ to say 
that this latency goalhas not been attainable on each link technology, 
once an attempt to support L4S on it reached some degree of maturity.

I suspect this goal will be challenging for radio links (which includes 
satellite), but I'd rather revise the goal /after/ those working on it 
have had time for a decent crack at it, not before.

BTW, note that, where the draft gives actual results, it specifies the 
link technology used.

> Between-flow induced delay from
> bursty links, such as competing WiFi traffic, can exceed it:
>

[BB] Via this link you make the point that "this is the Internet as it 
is today". But the point above about improving multiple links at once 
apply here too. Not just the point about the links in and out of a home 
gateway, but also the wider operational abilities of ISPs to tune links 
for customers taking the L4S service.

In the light of this experiment, you also make the point that "there is 
a discussion to be had around whether reliably low queueing delays will 
ultimately require flow separation". (BTW, bear in mind that L4S 
includes FQ.) But I think 'ultimately' is the wrong idea here. Perhaps 
"/initially/ require flow separation" would have been a better 
conclusion from this experiment. Users are going to want WiFi burstiness 
to reduce, so I don't think there's an argument for 'ultimately' here. 
But FQ deployments of L4S might ease some of the coordination challenges 
in the short term.

However, it's likely or at least possible that the root cause of this 
'between-flows' case is the same as the 'bursty traffic' case - if 
markings are biased towards the smoother flow they will also be biased 
away from the more bursty flow, which would fool the bursty flow into 
responding less than it would like, leading to increased queue delay 
spikes. If that is the cause, then the proposed remedy for the 'bursty 
traffic' case could also mitigate this 'between-flows' case. So let us 
look into this further. Because FQ-CoDel uses the same sojourn-based 
marking. So in cases like VPNs, FQ won't help if there is an underlying 
problem with immediate marking of bursty traffic based on sojourn time.

Returning to the job of editing drafts, again, I don't think the 
criticism of the text here is warranted. No-one expects the goals of the 
L4S Experiment to have already been attained everywhere - we haven't 
even got the experiments started over the real Internet yet.

>> l4s-arch Section 2(1):
>>
>> The Dual Queue Coupled AQM was developed as a minimal complexity
>> solution to this problem.  It acts like a 'semi-permeable' membrane
>> that partitions latency but not bandwidth.  As such, the two queues
>> are for transition from Classic to L4S behaviour, not bandwidth
>> prioritization.
> We have seen a bias in favor of L4S over non-L4S flows (see #2 above).

[BB] See response to the same point under the DualQ. I can't see any 
text making any claim about this here though, so nothing to change.

>
>
>> l4s-arch Section 4.1(b):
>>
>> [I-D.ietf-tsvwg-ecn-l4s-id] recommends ECT(1) is used as the
>> identifier to classify L4S packets into a separate treatment from
>> Classic packets.  This satisfies the requirements for identifying an
>> alternative ECN treatment in [RFC4774].
> It's not clear to me that the requirements are satisfied, given the
> text in RFC4774 below, in combination with the test results referred to
> in #1 above.
>
>    > RFC4774 Section 4:
>    >
>    > necessary to ensure that there are no problems caused by old
>    > routers along the path that don't understand the alternate ECN
>    > semantics.

[BB] I know this sounds picky, but this sentence was referring to the 
RFC4774 requirement "for identifying an alternative ECN treatment", not 
the one you refer to here. Whatever, I have done 
s/requirements/requirement/ which might have caused the misunderstanding.

To you point about the RFC4774 §4 requirement you quote, after guidance 
from the chairs, the co-authors have been working offlist on a fairly 
lengthy new subsection in ecn-l4s-id about where the RFC4774 
requirements are and are not satisfied, and how that is to be dealt 
with. You should see that before the meeting on Monday. And I'm sure the 
chairs will give everyone time to read it over, and review it.

>> l4s-arch Section 5.1:
>>
>> The consequent smaller amplitude sawteeth fit between an empty queue
>> and a very shallow marking threshold (~1 ms in the public Internet),
>> so queue delay variation can be very low, without risk of under-
>> utilization.
> Underutilization can be seen with bursty traffic and bursty links
> (see #3 above).

[BB] Point taken, and see previous responses about this being a problem 
for any approach that reduces queuing delay one link at a time. And 
about it being too early to tell how much the bursty links effect can be 
mitigated as the L4S experiment proceeds.

I don't think the text here is incorrect - it's explaining about the 
benefit of small sawteeth, not claiming anything. And it only says 
'/can/ be very low' anyway.

>
>
>> l4s-arch Section 5.2:
>>
>> In contrast, because L4S is for all traffic, it needs none of the
>> management baggage (traffic policing, traffic contracts) associated
>> with favouring some packets over others.  This baggage has probably
>> held Diffserv back from widespread end-to-end deployment.
> Comments on DiffServ seem like they'd be better made elsewhere.

[BB] We're aware that we're not doing ourselves any favours by pointing 
out weaknesses in other technologies that might be close to the hearts 
of people involved in the review process. But we also have to justify 
why the IETF should assign a scarce IP codepoint, which means we have to 
spell out what L4S adds over other approaches (hence the whole section 
this paragraph comes from). Nonetheless, I will turn the last sentence 
round into:

"This lack of management baggage ought to give L4S a better chance of 
end-to-end deployment."

>
>> l4s-arch Section 5.2:
>>
>> The L4S approach resolves this tension by ensuring hosts can minimize
>> the size of their sawteeth without appearing so aggressive to Classic
>> flows that they starve them.
> With current middleboxes that don't understand ECT(1), that can happen
> (see #1 above).

[BB] You're right, this sentence has so far been missed, and needs 
fixing. In fact, it's off the point for this section about Classic AQMs, 
and there is a point that ought to have been said about Classic AQMs, 
but hasn't. Here's proposed replacement text:

"The L4S approach resolves this tension between delay and utilization by 
enabling hosts to minimize the amplitude of their sawteeth. A 
single-queue Classic AQM is not sufficient to allow hosts to use small 
sawteeth for two reasons: i) smaller sawteeth would not get lower delay 
in an AQM designed for larger amplitude Classic sawteeth, because a 
queue can only have one length at a time; and ii) much smaller sawteeth 
implies much more frequent sawteeth, so L4S flows would drive a Classic 
AQM into a high level of ECN-marking, which would appear as heavy 
congestion to Classic flows, which in turn would greatly reduce their 
rate as a result (see Section 6.4.4)."

>> l4s-arch Section 5.2:
>>
>> Per-flow implementations still need to have support for scalable
>> congestion control added, which has already been done in FQ-CoDel
>> (see Sec.5.2.7 of [RFC8290]).
> This is a bit misleading as these are not L4S AQMs but just have a
> ce_threshold parameter. L4S support would require treating ECT(1)
> traffic differently from ECT(0).

[BB] You were right when you wrote this review, but the situation has 
changed now anyway. This sentence now says:

"Per-flow implementations need to have support for scalable congestion 
control added, which has already been done for FQ-CoDel in Linux (see 
Sec.5.2.7 of [RFC8290] and [FQ_CoDel_Thresh]."

[FQ_CoDel_Thresh] points to Toke's recent patch: 
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=dfcb63ce1de6b10b

>> l4s-arch Section 6.4.2(1):
>>
>> If no AQM at all was previously deployed for the downstream access,
>> an L4S AQM greatly improves the Classic service (as well as adding
>> the L4S service).  If an AQM was already deployed, the Classic
>> service will be unchanged (and L4S will add an improvement on top).
> If a Codel AQM was already deployed, then switching to PIE with the
> default targets used for the C queue may increase average latency,
> relative to the latency with Codel's defaults.

[BB] But this sentence doesn't give any implication that a Classic AQM 
would switch to a different AQM as well as adding L4S.

(And I'm not going to get into the whole discussion of how the two 
targets don't relate to the outcome in anything like the same way, or 
whether the choice of default for each is sensible, or the whole 
discussion of how an average is not a useful characterization of queue 
delay, or...).

Whatever, there's nothing to change here AFAICT.

>> l4s-arch Section 6.4.2(2):
>>
>> In this stage, the name 'TCP Prague'
>> [I-D.briscoe-iccrg-prague-congestion-control] is used to represent a
>> variant of DCTCP that is safe to use in a production Internet
>> environment.
> It's only safe to use in production when all existing middleboxes
> understand ECT(1) to be the L4S ID, which at present not all do (see #1
> above).

[BB] I'm happy to change 'safe to use' to 'designed to be used', given 
'safe' means different things to different people and depends on other 
aspects like whether its monitoring for a Classic ECN AQM, whether 
Classic AQM fall-back has been improved sufficiently to be on by 
default, whether the loss response is robust, whether it negotiates 
feedback, etc. Also most of the differences between Prague and DCTCP now 
are much wider than safety - stuff like corrected  averaging, smoother 
pacing and TSO, less jumpiness due to no suppression of additive 
increase, RTT-independence.

That's it for now.
Thank you for the full review, and much more importantly for all your 
experiments.

Cheers

Bob

> On Thu, 2021-07-29 at 12:18 -0400, Wesley Eddy wrote:
>> - Architecture:
>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>>
>> - DualQ:

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 7 Nov 2021 18:47:26 +0000

Pete,

Here's how I've dealt with your reviews of the 3rd of the 3 L4S drafts: 
draft-ietf-tsvwg-ecn-l4s-id-19
See [BB]

On 24/08/2021 21:27, Pete Heist wrote:
> # Comments on l4s-id-19
>
>> l4s-id Abstract:
>>
>> L4S uses 'scalable' congestion control, which induces much more
>> frequent control signals from the network and it responds to them
>> with much more fine-grained adjustments, so that very low (typically
>> sub-millisecond on average) and consistently low queuing delay
>> becomes possible for L4S traffic without compromising link
>> utilization.  Thus even capacity-seeking (TCP-like) traffic can have
>> high bandwidth and very low delay at the same time, even during
>> periods of high traffic load.
> Testing suggests lower link utilization with bursty links and/or bursty
> competing traffic (see #3 above).

[BB] See text inserted based on that promised to Sebastian.

>
>> l4s-id Introduction:
>>
>> The DualQ Coupled AQM, which is defined in a complementary
>> experimental specification [I-D.ietf-tsvwg-aqm-dualq-coupled], is an
>> AQM framework that enables scalable congestion controls derived from
>> DCTCP to co-exist with existing traffic, each getting roughly the
>> same flow rate when they compete under similar conditions.

[BB] See response to same point in DualQ review.

>
>
>> l4s-id Section 5.1:
>>
>> In case unforeseen problems arise with the L4S experiment, it MUST be
>> possible to configure an L4S implementation to disable the L4S
>> treatment.  Once disabled, all packets of all ECN codepoints will
>> receive Classic treatment and ECT(1) packets MUST be treated as if
>> they were {ToDo: Not-ECT / ECT(0) ?}.
> This ToDo needs completion.

[BB] Yes, sry about that.
Done after discussion on the list (selected Not-ECT)

>
>
>> l4s-id Section 5.4.1.2:
>>
>> An operator that excludes traffic carrying the L4S identifier from
>> L4S treatment MUST NOT treat such traffic as if it carries the ECT(0)
>> codepoint, which could confuse the sender.
> It might not directly relate to the section, but in order to comply
> with this MUST NOT, the operator would also have to make sure that all
> existing RFC3168 middleboxes are undeployed (see #1 above).

[BB] Ah, I can see the confusion here. We shouldn't have said "An 
operator ... MUST NOT treat...". We should have said:

"A network node that supports L4S but excludes certain traffic carrying 
the L4S identifier from L4S treatment MUST NOT treat..."

I've also changed the subject of the other normative requirements in 
this section from 'operator' to 'network node'.

>> l4s-id Section 5.5:
>>
>> As well as senders needing to limit packet bursts (Section 4.3),
>> links need to limit the degree of burstiness they introduce.  ...  as
>> suggested in Section 4.3 it would not seem necessary to limit bursts
>> lower than roughly 10% of the minimum base RTT expected in the
>> typical deployment scenario (e.g. 250 us burst duration for links
>> within the public Internet).
> 250 us is about 1 packet at 50 Mbps, with a 1500 byte MTU. WiFi
> aggregates can reach 4 ms. Ubiquiti's fixed framing modes can be either
> 5ms or 10ms.  Other possible sources of bursts in DSL:
> https://mailarchive.ietf.org/arch/msg/tsvwg/s3zYnEhnTnncMW2fIKwux753rok/
>
> At least with today's link technologies, 250 us seems like a difficult
> burst duration to stay under. I think we at least want to know more
> about different link types that are affected by this and what the
> affects are on latency and utilization. See utilization tests in #3
> above, and:

[BB] Point taken, but we intended this to set a challenging expectation 
- to see what link designers can achieve. The figures you quote are what 
happens when link designers are allowed to solely focus on bandwidth. 
(BTW, this is not intended to apply to anomalous bursts such as the DSL 
ones Sebastian and I were talking about in the thread you quote - this 
is about routine frame aggregation).

Nonetheless, there is certainly room to wriggle here - the words you 
quote above state this as the floor of what might be expected, not the 
ceiling. Also, for the benefit of readers on the list, the '...' in the 
above quote contains:
"This document does not presume to even give guidance on an appropriate 
target for such burst delay until there is more industry experience of 
L4S. "

Nonetheless, only new equipment (having been modified for L4S) will be 
having to comply with this requirement, so we thought it's not 
completely out of the question for high link rates. For instance, even 
this floor would allow a burst of 10x1500B packets on a 480Mb/s link.

250 μs is the figure we've been using for the sender's TSO. See:
https://datatracker.ietf.org/doc/html/draft-briscoe-iccrg-prague-congestion-control-00#section-2.5.2
Obviously if the rate is slower than 48Mb/s we're not expecting bursts 
to be smaller than one packet.

[BB] That's it. Thank you for all three reviews.
I'll be posting a new rev of ecn-l4s-id later today, once I've been 
through Gorry's review (and some other bits and pieces).

I'm sure we'll be talking about the tests you highlighted in the start 
of your email, but I've had to focus on the specific comments about the 
drafts first.

Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 7 Nov 2021 21:24:01 +0000

Gorry,

Thanks for picking up this editorial points.
Assume they are accepted unless there's a [BB] comment inline...

Note that there's one that I've highlighted as a normative change, not 
just a nit.

On 13/08/2021 12:11, Gorry Fairhurst wrote:
> I have reviewed draft-ietf-tsvwg-ecn-l4s-id-19 in WGLC and have some 
> comments that I expect are editorial.
>
> (I seperately sent a couple of WGLC issues concerning this draft.)
>
> Best wishes,
>
> Gorry
>
> (as individual)
> ---
> NiT: Terminology could be cross-referenced.
> - I’ve just tried to compare the difference between the terminology in 
> L4S arch and L4S ID, and I found them to be identical, with the 
> exception that L4S arch also defines “site”.
>
> So:
> (a) Could we add “site” also as a definition to L4S-ID, the term is 
> also used in this ID.
> (b) Could you consider adding a one line statement to L4S arch that 
> says the two sets of definitions are identical, so that a reader 
> doesn’t need to check them when they read each document. I expect that 
> L4S-ID as an EXP Spec provides the normative text.

[BB] Good idea (I just posted l4s-arch, but I've added this to the 
editor's copy). And I've put an x-ref in both. As you suggest, I've said 
ecn-l4s-id the master, in case of any accidental differences.

> ---
> NIT: In Sect 1.1:
> - Can you expand FQ or provide a reference?
> ----
> NIT: I think this:
> “in the IP headers even in
>       one direction of a TCP connection will imply that both ends must
>       be supporting accurate ECN feedback.”
> ... Could perhaps be clearer as:
>      “in the IP headers even in
>       one direction of a TCP connection will imply that both ends
>       support accurate ECN feedback.”
> ---
> NIT: I think this:
> “in which AQMs filter out
>    variations before signalling congestion.”
> ... Could perhaps be clearer as (removing “out”):
> “in which an AQM filters
>    variations (i.e. smooths over time) before signalling congestion.”
> ---
> NiT:
> In 5.4.1.1. : “In a typical case for”... followed by “In this case it”.
> - To me these seem like the same case, should therefore these both be 
> in one single paragraph, rather than two?
> ---
> NiT:
> “ To include additional traffic with L4S, a network element only reads
>    identifiers such as those itemized above.  It MUST NOT alter these
>    non-ECN identifiers, so that they survive for any potential use later
>    on the network path.”
>
> - Actually I think this MUST statemen t only applies to the L4S 
> processing within the device, and I think it doesn’t mean L4S is 
> unable to work across a policy enforcement point or some other device 
> that otherwise modifies the other packet headers.
>
> .... so is this better as:
>
> “ To include additional traffic with L4S, a network element only reads
>    identifiers such as those itemized above.  The L4S forwarding
>   MUST NOT alter these
>    non-ECN identifiers, so that they survive for any potential use later
>    on the network path.”

[BB] Note that this is not a nit - it is a correction to a normative 
statement.
Also, there is no definition of what the extent of an 'L4S forwarding 
[element]' is. Classification for L4S might be much earlier in the path. 
How about

"The process of including additional traffic with L4S only involves 
reading identifiers such as those itemized above. It MUST NOT alter 
these non-ECN identifiers, so that they survive for any potential use 
later on the network path."

This is actually rather meaningless, because if a network element wants 
to alter any of the identifiers, it doesn't have to be part of "the 
process of including additional traffic with L4S". So the other 
alternative would be to remove any normative text from this para, and 
just say "The process ... wouldn't normally be expected to alter these 
identifiers." But I've left it with the MUST NOT, so as not to radically 
alter the draft.

> ----
> NiT: To avoid future discussion of whether this is ever used in other 
> cases, it might be better not to assert the deployment extent
> “But they cannot
>    easily satisfy this loss recovery requirement.  However, it is
>    believed they do not need to, because such implementations are
>    believed to solely exist in controlled environments, where the
>    network technology keeps reordering extremely low anyway.”
>
> .... so is this better as:
>
> “But they cannot
>    easily satisfy this loss recovery requirement.  When such 
> implementations are
>    used in controlled environments, the network technology can keep 
> reordering extremely low.”

[BB] This loses the originally intended sense about "not needing to". 
I'd rather just leave it as it is, because the requirement to do loss 
recovery in time units is now only a SHOULD anyway. So it's not only 
controlled environments where exceptions are allowed.

>
> ---
>
> NiT: Would it be OK to tweak this a little?
> “Setting ECT in Control Packets and Retransmissions
>
>    Description: This item concerns TCP and its derivatives (e.g. SCTP)
>    as well as RTP/RTCP [RFC6679].  The original specification of ECN for
>    TCP precluded the use of ECN on control packets and retransmissions.
>    To improve performance, scalable transport protocols ought to enable
>    ECN at the IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs,
>    etc.) and in retransmitted packets.  The same is true for derivatives
>    of TCP, e.g. SCTP.”
> ... I think this might be better related to transports in general, and 
> rephrase the SCTP text - there is no IETF spec for ECN in SCTP (yet). 
> I suggest something like:
> “Setting ECT in Control Packets and Retransmissions
>
>    Description: This item concerns TCP and other transport protocols 
> (e.g. SCTP)
>    as well as RTP/RTCP [RFC6679].  The original specification of ECN for
>    TCP precluded the use of ECN on control packets and retransmissions.
>    To improve performance, scalable transport protocols ought to enable
>    ECN at the IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs,
>    etc.) and in retransmitted packets.  The same is true for other
>
>    transports, e.g. QUIC, SCTP.
>

[BB] I'm happy to say 'other transports' instead of 'derivatives of 
TCP', but I'd rather not include QUIC in the list of examples, because 
that could imply that QUIC needs to be changed, when it doesn't.

Instead, I've added RTCP to the list, and moved the sentence about RTCP 
have a similar problem to the earlier problem part of the paragraph. As 
follows:

Description: This item concerns TCP and its derivatives (e.g. SCTP) as 
well as RTP/RTCP [RFC6679]. The original specification of ECN for TCP 
precluded the use of ECN on control packets and retransmissions. 
Similarly [RFC6679] precludes the use of ECT on RTCP datagrams, in case 
the path changes after it has been checked for ECN traversal. To improve 
performance, scalable transport protocols ought to enable ECN at the IP 
layer in TCP control packets (SYN, SYN-ACK, pure ACKs, etc.) and in 
retransmitted packets. The same is true for other transports, e.g. SCTP, 
RTCP.

> ----
>
> NiT: - Add Reference?
> “TCP Cubic was
>    designed to solve this problem”
> - please site the TCPM work item: draft-ietf-tcpm-rfc8312bis-03

[BB] I've cited RFC8312 many times in this draft already, so I'd prefer 
to just repeat that. if the 'bis' get to through RFC-editor first we can 
change it, but this sentence is about what Cubic has always done right 
from the start.

Thank you yet again, for you usual careful review. Next I'll address 
your technical points on ecn-l4s-id.

Cheers

Bob

> =====
>> -----

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 7 Nov 2021 22:02:24 +0000

Gorry,

Thank you for this these points. See [BB] inline...

On 13/08/2021 12:11, Gorry Fairhurst wrote:
>
> I have reviewed draft-ietf-tsvwg-ecn-l4s-id-19 in WGLC.
>
> I think this draft provides methods that address an important need - 
> to provide techniques to control end to end latency. I provided a 
> detailed review of a recent previous version of this ID, and the 
> issues I found then have been addressed. Thanks editors. I have 
> checked this version and found the three additional issues below.
>
> I therefore would support *as an individual* progression of this draft 
> for publication as EXP.
>
> Best wishes,
>
> Gorry
> (as individual in tsvwg)
>
> P.S. I also have a small number of editorial NiTs that I will send 
> separately to the list.
> ---
>
> ISSUE 1: To-Do
>   “Once disabled, all packets of all ECN codepoints will
>    they were {ToDo: Not-ECT / ECT(0) ?}.”
> - This RFC-2119 statement is incomplete.
> - This issue needs to be discussed.
>

[BB] It has been discussed on the list, and Not-ECT was selected.

>
> ---
> ISSUE 2: Appendix A.
> This appendix is informative, and as such I think we should be careful 
> to not imply this as expressing requirements.
>
> However, the current text contains several lines mentioning 
> “requirements”, including the title!
>

[BB] You're right. This appendix was copied over from the original 
problem statement when the normative requirements were written in the 
body, but without being change to clarify everywhere that it is 
rationale for the requirements. And since I've sometimes been sloppy...

>   - is it possible to express this differently and avoid any ambiguity 
> as the document receives more review?
> I could suggest a few small changes which would seem to me resolve 
> this issue:
> The title likely needs to be renamed, because it creates most of the 
> problem: “The 'Prague L4S Recommendations’ or “The 'Prague L4S Design 
> Recommendations’, or similar.
>

[BB] Titles:
A. Rationale for the 'Prague L4S Requirements'
A.1 Rationale for the Requirements for Scalable Transport Protocols

> Also these small number of specific lines seem to need careful 
> rewording, e.g.:
> /safety improvements (requirements)/safety improvements ()/
> /implementation of the requirements./implementation of these 
> recommendations./
> /Requirements for Scalable Transport Protocols/Features for Scalable 
> Transport Protocols/
> /The requirement is written/This is written/
> /Similarly to the requirement in Appendix A.1.4 this 
> requirement/Similarly to Appendix A.1.4, this/
> /Having said all the above, the approach recommended in the 
> requirements is to monitor/Having said all the above, the approach 
> recommended is to monitor/
>

[BB] I've searched every occurrence of 'require-' and made sure it 
either refers to the normative requirement in the body, or it doesn't 
say require.
I should post the draft soon, so you'll be able to check the diff.

> ---
>
> ISSUE 3: References
> -Please confirm the intended status of:
> [I-D.briscoe-docsis-q-protection]
>

[BB] See previous response to the same question in your other review, 
which you confirmed was satisfactory.

Thank you again.

Cheers

Bob

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 8 Nov 2021 14:52:56 +0000

Hi Bob,

Thanks for your response. For the most part I'm satisfied, but with
regard to the responses to points #2 and #3, I'll top-post my remaining
problem and ask that you suggest something appropriate to address it.

The current text, as far as I can tell, does not *require* any attempt
to detect problems due to "legacy queue" issues (only "SHOULD" language
here, I think?), and then makes the safety response of failing over to
classic CC conditional on "detecting problems".  Certainly if there is
a "MUST" or "SHALL" requirement, it's hard for me to nail down exactly
where.

As has been discussed, without some quite complex detection efforts,
detection of problems is a major challenge with this technology, and
thus the current text overall seems too weak to me on this point.

IMO, the docs need to make it clear that sending with this technology
comes with a mandatory duty to look for and respond to likely problems
caused to other senders due to the non-backward-compatible signaling
response.

Best,
Jake

From: Bob Briscoe <[email-masked]>
Date: Sun,2021-11-07 at 8:40 AM
To: "Holland, Jake" <[email-masked]>, Wesley Eddy <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

Jake,

----

Major:

[snip]

[BB] If we make this requirement so that monitoring MUST report all Classic ECN AQMs, operators could just have a sea of data that doesn't say whether there are any of the single-queue AQMs out there - much more likley to be a potential problem than FQ AQMs. The key word in this requirement is 'problems'. 

Also note that the word 'might' in 'might be in a shared queue' allows for the case of a flow-queue that is exhibiting problems at the time it is monitored (e.g. during real time monitoring when within a VPN). BTW, we created a matrix of cases, and checked all the possible cases when we wordsmithed this requirement.

Also, detecting a flow-queue that is ECN marking but appears to exceed expected L4S delay is already in the flow's self-interest. So I wouldn't worry about trying to legislate over-precisely about this case. For instance, of the two OS developers that I have had detailed conversations with about this, both are working on the basis that it's worth taking sthg like the max of the responses to increasing delay and to ECN marks, which will automatically lead to a Classic-like response in an FQ-CoDel queue, as well as in other cases like suddenly reduced capacity.

[BB] Aside: Your middle sentence has (intentionally?) lost the 'SHOULD' preference for monitoring live, rather than out of band.

I think the next para in the draft (not given in your quote) with its mandatory final sentence is stronger than your alternative.
      The detection function SHOULD be capable of making the congestion
      control adapt its ECN-marking response to coexist safely with
      Classic congestion controls such as standard Reno [https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-uFrhRnEc$], as
      required by [https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-uqCDg5WI$].  Alternatively, if adaptation is not
      implemented and problems with such an AQM are detected, the
      scalable congestion control MUST be replaced by a Classic
      congestion control.
The differences are:
* Replacement is a 'MUST' not a 'SHOULD', but conditional on 'problems'
* It doesn't distinguish between shared queue and FQ.

We didn't think it was reasonable to give the sender a requirement that depends on what type of queue it encounters, because the sender doesn't know the type for sure. That's why we made this conditional on 'problems' instead. The last sentence of this passage from the referenced Appendix A.1.5 explains why:

In summary I think your suggestion of conditioning on the type of queue is likely to be infeasible and therefore prone to being ignored.
This is not easy but, on balance, I don't think I've seen anything here that is tighter than the existing wording.

[BB] After guidance from the chairs, the co-authors have been working offlist on a fairly lengthy new subsection in ecn-l4s-id about where the RFC4774 requirements are and are not satisfied, and how that is to be dealt with. Rather than pasting it all here (it's long) you should see it later today once the secretariat allow the draft through (or otherwise when the servers re-open in the morning). And I'm sure the chairs will give everyone time to read it over, and review it.

Nits:

   o

Continued...

----

+1 to Gorry's comments here, especially regarding the use of "all traffic":
https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/vMMsQpXs65lk1E7NpV5RlmpyqdI/__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-ul4tJbII$

Minor:
1. l4s-arch section 1:

[BB] I've added:
  [Dukkipati15]
              Dukkipati, N. and N. McKeown, "Why Flow-Completion Time is
              the Right Metric for Congestion Control", ACM CCR 
              36(1):59--62, January 2006,
              https://urldefense.com/v3/__https:/dl.acm.org/doi/10.1145/1111322.1111336__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-un5qa5gw$.
  [Rajiullah15]
              Rajiullah, M., "Towards a Low Latency Internet:
              Understanding and Solutions", Masters Thesis; Karlstad
              Uni, Dept of Maths & CS 2015:41, 2015, https://urldefense.com/v3/__https:/www.diva-portal.org/smash/get/diva2:846109/FULLTEXT01.pdf__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-ud1apytw$.

Editorial:

[BB] When it only referenced the DualQ draft and only gave a short summary, we were asked to summarize more fully how the DualQ works here (sigh!).

So we've kept the explanation here, but de-confused and de-walled it a bit:
* Removed the 'obvious/non-obvious' wording.
* Within the largest block: Introduced sub-bullets for the two parts of the semi-permeable membrane, each starting "Latency isolation:" and "Bandwidth pooling:". Then continued the part about the scheduler with 2 further bullets, again starting each with "for latency isolation" and "for bandwidth pooling".
* Cut out some redundancy.
* I considered shifting out the last para of the DualQ bullet, which is about the DualQ document, but I decided there was no better place for it.

[BB] I agree, use of emphasis is excessive. I've taken out /some/ but not /all/ ;)

The ones left are:
* "on /average/" (because the comparison with the other P99 is 3 lines of dense numbers away)
* "without the /need/ for per-flow operations" (to help the reader not to just run over the word without thinking)
* "without /requiring/ inspection" (---ditto---)

Hopefully, this is now down to a non-irritating level for those who don't like it, and now conforms to the guidance from the only source I could quickly find on international technical writing style:
"Do not use italics for
• mere emphasis. (Italics are acceptable if emphasis might otherwise be lost; in general, however, use syntax to provide emphasis.)" 
American Psychological Association (APA, 2009, pp. 104-106)
via https://urldefense.com/v3/__https:/writing.stackexchange.com/questions/10750/when-should-i-use-italics-in-scientific-writing__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-uHTBTecE$

(BTW, underscores in ASCII can mean either italics or underlining. In the HTML rendering, these words come out in italics - which was the intention)

----

[BB] On signalling design, I'd like to point out that it's not just the authors - the chairs have judged that the WG wants to go this way. 

Whatever, thank you - for your reviews, and for your continuing perseverance and diligence.

Cheers

Bob

Best,
Jake

On 07-29, 9:18 AM, "Wesley Eddy" mailto:[email-masked] wrote:

- Architecture: https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-u3NnimRI$ 

- DualQ: 
https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-uscUhkHw$ 

- ECN ID: https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-usmeCGjo$ 

-- 
________________________________________________________________
Bob Briscoe                               https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!Hnqqz6_j8SIdAfFyCkZ-d1cMB2kGbO96HKTk-QSU1VUYsO2YkIK23b-ucdq-oCs$

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 9 Nov 2021 00:14:31 +0000

Stuart,

On 27/08/2021 11:48, Stuart Cheshire wrote:
> On 29 Jul 2021, at 09:18, Wesley Eddy <[email-masked]> wrote:
>
>> This message is starting a combined working group last call on 3 of the L4S drafts:
>>
>> - DualQ: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
>>
> These are exceptionally high quality documents. The insights are profound, and the writing is excellent. I have a few minor observations, and one more major one.
>
> =====
>
> <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/>
>
> I find use of reference-as-noun makes some sentences hard to parse and understand. It assumes every reader has every RFC memorized and unconsciously treats every RFC number as a noun. To see the text as a normal reader would, assume that instead of [RFC1234] all references are written numerically, like [1]. Or, even better, as a little superscript number, as they were back in the days when we still had typesetting.
>
> For example, what does this mean, on page 8: “To enable L4S, the standards track¹ has had to be updated...”

[BB] I agree with you. But I ran out of time to find and convert all 
these. It's on my ToDo list for the next rev.

>
> --
>
> Page 7 defines Reno-friendly:
>
>     Reno-friendly:  The subset of Classic traffic that is friendly to the
>        standard Reno congestion control defined for TCP in [RFC5681].
>
> The term “friendly” is defined in terms of “friendly”, which seems a bit circular. What does “friendly” mean precisely?
[BB] I've added:

     The TFRC spec. [RFC5348] indirectly implies that 'friendly' is 
defined as "generally within a factor of two of the sending rate of a 
TCP flow under the same conditions".

>
> And what is “Classic traffic”? Traffic that uses Classic Congestion Control?
>
>     Classic Congestion Control:  A congestion control behaviour that can
>        co-exist with standard Reno [RFC5681] without causing
>        significantly negative impact on its flow rate [RFC5033].
>
> If Classic traffic does not negatively impact Reno, and Reno-friendly traffic is friendly to Reno, what’s the difference? Why is Reno-friendly a subset of Classic traffic? What is an example of traffic that is Classic (does not negatively impact Reno) but is not Reno-friendly?

[BB] Coexist includes non-congestion controlled stuff like DNS, and CCs 
that coexist "without causing  significantly impact" [RFC5033]. I'm not 
going to attempt to define significant here. When RFCs like RFC 5033 
were written, I believe the chosen wording was to deliberately dodge 
anything precise. Although the authors of the UDP Usage Guidelines 
[RFC8085] took it upon themselves to state another numeric ratio, albeit 
vaguely: "an order of magnitude".

I think the above additional definition of friendly should answer all 
three of your questions. I hope it's is now clearer that friendly is 
tighter than coexist.
Indeed, in terms of tightness of constraint: Reno-compatible > 
Reno-friendly > Classic
(I'm silently substituting 'Reno' for 'TCP' here)

Bear in mind that I have to grit my teeth when explaining all this, 
given I've written a reasonably highly cited paper saying the metric of 
flow-rate fairness is a load of tosh.

>
> --
>
> Page 17 mentions self-inflicted queuing delay:
>
>        1.  It might seem that self-inflicted queuing delay within a per-
>            flow queue should not be counted, because if the delay wasn't
>            in the network it would just shift to the sender.  However,
>            modern adaptive applications, e.g. HTTP/2 [RFC7540] or some
>            interactive media applications (see Section 6.1), can keep low
>            latency objects at the front of their local send queue by
>            shuffling priorities of other objects dependent on the
>            progress of other transfers.  They cannot shuffle objects once
>            they have released them into the network.
>
> It might be helpful here to mention the TCP_NOTSENT_LOWAT socket option.
>
> If you want a reference, something like this might do:
>
> <https://blog.cloudflare.com/http-2-prioritization-with-nginx/>

[BB] I would like to, but the RFC Editor kicks out non-archival 
references. I've put it in for now, and see if we can find something 
more archival later.

>
> --
>
> Page 22:
>
>     For any one L4S flow to work, it requires 3 parts to have been
>     deployed.
>
> I *think* the three parts you mean are: client, bottleneck queue, and server.
>
> But I’m not sure. It would be good for this text to specify what the 3 parts are.

[BB] Good point. Added:

"...deployed: i) the congestion control at the sender; ii) the AQM at 
the bottleneck; and iii) older transports (namely TCP) need upgraded 
receiver feedback too."

>
> --
>
> Page 27:
>
>     the L4S service relies on self-constraint
>
> I think you meant:
>
>     the L4S service relies on self-restraint

[BB] Sry, brain-fart (2 occurrences of self-restraint already elsewhere)

Thank you, continuing...

>
> =====
>
> <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/>
>
> Page 6:
>
>     Even with a perfectly tuned AQM, the additional
>     queuing delay will be of the same order as the underlying speed-of-
>     light delay across the network.
>
> I know what you’re saying here, but some readers might miss the significance. How about adding some words at the end:
>
>     Even with a perfectly tuned AQM, the additional
>     queuing delay will be of the same order as the underlying speed-of-
>     light delay across the network, thereby doubling the total round-trip time.

[BB] Thx. I've also added 'roughly' to doubling.

>
> --
>
> Page 16:
>
>     {ToDo: Not-ECT / ECT(0) ?}.
>
> Seems like this bit is not quite ready for IETF Last Call.

[BB] See list discussion since - Not-ECT now selected.
I noticed that one myself soon after I'd posted it, but before I'd read 
the numerous reviews that pointed that omission out.

>
> --
>
> Page 20, 'Safe' Unresponsive Traffic
>
>     The above section requires unresponsive traffic to be 'safe' to mix
>     with L4S traffic.  Ideally this means that the sender never sends any
>     sequence of packets at a rate that exceeds the available capacity of
>     the bottleneck link.  However, typically an unresponsive transport
>     does not even know the bottleneck capacity of the path, let alone its
>     available capacity.  Nonetheless, an application can be considered
>     safe enough if it paces packets out (not necessarily completely
>     regularly) such that its maximum instantaneous rate from packet to
>     packet stays well below a typical broadband access rate.
>
>     This is a vague but useful definition, because many low latency
>     applications of interest, such as DNS, voice, game sync packets, RPC,
>     ACKs, keep-alives, could match this description.
>
> This makes me very nervous.
>
> I agree that it is vague, but I don’t think it’s useful.
>
> A bunch of people doing voice calls over a shared slow Internet connection (think beach café on some remote island) could easily overwhelm the available capacity, and their clients should react by selecting a lower-rate voice codec, not by driving the network into massive packet loss so that nothing works. Conversely, if spare capacity is available, the clients should react by selecting a higher-rate voice codec to make use of that extra capacity and give better voice quality. I don’t see any good reason why voice clients, or video games, should not adapt to give the best user experience possible on a given network capacity. And that adaptation includes adapting down as well as adapting up. A voice call that adapts down (and works) gives a better user experience than a voice call that drives the network into congestion collapse (and completely fails).

[BB] Totally agree. This was meant to be explained by the mention of 
available capacity in "...does not even know the bottleneck capacity of 
the path, let alone its available capacity."
I've attempted to remedy this by adding the following at the end of the 
subsection about "'Safe' Unresponsive Traffic":

"Low rate streams such as voice and game sync packets, might not use 
continuously adapting ECN-based congestion control, but they ought to at 
least use a 'circuit-breaker' style of congestion response [RFC8083]. 
Then, if the volume of traffic from unresponsive applications is high 
enough to overload the link, this will at least protect the capacity 
available to responsive applications. However, queuing delay in the L 
queue will probably rise to that controlled by the Classic (drop-based) 
AQM. If a network operator considers that such self-restraint is not 
enough, it might want to police the L queue (see Section 8.2 of 
[I-D.ietf-tsvwg-l4s-arch]."

You didn't give an indication of what you wanted me to do. Does this help?

BTW, I resisted the temptation to also cite RFC7893, which favourably 
compares the ITU-recommended termination of voice calls due to high loss 
with TCP's congestion response, and goes on to calculate how large a 
pseudowire of multiple voice circuits can be before it is worse than 
TCP's response.

> Page 20:
> This seems like a disincentive for client applications to use the L4S identifier, because they might actually get lower latency and lower loss by simply sticking with ECT(0). (More on this below.)

[BB] I don't know whether you read the same ambiguity in this that Pete 
Heist pointed out in his review: 'operator' implied all the operator's 
nodes, not just this one. It now says :

"A network node that supports L4S but excludes certain traffic carrying 
the L4S identifier from L4S treatment MUST NOT treat ..."

It was a perhaps lame attempt not to add more nodes that treat ECT(1) 
the same as ECT(0) during deployment of L4S.
Admittedly, it was written on the assumption that the node had 
previously not supported any form of ECN at all, in which case this 
wouldn't be a regression. I guess that assumption isn't 100% likely.

I'll leave it as it is for now, but open to changing it after more 
discussion on the list.

>
> I didn’t find any mention of what these documents assume a Classic (e.g., FQ-CoDel) bottleneck will do with an ECT(1) packet. I think it is assumed that an FQ-CoDel bottleneck will treat ECT(1) exactly the same as ECT(0), but the documents don’t say so.

[BB] Since the recent patch to Linux FQ_CoDel, there's now text on an 
ETC(0/1) distinction in a couple of placed in l4s-arch (search for 
'FQ-CoDel').

more...

>
> =====
>
> Final Observation:
>
> With our work slaying the Bufferbloat beast almost complete now, I started thinking about what advice we should be giving to developers at next year’s Apple Worldwide Developer Conference (WWDC). Should their apps use ECT(0) for lower latency and lower loss? FQ-CoDel with ECN is deployed in some places, and increasing. With many home gateways today you just have to turn it on. Or should their apps use ECT(1) for even lower latency? L4S is not widely deployed, so ECT(1) gives zero immediate benefit to the developer using it, but possibly it might be the better long-term direction. As a developer I know what I’d pick: “I’ll use ECT(0) now for the immediate benefit. Once L4S overtakes FQ-CoDel let me know and I may switch.” But if all the app developers are using ECT(0), L4S will never overtake FQ-CoDel, because there will be no reason for anyone to ever deploy L4S with ECT(1) if all the apps are using ECT(0). If all the apps are using ECT(0) to get FQ-CoDel behaviour, then home gateway vendors will implement FQ-CoDel with ECT(0) because that’s what shows a measurable customer benefit immediately.
>
> So, let’s think about the pros and cons of an app developer choosing ECT(0) with Classic Congestion Control vs. ECT(1) with Scalable Congestion Control.
>
> Marking packets ECT(0) instead of “Not ECT” will get the benefit of lower loss and therefore lower retransmission delays, on both Classic FQ-CoDel bottlenecks and newer L4S bottlenecks. These congestion signals however are coarse, so the sender will need to make large sawtooth rate swings in response to a CE mark, so it will never get truly low delay and high throughput consistently. It will either over-fill the queue and get higher delay, or under-fill the queue and get lower throughput.
>
> Marking packets ECT(1) will get the same benefit of lower loss and therefore lower retransmission delays. In addition, if the bottleneck is actually L4S, the sender also gets the benefit of fine congestion signals, allowing more moderate rate adjustments, keeping the queue consistently short while keeping throughput high.
>
> However, if the bottleneck is actually FQ-CoDel, and a single CE mark is supposed to produce a large rate reduction, but the sender makes only a modest rate reduction, then the sender will continue to grow the bottleneck queue. Over time the FQ-CoDel algorithm will slowly get more persistent with its CE marks until eventually the sender makes a sufficient rate correction. In the meantime the bottleneck queue will have grown much worse than it would have done with a classic sender that reduced its congestion window more aggressively at the first CE mark.
>
> So, an L4S ECT(1) sender hitting a Classic FQ-CoDel bottleneck actually gets worse queueing delay than a Classic ECT(0) sender. This is a serious disincentive to client adoption of ECT(1) with Scalable Congestion Control.
>
> I don’t think heuristics to detect the bottleneck queue type can react fast enough to help.

[BB] See below....

>
> The bottleneck queue for a path can change rapidly.
>
> Consider a customer with 100Mb/s ISP service and a Wi-Fi Access Point in their home.
>
> When the customer is standing close to their Wi-Fi Access Point their Wi-Fi rate may be 300Mb/s, so their downstream bottleneck link is the ISP service coming in to their house. Suppose they have an enlightened ISP that already implements L4S Dual Queue at the downstream link. The customer’s L4S-dependent client software works great.
>
> When the customer moves a little further from their Wi-Fi Access Point their Wi-Fi rate may drop to 50Mb/s, so their downstream bottleneck link is now their Wi-Fi Access Point. Suppose their Wi-Fi Access Point implements Classic FQ-CoDel but not L4S. The customer’s L4S-dependent client software now works even worse than simple loss-based congestion control. Instead of making a drastic reduction in response to a CE mark, it makes a small reduction and the queue (and delay) keep growing.
>
> As this user walks around their house (or as other people walk around them) their downstream bottleneck link is going to flip flop between their ISP’s L4S downstream queue, and their Wi-Fi Access Point’s Classic FQ-CoDel queue.
>
> This suggests that we need a way for client software to use L4S in a way that takes advantage of an L4S bottleneck when available, but does no worse than Classic ECN when the bottleneck happens to be Classic ECN instead of L4S.

[BB] The Classic ECN AQM detection and fallback algo in TCP Prague 
(currently default off) responds to a switch of AQM within several tens 
of round trips. There are too many false positives in the current algo, 
but I'm just pushing back on the question of speed. If we worked on this 
more (which I'd like to) the way I'm thinking that the false positives 
could be removed shouldn't slow down the response.

To see how fast it reacts, visit:
     https://l4s.net/ecn-fbk/results_v2.2/full_heatmap/
Scroll down to the 'Switch AQMs' heatmaps,
Select and scenario and click on any 'Classic ECN Variable' link, which 
shows plots of this variable, which controls adaptation of the CC 
between scalable and Classic.
You can see how fast it switches between it's extremes - when it does.

There's explanation of the experiments on the pages, but I can walk you 
(or anyone) through it if it's all too demanding.

I'll leave the rest of your email as part of the ongoing discussion on 
this. Suffice to say your point about not creating a deployment dilemma 
is important and I had already taken it on board when you first hinted 
at this dilemma a couple of IETF's ago. As you know, we're working on 
making sure this is not a dilemma that app developers will have to face.

Thank you again for picking up some important points in your review.
I will now post another rev to l4s-arch (-14), and finally post a rev 
containing all the backed up changes to ecn-l4s-id (-22).

Bob

>
> Conclusion:
>
> Because of the reasons explained eloquently in the documents, queues are going to be shared. If traffic wants to be in an ultra-low latency shared queue, it needs to signal its agreement to play nicely with others and not be bursty (with associated penalty if it is caught misbehaving). This means we need three input code points:
>
> (a) I am not ECT (please drop my packets after the queue gets fairly big)
> (b) I understand Classic ECN (please mark after the queue gets fairly big)
> (c) I understand Low-Latency ECN (please mark my packets when the queue starts to grow even slightly, and I promise to reduce my rate promptly)
>
> On the output side what signals do we need?
>
> Input (a) does not have a congestion signal output (a dropped packet has no header fields)
> Input (b) can have output (d) below
> Input (c) can have output (d) or (e) below
>
> (d) Do drastic congestion window reduction
> (e) Do moderate congestion window reduction
>
> This means we need to encode five values. That doesn’t fit in a two-bit field. One of the values has to go somewhere else. Maybe a DiffServ codepoint for input code point (c)?
>
> I know this conclusion won’t make people happy. It doesn’t make me happy, but I don’t see any way around it.
>
> We need a way for Scalable Congestion Control senders to tag their packets in a way that tells the network: “I understand L4S. If my bottleneck queue at this instant in time happens to be a dumb FIFO whose queue is full, then this packet will get dropped and I will do a drastic congestion window reduction in response. If my bottleneck queue at this instant in time happens to support Classic FQ-CoDel and it has determined that it has a standing queue then this packet will be marked (d) and I will do a drastic congestion window reduction in response. If my bottleneck queue at this instant in time happens to support L4S and it has a moderate queue then this packet will be marked (e) and I will do a moderate congestion window reduction in response.”
>
> Stuart Cheshire
>

[BB] I think this is the critical point. Certainly, our Clss

[BB]

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 9 Nov 2021 16:40:12 +0000

Stuart,  see [BB2]

On 09/11/2021 00:14, Bob Briscoe wrote:
>> These are exceptionally high quality documents. The insights are 
>> profound, and the writing is excellent. I have a few minor 
>> observations, and one more major one.
>>
>> =====
>>
>> I find use of reference-as-noun makes some sentences hard to parse 
>> and understand. It assumes every reader has every RFC memorized and 
>> unconsciously treats every RFC number as a noun. To see the text as a 
>> normal reader would, assume that instead of [RFC1234] all references 
>> are written numerically, like [1]. Or, even better, as a little 
>> superscript number, as they were back in the days when we still had 
>> typesetting.
>>
>> For example, what does this mean, on page 8: “To enable L4S, the 
>> standards track¹ has had to be updated...”
>

[BB2] I've now scanned through and found some occurrences.

To check that my sensitivity gauge for this is about right, below I've 
given some examples of where I did (in bold) and didn't change the text. 
I could only find 2 cases that I thought needed changing. You'll see 
there are cases where I use citations as nouns - I don't think that is 
precisely the problem - it's more about making sure to use a natural 
language moniker when the citation is integral to the meaning of the 
sentence:

"[I-D.ietf-tsvwg-aqm-dualq-coupled] gives a full explanation of the 
DualQ Coupled AQM framework.  A specific marking algorithm is not 
mandated for L4S AQMs.  Appendices of I-D.ietf-tsvwg-aqm-dualq-coupled] 
give non-normative examples"

"*The L4S identifier spec. *[I-D.ietf-tsvwg-ecn-l4s-id] concludes that 
all alternatives involve compromises,..."

"To enable L4S, *the standards track Classic ECN spec.* [RFC3168] has 
had to be updated to allow L4S packets to depart from the 'equivalent to 
drop' constraint. [RFC8311] is a standards track update to relax 
specific requirements in RFC 3168 (and certain other standards track 
RFCs), which clears the way for the experimental changes proposed for 
L4S. [RFC8311] also reclassifies the original experimental assignment of 
the ECT(1) codepoint as an ECN nonce [RFC3540] as historic."

"[I-D.ietf-tsvwg-ecn-l4s-id] specifies that ECT(1) is used as the 
identifier to classify L4S packets into a separate treatment from 
Classic packets."

"Appendix B of [I-D.ietf-tsvwg-ecn-l4s-id] explains why five unlikely 
eventualities"

"The policy goal of the marking could be to differentiate flow rates 
(e.g. [Nadas20], which requires additional signalling of a per-flow 
'value'), or to equalize flow-rates (perhaps in a similar way to Approx 
Fair CoDel [AFCD], [I-D.morton-tsvwg-codel-approx-fair],..."

I adopt this stance myself, so I have no problem being pulled up on it. 
When I review papers, I point out cases where citations are discussed 
without identifying the citation with some natural language (though not 
necessary where the citation is incidental to comprehension). Indeed, 
when I was arguing for the use of 'Classic ECN', I argued against 
calling it RFC 3168 ECN for similar reasons - the RFC numbers that one 
person deals with day to day aren't the same for others.

Pls confirm these are OK.
Cheers

Bob

>> The term “friendly” is defined in terms of “friendly”, which seems a 
>> bit circular. What does “friendly” mean precisely?
> [BB] I've added:
>
>> If Classic traffic does not negatively impact Reno, and Reno-friendly 
>> traffic is friendly to Reno, what’s the difference? Why is 
>> Reno-friendly a subset of Classic traffic? What is an example of 
>> traffic that is Classic (does not negatively impact Reno) but is not 
>> Reno-friendly?
>
> [BB] Coexist includes non-congestion controlled stuff like DNS, and 
> CCs that coexist "without causing  significantly impact" [RFC5033]. 
> I'm not going to attempt to define significant here. When RFCs like 
> RFC 5033 were written, I believe the chosen wording was to 
> deliberately dodge anything precise. Although the authors of the UDP 
> Usage Guidelines [RFC8085] took it upon themselves to state another 
> numeric ratio, albeit vaguely: "an order of magnitude".
>
> given I've written a reasonably highly cited paper saying the metric 
> of flow-rate fairness is a load of tosh.
>
>>
>> -- 
>>
>> I know what you’re saying here, but some readers might miss the 
>> significance. How about adding some words at the end:
>>
> I noticed that one myself soon after I'd posted it, but before I'd 
> read the numerous reviews that pointed that omission out.
>
>>
>> -- 
>>
>> A bunch of people doing voice calls over a shared slow Internet 
>> connection (think beach café on some remote island) could easily 
>> overwhelm the available capacity, and their clients should react by 
>> selecting a lower-rate voice codec, not by driving the network into 
>> massive packet loss so that nothing works. Conversely, if spare 
>> capacity is available, the clients should react by selecting a 
>> higher-rate voice codec to make use of that extra capacity and give 
>> better voice quality. I don’t see any good reason why voice clients, 
>> or video games, should not adapt to give the best user experience 
>> possible on a given network capacity. And that adaptation includes 
>> adapting down as well as adapting up. A voice call that adapts down 
>> (and works) gives a better user experience than a voice call that 
>> drives the network into congestion collapse (and completely fails).
>
> available capacity in "...does not even know the bottleneck capacity 
> of the path, let alone its available capacity."
> I've attempted to remedy this by adding the following at the end of 
> the subsection about "'Safe' Unresponsive Traffic":
>
> continuously adapting ECN-based congestion control, but they ought to 
> at least use a 'circuit-breaker' style of congestion response 
> [RFC8083]. Then, if the volume of traffic from unresponsive 
> applications is high enough to overload the link, this will at least 
> protect the capacity available to responsive applications. However, 
> queuing delay in the L queue will probably rise to that controlled by 
> the Classic (drop-based) AQM. If a network operator considers that 
> such self-restraint is not enough, it might want to police the L queue 
> (see Section 8.2 of [I-D.ietf-tsvwg-l4s-arch]."
>
> compares the ITU-recommended termination of voice calls due to high 
> loss with TCP's congestion response, and goes on to calculate how 
> large a pseudowire of multiple voice circuits can be before it is 
> worse than TCP's response.
>
>
>>
>> -- 
>>
>> Page 20:
>>
>> This seems like a disincentive for client applications to use the L4S 
>> identifier, because they might actually get lower latency and lower 
>> loss by simply sticking with ECT(0). (More on this below.)
>
> [BB] I don't know whether you read the same ambiguity in this that 
> Pete Heist pointed out in his review: 'operator' implied all the 
> operator's nodes, not just this one. It now says :
>
> "A network node that supports L4S but excludes certain traffic 
> carrying the L4S identifier from L4S treatment MUST NOT treat ..."
>
>> I didn’t find any mention of what these documents assume a Classic 
>> (e.g., FQ-CoDel) bottleneck will do with an ECT(1) packet. I think it 
>> is assumed that an FQ-CoDel bottleneck will treat ECT(1) exactly the 
>> same as ECT(0), but the documents don’t say so.
>
>> With our work slaying the Bufferbloat beast almost complete now, I 
>> started thinking about what advice we should be giving to developers 
>> at next year’s Apple Worldwide Developer Conference (WWDC). Should 
>> their apps use ECT(0) for lower latency and lower loss? FQ-CoDel with 
>> ECN is deployed in some places, and increasing. With many home 
>> gateways today you just have to turn it on. Or should their apps use 
>> ECT(1) for even lower latency? L4S is not widely deployed, so ECT(1) 
>> gives zero immediate benefit to the developer using it, but possibly 
>> it might be the better long-term direction. As a developer I know 
>> what I’d pick: “I’ll use ECT(0) now for the immediate benefit. Once 
>> L4S overtakes FQ-CoDel let me know and I may switch.” But if all the 
>> app developers are using ECT(0), L4S will never overtake FQ-CoDel, 
>> because there will be no reason for anyone to ever deploy L4S with 
>> ECT(1) if all the apps are using ECT(0). If all the apps are using 
>> ECT(0) to get FQ-CoDel behaviour, then home gateway vendors will 
>> implement FQ-CoDel with ECT(0) because that’s what shows a measurable 
>> customer benefit immediately.
>>
>> So, let’s think about the pros and cons of an app developer choosing 
>> ECT(0) with Classic Congestion Control vs. ECT(1) with Scalable 
>> Congestion Control.
>>
>> Marking packets ECT(0) instead of “Not ECT” will get the benefit of 
>> lower loss and therefore lower retransmission delays, on both Classic 
>> FQ-CoDel bottlenecks and newer L4S bottlenecks. These congestion 
>> signals however are coarse, so the sender will need to make large 
>> sawtooth rate swings in response to a CE mark, so it will never get 
>> truly low delay and high throughput consistently. It will either 
>> over-fill the queue and get higher delay, or under-fill the queue and 
>> get lower throughput.
>>
>> Marking packets ECT(1) will get the same benefit of lower loss and 
>> therefore lower retransmission delays. In addition, if the bottleneck 
>> is actually L4S, the sender also gets the benefit of fine congestion 
>> signals, allowing more moderate rate adjustments, keeping the queue 
>> consistently short while keeping throughput high.
>>
>> However, if the bottleneck is actually FQ-CoDel, and a single CE mark 
>> is supposed to produce a large rate reduction, but the sender makes 
>> only a modest rate reduction, then the sender will continue to grow 
>> the bottleneck queue. Over time the FQ-CoDel algorithm will slowly 
>> get more persistent with its CE marks until eventually the sender 
>> makes a sufficient rate correction. In the meantime the bottleneck 
>> queue will have grown much worse than it would have done with a 
>> classic sender that reduced its congestion window more aggressively 
>> at the first CE mark.
>>
>> So, an L4S ECT(1) sender hitting a Classic FQ-CoDel bottleneck 
>> actually gets worse queueing delay than a Classic ECT(0) sender. This 
>> is a serious disincentive to client adoption of ECT(1) with Scalable 
>> Congestion Control.
>>
>> When the customer is standing close to their Wi-Fi Access Point their 
>> Wi-Fi rate may be 300Mb/s, so their downstream bottleneck link is the 
>> ISP service coming in to their house. Suppose they have an 
>> enlightened ISP that already implements L4S Dual Queue at the 
>> downstream link. The customer’s L4S-dependent client software works 
>> great.
>>
>> When the customer moves a little further from their Wi-Fi Access 
>> Point their Wi-Fi rate may drop to 50Mb/s, so their downstream 
>> bottleneck link is now their Wi-Fi Access Point. Suppose their Wi-Fi 
>> Access Point implements Classic FQ-CoDel but not L4S. The customer’s 
>> L4S-dependent client software now works even worse than simple 
>> loss-based congestion control. Instead of making a drastic reduction 
>> in response to a CE mark, it makes a small reduction and the queue 
>> (and delay) keep growing.
>>
>> As this user walks around their house (or as other people walk around 
>> them) their downstream bottleneck link is going to flip flop between 
>> their ISP’s L4S downstream queue, and their Wi-Fi Access Point’s 
>> Classic FQ-CoDel queue.
>>
>> This suggests that we need a way for client software to use L4S in a 
>> way that takes advantage of an L4S bottleneck when available, but 
>> does no worse than Classic ECN when the bottleneck happens to be 
>> Classic ECN instead of L4S.
>
> (currently default off) responds to a switch of AQM within several 
> tens of round trips. There are too many false positives in the current 
> algo, but I'm just pushing back on the question of speed. If we worked 
> on this more (which I'd like to) the way I'm thinking that the false 
> positives could be removed shouldn't slow down the response.
>
> Select and scenario and click on any 'Classic ECN Variable' link, 
> which shows plots of this variable, which controls adaptation of the 
> CC between scalable and Classic.
> There's explanation of the experiments on the pages, but I can walk 
> you (or anyone) through it if it's all too demanding.
>
> this. Suffice to say your point about not creating a deployment 
> dilemma is important and I had already taken it on board when you 
> first hinted at this dilemma a couple of IETF's ago. As you know, 
> we're working on making sure this is not a dilemma that app developers 
> will have to face.
>
>> Because of the reasons explained eloquently in the documents, queues 
>> are going to be shared. If traffic wants to be in an ultra-low 
>> latency shared queue, it needs to signal its agreement to play nicely 
>> with others and not be bursty (with associated penalty if it is 
>> caught misbehaving). This means we need three input code points:
>>
>> (c) I understand Low-Latency ECN (please mark my packets when the 
>> queue starts to grow even slightly, and I promise to reduce my rate 
>> promptly)
>>
>> This means we need to encode five values. That doesn’t fit in a 
>> two-bit field. One of the values has to go somewhere else. Maybe a 
>> DiffServ codepoint for input code point (c)?
>>
>> I know this conclusion won’t make people happy. It doesn’t make me 
>> happy, but I don’t see any way around it.
>>
>> We need a way for Scalable Congestion Control senders to tag their 
>> packets in a way that tells the network: “I understand L4S. If my 
>> bottleneck queue at this instant in time happens to be a dumb FIFO 
>> whose queue is full, then this packet will get dropped and I will do 
>> a drastic congestion window reduction in response. If my bottleneck 
>> queue at this instant in time happens to support Classic FQ-CoDel and 
>> it has determined that it has a standing queue then this packet will 
>> be marked (d) and I will do a drastic congestion window reduction in 
>> response. If my bottleneck queue at this instant in time happens to 
>> support L4S and it has a moderate queue then this packet will be 
>> marked (e) and I will do a moderate congestion window reduction in 
>> response.”
>>
>> Stuart Cheshire
>>
>

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 11 Nov 2021 02:21:32 +0200

> On 8 Nov, 2021, at 4:52 pm, Holland, Jake <jholland=[email-masked]> wrote:
> 

During Monday's session, I noted that L4S does not meet the requirements of RFC-4774 Option 3, and that this is not merely a matter of minor compromises from the letter of the definition.  Without having followed every single entry in this thread, I'd like to highlight the above as illustrative of what I meant.

The very fact that L4S transports have a need to detect problems and respond to them by falling back to a compatibility mode is directly characteristic of RFC-4774 Option 2.  The fact that they cannot presently do so with any reasonable level of reliability places them in RFC-4774 Option 1.  Option 3 requires that friendly coexistence with conventional traffic (both ECT and Not-ECT) is inherent to the system, not merely a mode that is explicitly switched into.

I do not believe that L4S can comply with RFC-4774 Option 3 without a fundamental redesign, starting with the signalling method.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Thu, 11 Nov 2021 09:34:59 +0100

Hi Bob,

I read the responses. To keep things short, I'll just reply to a couple
points for clarification...

On Tue, 2021-10-26 at 00:12 +0100, Bob Briscoe wrote:
> well within the "order of magnitude" required in the most recent RFC
> on this subject [RFC8085] (altho' that was about bulk UDP transfers,
> I can't see how fairness would depend on which transport protocol is
> in use).
> 
> 
> > 
> > > dualq-coupled Abstract:
> > > 
> [BB] Gorry asked me to cut down the abstract, so I've removed all the
> stuff about testing in a residential broadband setting anyway. 
> See the mail I just sent him for the new wording. However, I can see
> now that changing it is probably going to lead to further haggling
> over different wording...
> 
> Nonetheless, let's move from wording to your technical point itself.
> Before I was hired by CableLabs to work on Low Latency DOCSIS I wrote
> a tech report about why bursty effects like this might happen and how
> to deal with it if they were a problem [SigQ-Dyn]. It's about how
> sojourn time doesn't measure a burst even once the whole burst is
> queued. You'll find this is already referenced from the dualq draft.
> We implemented the alternative metric ('scaled sojourn time'), but
> didn't use it in the end because we couldn't find a good reason for
> it being necessary. Since that time, I've thought of a simpler way of
> implementing an improved variant of the metric, which I wrote into
> the next rev of the draft recently.
> 
> But before considering replacing sojourn time we need to bear in mind
> that there's only one run of one experiment that looks odd. So we
> need to set ourselves up with a lot more solid evidence. If there is
> a performance degradation in more typical scenarios, we need to know
> whether it's big enough to worry about. If it is, the modified metric
> should improve the performance again.
> 
> Technical Report TR-BB-2017-001 arXiv:1904.07044 [cs.NI], September
> 2017, <https://arxiv.org/abs/1904.07044>. 
> 
> 
> > 

What was meant was primarily "that every packet in every flow can enjoy
very low latency". The addition of "potentially" should cover the cases
where applications, link layers or any unforeseen factors don't
cooperate to achieve very low latency.

> [BB] The draft says "below 100ms". And it gives the ref to the paper
> with the spec of the tests - it says they were over DSL, then
> emulated with an ethernet testbed for higher speeds than the DSL
> could achieve, after comparing the validity of the emulation.
> 
> 
> > Bursty
> * "Updated link designs are not going to appear any time soon. In the
> mean time, the parameters of any flaky links and the L4S target Q
> delay can be set to levels that trade off between utilization and
> delay."
> To be clearer, I'll add "... unresponsive traffic gains any advantage
> from using the ECN capability (relative to being unresponsive without
> ECN)"

I did misunderstand this to mean that drop on saturation would address
the problem of flows that are unresponsive to ECN signals, which is
where the straightforward result I posted earlier came from. That can
be ignored in this context so I'll remove the link to avoid confusion.

There's one more result related to gaining an advantage by marking
ECT(1) it doesn't look like we posted before. I need to find and verify
that, and I'll post it in a separate thread if I think it's relevant.

Regards,
Pete

---

From: Pete Heist <[email-masked]>
Date: Thu, 11 Nov 2021 11:40:13 +0100

Hi Bob, 

> experiments, but also putting in the time to review all theree

I'm glad if something useful came from it. I've read all the replies,
and will keep my response short overall...

> But I think you're right that 'never' is rather too ambitious, so
> I've toned that down to 'rarely'.
> 
> configured with a shallow threshold and there is a bursty upstream
> link. This general issue is well-known - I doubt it is specific to
> L4S. It's one of the challenges of trying to reduce queuing delay at
> one link at a time, while not being able to improve all the other
> links at the same time. 
> nature of this problem into a 'current worst-case' problem. I mean
> that, at the moment you have to relax the target delay of a AQM to
> cater for the /potential/ worst-case upstream link burstiness. I
> would like to be able to sense the /actual/ worst case upstream link
> and _very slowly_ adapt the target configuration to it. But it's not
> so simple.)
> 
> improvements even without actors having to coordinate. As I said in
> my response to your similar comment on the DualQ draft, see my
> posting on the list to Sebastian on 31 Aug 21: "[tsvwg] tests of
> Prague utilization with bursty competition in L and C [more burst
> sources]"
>    
>     In that email there is sthg like the text that will appear in a
> new section 5.6 in ecn-l4s-id about upstream bursty links.
>     For instance, dealing with the burstiness of all WiFi interfaces
> on an Access Point at the same time as deploying an L4S AQM on the
> upstream WAN interface.

I raised a point in Monday's meeting that I thought the burstiness
issue to be of lesser importance than those around signaling confusion
with existing middleboxes. The reason was, I understand that bursts are
created prior to arriving at bottlenecks, and are not something that
middleboxes or L4S can undo. But I am glad you're considering this
important issue, as it's relevant to L4S's stated goals. If they're not
seeing those goals realized at the outset, operators and vendors will
probably want a sense of why, and what they have to do to get there.
I'll avoid further comment on burstiness as I have the sense it's been
heard.

> (not just behind us). I can't see that there's any evidence /yet/ to
> say that this latency goalhas not been attainable on each link
> technology, once an attempt to support L4S on it reached some degree
> of maturity. 
> 
> I suspect this goal will be challenging for radio links (which
> includes satellite), but I'd rather revise the goal /after/ those
> working on it have had time for a decent crack at it, not before.
> 
> BTW, note that, where the draft gives actual results, it specifies
> the link technology used.
> 
> 
> 
> [BB] Via this link you make the point that "this is the Internet as
> it is today". But the point above about improving multiple links at
> once apply here too. Not just the point about the links in and out of
> a home gateway, but also the wider operational abilities of ISPs to
> tune links for customers taking the L4S service. 
> 
> In the light of this experiment, you also make the point that "there
> is a discussion to be had around whether reliably low queueing delays
> will ultimately require flow separation". (BTW, bear in mind that L4S
> includes FQ.) But I think 'ultimately' is the wrong idea here.
> Perhaps "/initially/ require flow separation" would have been a
> better conclusion from this experiment. Users are going to want WiFi
> burstiness to reduce, so I don't think there's an argument for
> 'ultimately' here. But FQ deployments of L4S might ease some of the
> coordination challenges in the short term.
> 
> markings are biased towards the smoother flow they will also be
> biased away from the more bursty flow, which would fool the bursty
> flow into responding less than it would like, leading to increased
> queue delay spikes. If that is the cause, then the proposed remedy
> for the 'bursty traffic' case could also mitigate this 'between-
> flows' case. So let us look into this further. Because FQ-CoDel uses
> the same sojourn-based marking. So in cases like VPNs, FQ won't help
> if there is an underlying problem with immediate marking of bursty
> traffic based on sojourn time.
> 
> 
> criticism of the text here is warranted. No-one expects the goals of
> the L4S Experiment to have already been attained everywhere - we
> haven't even got the experiments started over the real Internet yet.
> 
> 
> > 
> > > l4s-arch Section 2(1):
> > > 
> RFC4774 requirement "for identifying an alternative ECN treatment",
> not the one you refer to here. Whatever, I have done
> To you point about the RFC4774 §4 requirement you quote, after
> guidance from the chairs, the co-authors have been working offlist on
> a fairly lengthy new subsection in ecn-l4s-id about where the RFC4774
> with. You should see that before the meeting on Monday. And I'm sure
> the chairs will give everyone time to read it over, and review it.

> [BB] Point taken, and see previous responses about this being a
> problem for any approach that reduces queuing delay one link at a
> time. And about it being too early to tell how much the bursty links
> effect can be mitigated as the L4S experiment proceeds. 
> 
> [BB] We're aware that we're not doing ourselves any favours by
> pointing out weaknesses in other technologies that might be close to
> the hearts of people involved in the review process. But we also have
> to justify why the IETF should assign a scarce IP codepoint, which
> means we have to spell out what L4S adds over other approaches (hence
> the whole section this paragraph comes from). Nonetheless, I will
> turn the last sentence round into:
> 
> fixing. In fact, it's off the point for this section about Classic
> AQMs, and there is a point that ought to have been said about Classic
> AQMs, but hasn't. Here's proposed replacement text:
> 
> "The L4S approach resolves this tension between delay and utilization
> by enabling hosts to minimize the amplitude of their sawteeth. A
> single-queue Classic AQM is not sufficient to allow hosts to use
> small sawteeth for two reasons: i) smaller sawteeth would not get
> lower delay in an AQM designed for larger amplitude Classic sawteeth,
> because a queue can only have one length at a time; and ii) much
> smaller sawteeth implies much more frequent sawteeth, so L4S flows
> would drive a Classic AQM into a high level of ECN-marking, which
> would appear as heavy congestion to Classic flows, which in turn
> would greatly reduce their rate as a result (see Section 6.4.4)."

Ok, I see that my point came across (*).

> "Per-flow implementations need to have support for scalable
> congestion control added, which has already been done for FQ-CoDel in
> Linux (see Sec.5.2.7 of [RFC8290] and [FQ_CoDel_Thresh]."
> 
> [FQ_CoDel_Thresh] points to Toke's recent
> patch:https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=dfcb63ce1de6b10b
> 
> 
> > 
> [BB] But this sentence doesn't give any implication that a Classic
> AQM would switch to a different AQM as well as adding L4S.
> 
> discussion of how an average is not a useful characterization of
> queue delay, or...).
> 
> [BB] I'm happy to change 'safe to use' to 'designed to be used',
> given 'safe' means different things to different people and depends
> on other aspects like whether its monitoring for a Classic ECN AQM,
> whether Classic AQM fall-back has been improved sufficiently to be on
> by default, whether the loss response is robust, whether it
> negotiates feedback, etc. Also most of the differences between Prague
> and DCTCP now are much wider than safety - stuff like corrected 
> averaging, smoother pacing and TSO, less jumpiness due to no
> suppression of additive increase, RTT-independence.

IMO a language change isn't sufficient here, likewise on the topic of
the comment marked (*), but signaling is a fundamental and important
area where we differ, as we know.

Thanks for incorporating some changes though...

Regards,
Pete

---

From: Pete Heist <[email-masked]>
Date: Thu, 11 Nov 2021 12:04:31 +0100

Hi Bob, for this one, I'll just acknowledge that I read the reply. No
further comments below...

Pete

On Sun, 2021-11-07 at 18:47 +0000, Bob Briscoe wrote:
> Pete,
> 
> Here's how I've dealt with your reviews of the 3rd of the 3 L4S
> drafts: draft-ietf-tsvwg-ecn-l4s-id-19
> See [BB]
> 
> carrying the L4S identifier from L4S treatment MUST NOT treat..."
> 
> [BB] Point taken, but we intended this to set a challenging
> expectation - to see what link designers can achieve. The figures you
> quote are what happens when link designers are allowed to solely
> focus on bandwidth. (BTW, this is not intended to apply to anomalous
> bursts such as the DSL ones Sebastian and I were talking about in the
> thread you quote - this is about routine frame aggregation).
> 
> quote above state this as the floor of what might be expected, not
> the ceiling. Also, for the benefit of readers on the list, the '...'
> in the above quote contains:
> "This document does not presume to even give guidance on an
> appropriate target for such burst delay until there is more industry
> experience of L4S. "
> 
> Nonetheless, only new equipment (having been modified for L4S) will
> be having to comply with this requirement, so we thought it's not
> completely out of the question for high link rates. For instance,
> even this floor would allow a burst of 10x1500B packets on a 480Mb/s
> link.
> 
> Obviously if the rate is slower than 48Mb/s we're not expecting
> bursts to be smaller than one packet.
> 
> 
> I'm sure we'll be talking about the tests you highlighted in the
> start of your email, but I've had to focus on the specific comments
> about the drafts first.
> 
> Cheers
> 
> 
> Bob
> 
> 
>

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Nov 2021 11:52:23 +0000

Pete,

On 11/11/2021 08:34, Pete Heist wrote:
> Hi Bob,
>
> I read the responses. To keep things short, I'll just reply to a 
> couple points for clarification...
>
>> briefest way of summarizing loads of results for words in an abstract.
>>
> What was meant was primarily "that every packet in every flow can 
> enjoy very low latency". The addition of "potentially" should cover 
> the cases where applications, link layers or any unforeseen factors 
> don't cooperate to achieve very low latency.

[BB2] OK. But I'm afraid I can read your 'should' here in two opposite 
ways. You could be meaning either of:
* I'm now happy that... 'the addition of "potentially" should cover the 
cases...'
* 'The addition of "potentially" should cover the cases...' but it 
doesn't, at least not to my satisfaction.

>> delay can be set to levels that trade off between utilization and delay."
>> [BB] That section is about unresponsive traffic gaining advantage 
>> *from using* the ECN capability.
> ECT(1) it doesn't look like we posted before. I need to find and 
> verify that, and I'll post it in a separate thread if I think it's 
> relevant.

[BB2] OK. Please confirm to the list if you don't think it's relevant as 
well.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Nov 2021 12:32:06 +0000

Pete, see [BB2]

On 11/11/2021 10:40, Pete Heist wrote:
> with existing middleboxes. The reason was, I understand that bursts 
> are created prior to arriving at bottlenecks, and are not something 
> that middleboxes or L4S can undo. But I *am* glad you're considering 
> this important issue, as it's relevant to L4S's stated goals. If 
> they're not seeing those goals realized at the outset, operators and 
> vendors will probably want a sense of why, and what they have to do to 
> get there. I'll avoid further comment on burstiness as I have the 
> sense it's been heard.

[BB2] Yes I picked up your point from the jabber transcript afterwards. 
This is a useful data point because, as you've detected, the strengths 
of the priority we each give to these different sets of issues 
(burstiness and signal ambiguity) are inverted.

That is an instance of a general point I have been trying to make to 
others - that there is strongly differing strength of concern about 
different issues from all the participants in the WG. The consensus 
process can ensure that we all understand the facts of different 
empirical data. But it can't make us all align our beliefs on which 
issues are more important. We can try to understand better why others 
are more concerned about different issues, but differences will remain - 
there will never be consensus on priorities.

>> for the 'bursty traffic' case could also mitigate this 
>> 'between-flows' case. So let us look into this further. Because 
>> FQ-CoDel uses the same sojourn-based marking. So in cases like VPNs, 
>> FQ won't help if there is an underlying problem with immediate 
>> marking of bursty traffic based on sojourn time.
>>
>>

[BB2] I think this text change at least state the problem, without 
downplaying it or up-playing it.
Then the only open question is what we do about it, which all funnels 
back to section 4.3 of ecn-l4s-id. See * below.

> area where we differ, as we know.

[BB] I've also added "...is designed to be used in a production Internet 
environment (assuming it complies with the requirements in Section 4 of 
[I-D.ietf-tsvwg-ecn-l4s-id].
That then removes any judgement about whether it does, thus funnelling 
any remaining disagreement to whatever the WG agrees for the Classic ECN 
AQM requirement. Because at the moment, signalling is how it is.

Thank you

Bob

---

From: Pete Heist <[email-masked]>
Date: Thu, 11 Nov 2021 15:37:26 +0100

Hi Bob, two more responses...

On Thu, 2021-11-11 at 11:52 +0000, Bob Briscoe wrote:
> > > [BB] I've changed it to "roughly equivalent", which I think is
> > > the briefest way of summarizing loads of results for words in an
> > > abstract.
> > > 
> > > improved variant of the metric, which I wrote into the next rev
> > > of the draft recently.
> > > 
> > > need to know whether it's big enough to worry about. If it is,
> > > the modified metric should improve the performance again.
> > > 
> [BB2] OK. But I'm afraid I can read your 'should' here in two
> opposite ways. You could be meaning either of:
> * I'm now happy that... 'the addition of "potentially" should cover
> the cases...'

Right, in this case it was the first option, but the results will
ultimately be more important than the words.

> > > There's a lot in that conversation, and you'll see some results
> > > of it appear in the ecn-l4s-id draft shortly. 
> > > * "Whichever technology enables reduced queuing delay
> > > variability, the world would want to change to reduce burst
> > > delays as well. "
> > > 
> > > 
> > > 
> > > 
> > > > 
> > > > 
> > > from using the ECN capability.
> > I did misunderstand this to mean that drop on saturation would
> > address the problem of flows that are unresponsive to ECN signals,
> > which is where the straightforward result I posted earlier came
> > from. That can be ignored in this context so I'll remove the link
> > to avoid confusion.
> > 
> [BB2] OK. Please confirm to the list if you don't think it's relevant
> as well.

I found we've already covered this elsewhere, so there's no need to re-
post it.

Thanks,
Pete

---

From: <[email-masked]>
Date: Tue, 16 Nov 2021 11:51:48 +0000

Bob,
Thanks, the edits all look good to me
Best wishes
Phil

From: tsvwg <[email-masked]> On Behalf Of Bob Briscoe
Sent: 25 October 2021 21:05
To: philip.eardley=[email-masked]; [email-masked]; [email-masked]
Subject: Re: [tsvwg] start of WGLC on L4S drafts

Phil,
On 26/08/2021 09:08, philip.eardley=[email-masked]<mailto:philip.eardley=[email-masked]> wrote:

Some comments on the Architecture draft.

It is a nice document, thanks for all the work.

Some minor suggestions that I think are easy to resolve.

Section 1 Intro

Para 1- maybe some refs could be added, eg "spikes of hundreds of milliseconds are common"

[BB] That comes from a slow start of a short RTT flow in fast links with a state of the art AQM, which is generally tuned to respond over a max RTT (e.g. 100ms). Or multiple parallel flow starts. I've added a couple of references, which are the evaluations of the two main families of solutions prepared by those who developed them:
[COBALT] (which also compares with FQ_CoDel) and [DOCSIS3AQM] which evaluates CoDel, SFQ-CoDel and PIE.

   [COBALT]   Palmei, J., Gupta, S., Imputato, P., Morton, J.,

              Tahiliani, M., Avallone, S., and D. Taeht, "Design and

              Evaluation of COBALT Queue Discipline", In Proc. IEEE

              Int'l Symp. Local and Metropolitan Area Networks

              (LANMAN'19) 2019:1-6, July 2019,

              <https://ieeexplore.ieee.org/abstract/document/8847054><https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fieeexplore.ieee.org%2Fabstract%2Fdocument%2F8847054&data=04%7C01%7Cphilip.eardley%40bt.com%7Cc5762b3bea68444699cb08d997f2d052%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637707891399543930%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Umg1zBPUtfhN1xgG4a%2BObyAH5c6KqHO2FWOz6eGE7bw%3D&reserved=0>.

   [DOCSIS3AQM]

              White, G., "Active Queue Management Algorithms for DOCSIS

              3.0; A Simulation Study of CoDel, SFQ-CoDel and PIE in

              DOCSIS 3.0 Networks", CableLabs Technical Report , April

              2013, <{http://www.cablelabs.com/wp<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cablelabs.com%2Fwp&data=04%7C01%7Cphilip.eardley%40bt.com%7Cc5762b3bea68444699cb08d997f2d052%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637707891399553873%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MEaSLe2%2F2r1ciG%2BVws4KKkB3gLqUA%2Fea06RKpp766pQ%3D&reserved=0>-

              content/uploads/2013/11/

              Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf>.

I think we'll need to temper "common" to "not uncommon" though, 'cos it's difficult to be sure how common the test scenarios are in practice.

and "queuing is typically configured to cause overall network delay to roughly double relative to expected base (unloaded) path delay."

[BB] "Roughly double" refers to the 1 BDP rule of thumb for buffer sizing in access links with low stat mux. I've cited the well-known SIGCOMM'04 paper on 'Sizing Router Buffers'. Altho this is primarily about why the rule does /not/ apply where flows are highly multiplexed (core networks), it starts by explaining the 1 BDP rule for access networks.

Section 2 L4S Architecture Overview

I think you could expand this section a little to provide a 'protocol model' (RFC4101: "to answer three basic questions:   1. What problem is the protocol trying to achieve?     2. What messages are being transmitted and what do they mean?     3. What are the important, but unobvious, features of the protocol?")

The intro does a good job of answering question 1.

For me, S2 should lay out (at a high-level) the solution. The current Section gives the 'components' without spelling out how they fit together, and also includes justification and background (about state of the art), which I think would be better to discuss separately. Also, it's stated that ECT(1) and CE are used, but not how. I think it would be good to include a figure here (Fig 1, or some version)

[BB] OK I've caved to your demands, esp. given I've batted off a number of similar comments in the past.

But first, I've added a Document Roadmap section at the end of the Introduction to explain that the way the document is written deliberately starts with assertion not explanation. Then the rationale follows later. Some readers prefer to just be told. Others prefer to know why. This serves both. It's done this way, because, if the reader doesn't have the whole picture, they can get lost in all the justification before the big picture has become clear.

Then I've started the Architecture Overview with a para that gives the 1-minute explanation based around the 3 parts.
Then I've changed the order of the 3 parts, to put the host first, 'cos scalable CC is the thing that creates the low latency. And I've moved the explanation of the problem with Classic and why scalable CC solves it from the terminology section to here.

I won't paste it all here, I'll post it in a few hours time anyway.

I haven't moved Fig 1 to here. Excuse: This is a symptom of the document having originally been written with the DualQ as /the/ architecture. Then we made FQ and DualQ equally important, but still had Fig 1 in this section to illustrate the DualQ side. Then we were asked to explain the DualQ and FQ solutions more fully (in §4.2) rather than relying on the dualq draft, and we were asked to move the DualQ figure down into §4.1.

Section 4.2

Fig 1

The '1' between the functions Mark and Scheduler could be explained?

[BB]
s/Then the scheduler can serve the L4S queue with priority/
  /Then the scheduler can serve the L4S queue with priority (denoted by the '1' on the priority input)/

That's common scheduler notation apparently.

I found S4.2 a bit hard to parse. I think my issue is that the intro para implies that bullets are about different per flow options. Bullets b and c are about this, but bullet a is (I think) a solution without per flow, and is actually the preferred approach. Bullet a also includes explanation of the dual queue that would apply to any solution.

Perhaps: re-write the intro para, especially delete the "Nevertheless" sentence, make bullet (a) main text and explain dual q etc. Then say there are different per flow options, and start bullet (a) by stating it is the option without per-flow.

[BB] OK, we had obviously confused you (so presumably others as well): I've added this sentence to introduce the table, and I've added a heading at the start of each bullet:

"The following bullets describe the known arrangements: a) the DualQ Coupled AQM with an L4S AQM in one queue coupled from a Classic AQM in the other; b) Per-Flow Queues with an instance of a Classic and an L4S AQM in each queue; c) Dual queues with per-flow AQMs, but no per-flow queues:"

I've also added a reference to the Linux patch that adds the necessary classification to the FQ_CoDel qdisc in order to implement (b).

BTW, there isn't a "preferred approach", from the IETF's point of view.

Section 8.3

Is "catastrophically" the best word?

[BB] Yes ;)

Thank you for all these. Some were quite hard, but necessary.

Cheers

Bob

Best wishes

Phil

-----Original Message-----

From: tsvwg <[email-masked]><mailto:[email-masked]> On Behalf Of Wesley Eddy

Sent: 29 July 2021 17:18

To: [email-masked]<mailto:[email-masked]>

Subject: [tsvwg] start of WGLC on L4S drafts

- Architecture:

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf%2F&data=04%7C01%7Cphilip.eardley%40bt.com%7Cc5762b3bea68444699cb08d997f2d052%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637707891399563827%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=b%2FVGMG4fErHp9nUDEyOelTiiAu7lI1ce5RYGUY8t2bM%3D&reserved=0>

.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-

arch%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed467

1142f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637

631723313409962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ

QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=fATwA9c

bixSYOkh3vM2IugJ29oPEpvdgEIxbhAx6lMw%3D&amp;reserved=0

- DualQ:

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf%2F&data=04%7C01%7Cphilip.eardley%40bt.com%7Cc5762b3bea68444699cb08d997f2d052%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637707891399573785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=J0PzmCZq%2FIRj8CvHjAv7IucZC0y3ZnO4%2B8lG5g9nG7k%3D&reserved=0>

.org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-

coupled%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed

4671142f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C

637631723313419949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL

CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PR8Rj

3%2BmElcld7NBD7TxOgdkkvEyvbjda2qeUANy%2FAY%3D&amp;reserved=0

- ECN ID:

https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf%2F&data=04%7C01%7Cphilip.eardley%40bt.com%7Cc5762b3bea68444699cb08d997f2d052%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637707891399573785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=J0PzmCZq%2FIRj8CvHjAv7IucZC0y3ZnO4%2B8lG5g9nG7k%3D&reserved=0>

.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-

id%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed46711

42f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C63763

1723313419949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj

oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=oAg1Pi24Z

Uc4lZK9zwtXA95tm%2F%2B5Y9V2Fz8pEmBMv9g%3D&amp;reserved=0

The WGLC will last through 4 weeks from today, and then we'll see what to do

next.  Please submit any comments you have on these to the TSVWG list in that

timeframe.

The chairs are considering a possible virtual interim following the close in order to

work through feedback received.

The work on the L4S operational guidance draft is continuing in parallel, but that draft

is not being last called yet.

--

Bob Briscoe                               http://bobbriscoe.net/<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=04%7C01%7Cphilip.eardley%40bt.com%7Cc5762b3bea68444699cb08d997f2d052%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637707891399583744%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qSBGphsdXqw2IVawmmWOZ5AInFDNGKSkHladyJxfNQE%3D&reserved=0>

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 17 Nov 2021 19:35:21 +0000

Just catching up on the email threads - and "yes" these appear to h`ve 
been addressed. I'll review at a later stage,

Gorry

On 07/11/2021 22:02, Bob Briscoe wrote:
>> I therefore would support *as an individual* progression of this 
>> draft for publication as EXP.
>>
>> Best wishes,
>>
>> Gorry
>> (as individual in tsvwg)
>>
>> This appendix is informative, and as such I think we should be 
>> careful to not imply this as expressing requirements.
>>
>>   - is it possible to express this differently and avoid any 
>> ambiguity as the document receives more review?

-- 
G. Fairhurst, School of Engineering

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Nov 2021 13:39:22 +0000

Gorry, Pete, Koen, list,
See [BB2]

On 07/11/2021 21:24, Bob Briscoe wrote:
> [element]' is. Classification for L4S might be much earlier in the 
> path. How about
>
> This is actually rather meaningless, because if a network element 
> wants to alter any of the identifiers, it doesn't have to be part of 
> "the process of including additional traffic with L4S". So the other 
> identifiers." But I've left it with the MUST NOT, so as not to 
> radically alter the draft. 

[BB2] I've had offlist pushback from Koen about the latest change to 
deal with your concern here. 
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19#section-5.4.1.1
He also disagrees with the non-normative alternative I suggested of 
"...wouldn't normally be expected to alter non-L4S identifiers" because 
that's not necessarily true for Diffserv. He proposes that saying 
nothing about altering non-L4S identifiers would be best, and all the 
co-authors now agree with that.

However, Koen has also said that there should be a "MUST NOT" to stop 
anyone thinking they can include additional traffic in the L queue by 
altering the ECN field to ECT(1) based on other non-ECN identifiers.

So for the para at the end of "5.4.1.1.  Inclusion of Additional Traffic 
with L4S" 
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id#section-5.4.1.1
the new proposal for the list to consider is:

CURRENT:

    The process of including additional traffic with L4S only involves
    reading identifiers such as those itemized above.  It MUST NOT alter
    these non-ECN identifiers, so that they survive for any potential use
    later on the network path.

PROPOSED:
    In order to include non-L4S packets in the L queue, a network node
    MUST NOT alter Not-ECT or ECT(0) in the IP-ECN field to an L4S
    identifier. This ensures that these codepoints survive for any
    potential use later on the network path.

> this section from 'operator' to 'network node'.

[BB2] Koen also pushed back on the above change in "5.4.1.2. Exclusion 
of Traffic From L4S Treatment". He pointed out that the approach ought 
to be consistent with the similar but converse position in "2.5.1.1.  
Requirements in Unexpected Cases" in the DualQ draft: 
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled#section-2.5.1.1

The principle here is to deal with queue classification separately from 
AQM classification, but to recommend rather than mandate where possible, 
because operator-specific exclusion of traffic is meant to be for 
flexibility, so we shouldn't prevent flexibility unless important for 
interop.

So, if a network node that supports L4S classifies an ECT(1) packet into 
a Classic queue (due to some other non-ECN identifier), the output from 
the AQM in the Classic queue ought to still be compatible with an L4S 
congestion control.

    A network node that supports L4S but excludes certain L4S packets 
from L4S treatment MUST still apply marking or dropping that is 
compatible with an L4S congestion response. For instance, it could 
either drop such packets with the same likelihood as Classic packets or 
it could ECN-mark them with a likelihood appropriate to L4S traffic 
(e.g. the coupled probability in a DualQ coupled AQM) but aiming for the 
Classic delay target. It MUST NOT ECN-mark such packets with a Classic 
marking probability, which could confuse the sender.

Bob

---

From: Pete Heist <[email-masked]>
Date: Fri, 19 Nov 2021 20:44:07 +0100

Hi Bob/list,

I'm just asking kindly that since I oppose using ECT(1) as the L4S-ID,
and there aren't any wording changes that would change that firm
position, please remove from this thread. Thanks...

Pete

On Fri, 2021-11-19 at 13:39 +0000, Bob Briscoe wrote:
> > > unable to work across a policy enforcement point or some other
> > > device that otherwise modifies the other packet headers. 
> > > 
> > [BB] Note that this is not a nit - it is a correction to a
> > normative statement. 
> > Also, there is no definition of what the extent of an 'L4S
> > forwarding [element]' is. Classification for L4S might be much
> > earlier in the path. How about 
> > 
> > these non-ECN identifiers, so that they survive for any potential
> > use later on the network path." 
> > 
> > wants to alter any of the identifiers, it doesn't have to be part
> > of "the process of including additional traffic with L4S". So the
> > other alternative would be to remove any normative text from this
> > para, and just say "The process ... wouldn't normally be expected
> > to alter these identifiers." But I've left it with the MUST NOT, so
> > as not to radically alter the draft. 
> 
> deal with your concern
> here.https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19#section-5.4.1.1
> "...wouldn't normally be expected to alter non-L4S identifiers"
> because that's not necessarily true for Diffserv. He proposes that
> saying nothing about altering non-L4S identifiers would be best, and
> all the co-authors now agree with that.
> 
> So for the para at the end of "5.4.1.1.  Inclusion of Additional
> Traffic with
> L4S"https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id#section-5.4.1.1
> > I've also changed the subject of the other normative requirements
> > in this section from 'operator' to 'network node'.
> 
> [BB2] Koen also pushed back on the above change in "5.4.1.2. 
> Exclusion of Traffic From L4S Treatment". He pointed out that the
> approach ought to be consistent with the similar but converse
> position in "2.5.1.1.  Requirements in Unexpected Cases" in the DualQ
> draft:
> The principle here is to deal with queue classification separately
> from AQM classification, but to recommend rather than mandate where
> possible, because operator-specific exclusion of traffic is meant to
> be for flexibility, so we shouldn't prevent flexibility unless
> important for interop.
> 
> So, if a network node that supports L4S classifies an ECT(1) packet
> into a Classic queue (due to some other non-ECN identifier), the
> output from the AQM in the Classic queue ought to still be compatible
> with an L4S congestion control.
> 
> either drop such packets with the same likelihood as Classic packets
> or it could ECN-mark them with a likelihood appropriate to L4S
> traffic (e.g. the coupled probability in a DualQ coupled AQM) but
> aiming for the Classic delay target. It MUST NOT ECN-mark such
> packets with a Classic marking probability, which could confuse the
> sender.
> 
> 
> Bob
> 
>