Subject: Re: [tsvwg] start of WGLC on L4S drafts

From: Bob Briscoe <[email-masked]>
Date: Tue, 12 Oct 2021 23:02:45 +0100

Sebastian,

Sorry - just noticed I missed this email at the time. See [BB]...

On 21/09/2021 21:54, Sebastian Moeller wrote:
> Hi Bob,
>
>
>
>> On Sep 20, 2021, at 18:40, Bob Briscoe <[email-masked]> wrote:
>>
>> Alex, Gorry, [Sry, just discovered I didn't click end on this]
>>
>> In "all of a user's applications can shift" the word "can" is important.
>> Similarly, in "The L4S architecture is intended to enable all Internet applications to transition" the word "enable" is important.
>>
>> Neither statement was meant to imply that all applications would move, or should move, just that they would all be _able_ to.
>> These statements are primarily intended as counterpoints to the commonly held view that low latency comes from priority scheduling (e.g. via differentiated service) so low latency can only be provided for a subset of traffic, because if all packets requested higher priority, there would be no difference to any packet's latency.
> 	[SM] Given that at the core of the L4S reference AQM there sits a (conditional) priority scheduler*, I fail to see how L4S' low latency does not come from priority scheduling? Also in L4S low latency can only be provided to a subset of traffic, namely flows that are well behaved and ECT(1)-marked (given the shared queue design, one bad apple in the L-queue can affect latency/throughput for all flows). I am not sure whether L4S really is contrary to the "commonly held view" here, sure the details differ a bit, but at its core it seems rather similar to traditional techniques for low latency.

[BB] You say 'L4S low latency can *only* be provided to a subset' (my 
emphasis), but the word 'only' is wrong.
That is the point. L4S low latency /can/ be provided to all traffic.

Yes, L4S low latency /is/ only provided to flows that are well-behaved 
and ECT(1) marked, but all traffic /can/ behave smoothly without losing 
out, and all traffic /can/ use ECT(1) without losing out.

The priority scheduler really is not at the core.
* Consider an L4S FQ scheduler like FQ-CoDel with the ce_threshold 
parameter applied to ECT(1) traffic. There is no priority scheduling but 
all traffic can still have very low latency.
* Similarly, an L4S DualQ Coupled AQM can work with any scheduler that 
provides isolation, for instance the Time-Shifted FIFO. Scheduling 
priority is not 'at the core', 'cos it's not even necessary. We only 
ended up choosing a priority scheduler because it gave slightly better 
tail latency.

The real core of L4S is scalable congestion control. That is the 
difference that allows even capacity-seeking traffic to also maintain 
consistently very low latency. It's easier to see in the L4S FQ example, 
that it's what keeps a flow's latency down. Not the scheduler.

If TCP had been designed with a scalable congestion control from the 
start, we would have had L4S with no need for a scheduler at all. The 
scheduler is primarily a transition mechanism, to isolate from Classic 
traffic. But no traffic /needs/ to be Classic, it's just how "stuff 
happened".

Bob

>
> Regards
> 	Sebastian
>
>
> *) https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19:
> "In this case it would not be appropriate to call the queue an L4S
>     queue, because it is shared by L4S and non-L4S traffic.  Instead it
>     will be called the low latency or L queue.  The L queue then offers
>     two different treatments:
>
>     o  The L4S treatment, which is a combination of the L4S AQM treatment
>        and a priority scheduling treatment;
>
>     o  The low latency treatment, which is solely the priority scheduling
>        treatment, without ECN-marking by the AQM."
>
>
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-10:
> "The tension between prioritizing L4S and coupling the
>         marking from the Classic AQM results in approximate per-flow
>         fairness.  To protect against unresponsive traffic in the L4S
>         queue taking advantage of the prioritization and starving the
>         Classic queue, it is advisable not to use strict priority, but
>         instead to use a weighted scheduler (see Appendix A of [I-D.ietf-tsvwg-aqm-dualq-coupled])."
>
> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled
> "Classic traffic needs to build a large queue to prevent under-
>     utilization.  Therefore a separate queue is provided for L4S traffic,
>     and it is scheduled with priority over the Classic queue.  Priority
>     is conditional to prevent starvation of Classic traffic."
>
> I ignore the yarn spun in those drafts, why this prioritization would only result in latency but not longer-time-scale throughput prioritization, because Pete's data show that for quite a number of realistic traffic patterns DualQ fails to arbitrate capacity equitably between the two queues.
>
>
>
>> I've made a note to myself to make it clear that this is not an intention that all applications will transition, rather it is an intention not to prevent any application from transitioning.
>>
>>
>> Bob
>>
>> On 02/09/2021 19:26, [email-masked] wrote:
>>> Hi Gorry, tsvwg;
>>>
>>>
>>> See inline [AB]
>>>
>>>
>>>
>>> On Thursday, August 12, 2021, 11:23:49 AM GMT+1, Gorry Fairhurst<[email-masked]>  wrote:
>>>
>>> ---
>>> 1b) Issue: I’d also quibble with: “so that all of a user's applications
>>> can shift to it when their stack is updated.“ - again, is the word “all”
>>> really necessary or correct here?
>>> ---
>>>
>>> [AB]
>>> Not  answering your point, but related:
>>>
>>> L4s-arch has at the very start the following: “The L4S architecture is intended to enable   _all_ Internet applications to transition away from congestion control algorithms that cause queuing delay, to a new class of  congestion controls that induce very little queuing, aided by  explicit congestion signaling from the network. “
>>>
>>>
>>>   There are two quite different end states here:
>>>
>>> * An internet in which there are, end-to-end, two classes of traffic on an ongoing basis.
>>>
>>> * An internet in which nearly all traffic has transitioned to L4S, remaining non-L4S traffic being supported on a backwards-compatibility basis
>>>
>>> As I understand it, if L4S is deployed, either of these could be possible end states. At present it is not clear that all use cases could migrate (eg, long RTT traffic)
>>>
>>>
>>> It seems like it would be a good idea for the draft to make clear:
>>> * That either of these end states are possible
>>> * What the formal position of the WG would be if the drafts are accepted, IE what decision has been made (or left for later) regarding the desired end state.
>>>
>>>
>>> regards,
>>>
>>> Alex
>>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 13 Oct 2021 09:25:09 +0200

Hi Bob,

more bellow, prefixed [SM2]

> On Oct 13, 2021, at 00:02, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> [BB] You say 'L4S low latency can *only* be provided to a subset' (my emphasis), but the word 'only' is wrong.

	[SM2] Well low latency "can" be provided by a dumb FIFO then, as long as the aggregate ingress traffic never exceeds the egress capacity... this is similarly unhelpful as your "can provide". 

> Yes, L4S low latency /is/ only provided to flows that are well-behaved and ECT(1) marked, but all traffic /can/ behave smoothly without losing out, and all traffic /can/ use ECT(1) without losing out.
> 

	[SM2] My words were "at the core of the L4S reference AQM" and I maintain that that is strictly true, DualQ being:
a) the reference AQM for L4S
b) the only L4S AQM that has an open reference implementation for Linux
c) has seen some modicum of testing by others than the developers

I maintain, that DualQ is at its core a (weighed) priority scheduler.

> * Consider an L4S FQ scheduler like FQ-CoDel with the ce_threshold parameter applied to ECT(1) traffic. There is no priority scheduling but all traffic can still have very low latency.

	[SM2] Well, show me a) an operational reference implementation of such an AQM and b) sufficient testing data to demonstrate that it works under a broad set of realistic traffic conditions. Sure, I would assume an FQ based AQM will have fewer downsides that DualQ, but I would still want to see data. Plus, as long as DualQ is L4S's reference implementation that will be deployed in scale by cable-ISPs it does really matter that there might be better L4S-AQM designs possible, does it?

> * Similarly, an L4S DualQ Coupled AQM can work with any scheduler that provides isolation, for instance the Time-Shifted FIFO. Scheduling priority is not 'at the core', 'cos it's not even necessary. We only ended up choosing a priority scheduler because it gave slightly better tail latency.

	[SM2] Well, I am talking about the selected reference AQM for L4S, if you drop the DualQ draft and replace is by  Time-Shifted FIFO I will have a look at that AQM and discuss that. But at the current time DualQ is the relevant AQM, since it is the only one you are currently trying to standardize.

> 
> The real core of L4S is scalable congestion control. That is the difference that allows even capacity-seeking traffic to also maintain consistently very low latency. It's easier to see in the L4S FQ example, that it's what keeps a flow's latency down. Not the scheduler.

	[SM2] I would have assumed, if that was the core that the L4S effort would have started with a draft for such a scalable congestion controller, instead TCP Prague came as an after thought relatively late in the game... 

> If TCP had been designed with a scalable congestion control from the start, we would have had L4S with no need for a scheduler at all. The scheduler is primarily a transition mechanism, to isolate from Classic traffic. But no traffic /needs/ to be Classic, it's just how "stuff happened".

	[SM2] Realistically, a considerable fraction of internet traffic is going t stay on classic congestion controller for a long time, so I fail to see how this is relevant for the near and intermediate future. Especially given the demonstrated failure modes of TCP Prague, where realistic traffic patterns result in abysmal throughput...

Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 14 Oct 2021 01:48:54 +0100

Sebastian, see [BB2] (2 response)

On 13/10/2021 08:25, Sebastian Moeller wrote:
> 	[SM2] Well, show me a) an operational reference implementation of such an AQM and b) sufficient testing data to demonstrate that it works under a broad set of realistic traffic conditions. Sure, I would assume an FQ based AQM will have fewer downsides that DualQ, but I would still want to see data.

[BB2]  We published all the test data for the ce-threshold variant of 
FQ_CoDel alongside the DualQ results in July 2019 when I posted this 
link to this list:
     https://bobbriscoe.net/pubs.html#DCttH_TR
The FQ-CoDel tests with ce-threshold enabled for ECT(1) traffic are in 
the right-hand column of all the pages of plots at the end of the paper.
The Evaluation section explains the setup of all the tests plotted at 
the end, and provides commentary on the results.

This reveals to me that you have never read this material. I think you 
need to justify on this list why you have not read the primary 
performance evaluation material about L4S, given everything you have 
said about L4S performance.

You will note from Fig 7 that the tail latency of FQ-L4S under load from 
a mix of short and long flows is pretty good, but not quite as good as 
the DualQ. That's inevitable because when an approach gives priority to 
'new' flows, there will always be runs of 'new' packets that cause 
packets in the 'old' queues to have to wait longer than they would in a 
FIFO. Put another way, if you allow some other packet than the first in 
to be the first out, then you have to delay the first in, which is often 
counterproductive when the scalable congestion controls are collectively 
ensuring that nearly all packets are 'new' and any 'old' queues are 
extremely short anyway.

> Plus, as long as DualQ is L4S's reference implementation that will be deployed in scale by cable-ISPs it does really matter that there might be better L4S-AQM designs possible, does it?

[BB2] I won't respond any further to everything above this point, 
because it has just descended into two opinions about what is the core 
idea of a technology.

But I will correct two possible misunderstandings about what is being 
standardized here.

#1/ DualQ and FQ are equally applicable for L4S. That change was made to 
the l4s-arch and ecn-l4s-id drafts in Feb 2020. The fact that there is a 
draft about a DualQ solution and not an FQ draft is partly because the 
FQ one is so similar to FQ_CoDel that it only needs a short sentence 
describing the difference.

Nonetheless, this is not a spectator sport. If you or anyone else on 
this list wants to prove that an FQ-L4S is better, or as good, or 
whatever, go ahead and try to repeat or improve on the results given 
above. A draft about FQ-L4S would also be extremely quick to write - 
probably less than a page, not including all the top and tail guff.

#2/ And there is a difference between standardization and implementation:
The DualQ Coupled AQM draft does not (experimentally) standardize a WRR 
scheduler. Everything in the appendix about DualPI2 (and the other 
appendix about Curvy RED) is informative. The aqm-dualq-coupled draft 
experimentally standardizes a framework for coupling AQMs across two 
queues (the first line of the introduction says this). The words chosen 
to be capitalized (i.e. normative) have been carefully selected to be as 
concrete as possible without limiting choice.

The only normative statements about the scheduler are:
"The scheduler draining the two queues MUST give L4S packets priority 
over Classic, although priority MUST be bounded in order not to starve 
Classic traffic. The scheduler SHOULD be work-conserving."
This is deliberately written to apply to the TS-FIFO, and all sorts of 
other possible schedulers.

Regards

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 14 Oct 2021 11:15:24 +0200

Bob,

[SM3] below.

> On Oct 14, 2021, at 02:48, Bob Briscoe <[email-masked]> wrote:
> 
> [BB2]  We published all the test data for the ce-threshold variant of FQ_CoDel alongside the DualQ results in July 2019 when I posted this link to this list:
> The FQ-CoDel tests with ce-threshold enabled for ECT(1) traffic are in the right-hand column of all the pages of plots at the end of the paper.
> The Evaluation section explains the setup of all the tests plotted at the end, and provides commentary on the results.

	[SM3] To get to my points:
a) an operational reference implementation: where is the repository of your modified fq_codel?
b) sufficient testing: I have argued before that the set of conditions under which you test is to narrow to be representative for the wider internet
	also note: that paper tests using DCTCP as transport protocol, which is not what is on the table IIUC for internet-wide use, how about rerunning those tests with TCP Prague*?

So how does citing DCttH_TR help making your point in addressing my concerns?

*) Sure TCP Prague is also neither finished nor meeting its design goals yet, but at least it is one available protocol implementation that can be used for testing if the L4S experiment should start. 

> 
> This reveals to me that you have never read this material.

[SM3] Well, I guess I must have imagined that I posted https://mailarchive.ietf.org/arch/msg/tsvwg/pCPX4rhSSCuIKMRZRYF4zhji6PA/ to ask a question about missing details in that very paper, I guess...
         Also see as of yet unanswered question regarding that still not peer reviewed paper under https://mailarchive.ietf.org/arch/msg/tsvwg/undNo81qGisaS7x-6iuLfJvXmvU/

	Having read something, does not mean that one agrees with the content, and I have remarked multiple times, that the conditions tested in that paper are not really representative for what is to be expected in the wider internet, but a repetition of conditions under which we already know the current L4S design and implementation works reasonably well. The mistake IMHO is to deduce from this large collection of decent results under specific limited conditions that L4S design and implementation will work well under most conditions relevant on the open internet. A Hypothesis that has been clearly revealed as too optimistic by Pete and Jonathan's measurements.

> I think you need to justify on this list why you have not read the primary performance evaluation material about L4S, given everything you have said about L4S performance.

	[SM3] @ chairs, I consider this to be out of line from Bob, I have spend more time actually reading L4S material than many on this list, as evidenced by the fact that I understood that the 34ms average RTT-to-CDN claim in the current DualQ draft is based on Bob misunderstanding his primary data source.
@Bob: what do you intend to do about that problem? Will you at least acknoledge that there is an issue or will you continue to ignore this?
See https://mailarchive.ietf.org/arch/msg/tsvwg/gwGO6fFCGTcryRMjn5kn9FNTSvA/ and https://mailarchive.ietf.org/arch/msg/tsvwg/yNWAPHhgVX3fou3xugtVjXZ0F3U/ for why the RIPE-ATLAS numbers are not showing what you argue they show. 

I am a bit puzzled that you seem to ignore this issue. But I was beginning to think anyway that the PI2 parameters paper just tries to retroactively justify the already committed-on 15ms classic delay and is not an attempt to cleanly figure out what reference delay would be suitable/ideal. 

> You will note from Fig 7 that the tail latency of FQ-L4S under load from a mix of short and long flows is pretty good, but not quite as good as the DualQ. That's inevitable because when an approach gives priority to 'new' flows, there will always be runs of 'new' packets that cause packets in the 'old' queues to have to wait longer than they would in a FIFO. Put another way, if you allow some other packet than the first in to be the first out, then you have to delay the first in, which is often counterproductive when the scalable congestion controls are collectively ensuring that nearly all packets are 'new' and any 'old' queues are extremely short anyway.

[SM3] Well in academia, we call this a strawman, you pit your DualQ AQM that was under development and tuning for several years against a "quick hack" to LInux fq_codel (using a function that was designed for use with DCTCP in data centers with extremely low RTTs, and not longer RTT path over the internet), if at all it shows how well fq_codel does without any specific tuning*.... 
	Whether boosting new flows is a win or not, can and should not be measured at the AQM as queueing delay, but that is something to be measured E2E, because it is not tailored for satisfying the AQM statistics but to deliver "better" performance to the end-user. This is also the method by which fq_codel/cake "reward" well behaving applications, the sparser the traffic pattern the more likely it is to receive the sparse/new-flow boost, which (in an fq_codel/cake world) gives a direct incentive for senders to pace their packets, as pacing results in the highest relative sparseness.... 

*) At the very least I would have expected some level of exploration of the marking threshold parameter to see how it affects both throughput and queuing delay variation... 

> [BB2] I won't respond any further to everything above this point, because it has just descended into two opinions about what is the core idea of a technology.

	[SM3] Bob, I am looking at the drafts that you are trying to get approved, on the AQM side there is only the DualQ draft (and that is also close to what cablelabs intends to deploy soonish), so please do not bring up TS-FIFO and friends as these simply are not relevant today. The DualQ draft mostly is concerned with the DualQ weighted priority scheduler, but does offer some sections about alternative designs. I also assume that out of your possible design space you seected the best candidate and that there is a reason why current DualQ won against CurvyRed and TS-FIFO.

> 
> But I will correct two possible misunderstandings about what is being standardized here.
> 
> #1/ DualQ and FQ are equally applicable for L4S. That change was made to the l4s-arch and ecn-l4s-id drafts in Feb 2020. The fact that there is a draft about a DualQ solution and not an FQ draft is partly because the FQ one is so similar to FQ_CoDel that it only needs a short sentence describing the difference.
> 
> Nonetheless, this is not a spectator sport. If you or anyone else on this list wants to prove that an FQ-L4S is better, or as good, or whatever, go ahead and try to repeat or improve on the results given above. A draft about FQ-L4S would also be extremely quick to write - probably less than a page, not including all the top and tail guff.
> 

	[SM3] Sure, but a standard without an existing robust and reliable implementation is mostly just words... hence the idea of rough consensus and WORKING CODE. The code is the proof of the pudding.

> The DualQ Coupled AQM draft does not (experimentally) standardize a WRR scheduler.

	[SM3] But sch_dualq in your Linux repository does.

> Everything in the appendix about DualPI2 (and the other appendix about Curvy RED) is informative. The aqm-dualq-coupled draft experimentally standardizes a framework for coupling AQMs across two queues (the first line of the introduction says this). The words chosen to be capitalized (i.e. normative) have been carefully selected to be as concrete as possible without limiting choice.

	[SM3] Yes, I consider that to be a misfeature of your drafts in general, you try to leave way to much freedom for implementation, instead of doing the research before hand what does and what does not work and standardize only what works.

> "The scheduler draining the two queues MUST give L4S packets priority over Classic, although priority MUST be bounded in order not to starve Classic traffic. The scheduler SHOULD be work-conserving."

	[SM3] Which fits very well with my "Given that at the core of the L4S reference AQM there sits a (conditional) priority scheduler*" counter to your claim "counterpoints to the commonly held view that low latency comes from priority scheduling". If your design mandates priority of one class over another then your low latency class's low latency comes from that priority scheduler... 

> This is deliberately written to apply to the TS-FIFO, and all sorts of other possible schedulers.

	[SM3] Again the working code component that you use to "demonstrate" that your ideas are feasible is the DualQ. I can understand why you want to jettison that thing quickly (as it simply does not deliver on your promises), but that leaves you without a working implementation, so just with "words" (and IMHO over the three drafts way to many of those).

Sebastian

---

From: Luca Muscariello <[email-masked]>
Date: Fri, 15 Oct 2021 19:02:56 +0200

On Wed, Oct 13, 2021 at 12:03 AM Bob Briscoe <[email-masked]> wrote:

> >> Similarly, in "The L4S architecture is intended to enable all Internet
> applications to transition" the word "enable" is important.
> >>
> >> Neither statement was meant to imply that all applications would move,
> or should move, just that they would all be _able_ to.
> >> These statements are primarily intended as counterpoints to the
> commonly held view that low latency comes from priority scheduling (e.g.
> via differentiated service) so low latency can only be provided for a
> subset of traffic, because if all packets requested higher priority, there
> would be no difference to any packet's latency.
> >       [SM] Given that at the core of the L4S reference AQM there sits a
> (conditional) priority scheduler*, I fail to see how L4S' low latency does
> not come from priority scheduling? Also in L4S low latency can only be
> provided to a subset of traffic, namely flows that are well behaved and
> ECT(1)-marked (given the shared queue design, one bad apple in the L-queue
> can affect latency/throughput for all flows). I am not sure whether L4S
> really is contrary to the "commonly held view" here, sure the details
> differ a bit, but at its core it seems rather similar to traditional
> techniques for low latency.
>

I do not think this is true in this specific case.
There is an advantage in terms of goodput for applications to use
loss-based congestion control and create bigger queues (in
absence of isolation).
Unless these applications are policed one way or the other
it is hard to create incentives to move towards protocols that
generate little queuing. Isolation is one such policy as the reward
is immediate.

There is also a major implication between priority or scheduling
as an isolation mechanism: for the latter there is no need to
allocate a bit in the network header to identify traffic types.

> > I ignore the yarn spun in those drafts, why this prioritization would
> only result in latency but not longer-time-scale throughput prioritization,
> because Pete's data show that for quite a number of realistic traffic
> patterns DualQ fails to arbitrate capacity equitably between the two queues.
> >
> >
> >
> >> I've made a note to myself to make it clear that this is not an
> intention that all applications will transition, rather it is an intention
> not to prevent any application from transitioning.
> >>
> >>
> >> Bob
> >>
> >>> L4s-arch has at the very start the following: “The L4S architecture is
> intended to enable   _all_ Internet applications to transition away from
> congestion control algorithms that cause queuing delay, to a new class of
> congestion controls that induce very little queuing, aided by  explicit
> congestion signaling from the network. “
> >>>
> >>>
> >>> * An internet in which nearly all traffic has transitioned to L4S,
> remaining non-L4S traffic being supported on a backwards-compatibility basis
> >>>
> >>> As I understand it, if L4S is deployed, either of these could be
> possible end states. At present it is not clear that all use cases could
> migrate (eg, long RTT traffic)
> >>>
> >>>
> >>> * What the formal position of the WG would be if the drafts are
> accepted, IE what decision has been made (or left for later) regarding the
> desired end state.
> >>>
> >>>
> >>> regards,
> >>>
> >>> Alex
> >>>
> >> --

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 17 Oct 2021 00:38:20 +0100

Luca, see [BB]

On 15/10/2021 18:02, Luca Muscariello wrote:
>
>
> On Wed, Oct 13, 2021 at 12:03 AM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Sebastian,
>
>     >> On Sep 20, 2021, at 18:40, Bob Briscoe <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >>
>     >> Similarly, in "The L4S architecture is intended to enable all
>     Internet applications to transition" the word "enable" is important.
>     >>
>     >> Neither statement was meant to imply that all applications
>     would move, or should move, just that they would all be _able_ to.
>     commonly held view that low latency comes from priority scheduling
>     (e.g. via differentiated service) so low latency can only be
>     provided for a subset of traffic, because if all packets requested
>     higher priority, there would be no difference to any packet's latency.
>     >       [SM] Given that at the core of the L4S reference AQM there
>     sits a (conditional) priority scheduler*, I fail to see how L4S'
>     low latency does not come from priority scheduling? Also in L4S
>     low latency can only be provided to a subset of traffic, namely
>     flows that are well behaved and ECT(1)-marked (given the shared
>     queue design, one bad apple in the L-queue can affect
>     latency/throughput for all flows). I am not sure whether L4S
>     really is contrary to the "commonly held view" here, sure the
>     details differ a bit, but at its core it seems rather similar to
>     traditional techniques for low latency.
>

[BB] This may be true in the short term for non-latency sensitive bulk 
traffic, especially over variable capacity (e.g. radio) links where 
buffered packets can fill newly available capacity while the e2e 
congestion control catches up.

Nonetheless, there is also the unscalable aspect of loss-based 
congestion control where, as flow rate scales, it takes longer and 
longer (minutes) to recover after each loss [RFC3649]. As was pointed 
out in footnote 6 of [Jac88], the noise sensitivity gets worse as flow 
rate scales. Meaning, even if there's a very low level of transmission 
losses or new flows arriving, a flow keeps getting knocked down by each 
noise event, while it's slowly trying to regain its previous position.

It's possible that some new approach will get out of that unscalable 
rut, but it's been quite resistant to solutions so far. BBRv2 has 
already had to reinstate a capped dependence on loss probability, in 
order to remain backward compatible with the legacy of loss-based 
congestion controls. The setting of that cap will have to keep being 
reduced in order to scale flow rate, so it is trapped in the same noise 
sensitivity problem.

I don't know whether bulk flows will prefer to continue to use the 
classical approach with more buffering, or the scalable approach with 
frequent control signals. I'm just saying it's not as clear-cut as you 
make out.

[Jac88] Van Jacobson and Michael J. Karels. Congestion
Avoidance and Control. Technical report, Lau-
rence Berkeley Labs, November 1988. (a slightly
modified version of the original published at SIG-
COMM in Aug’88)

> generate little queuing. Isolation is one such policy as the reward

[BB] When you have scalable congestion controls that fill capacity with 
consistently very low queueing, you can just use them (without policing) 
whether you need low latency, high throughput or both.

As a fairly close analogy, can you imagine any individual or company 
gaming a PIE AQM to induce more latency on their own flow, in order to 
impose more latency on others in the same household?

Policing hasn't even been necessary for 'TCP-friendliness' between 
application flows within a household, where there is bandwidth to gain. 
I don't know whether policing will be necessary for 'latency 
friendliness'. Maybe it will at first while there are people around who 
don't understand that there's nothing to gain.

My prediction, FWIW: I suspect that some policing within some paths 
(e.g. in DOCSIS links) will be enough at first to keep globally deployed 
code disciplined, whether or not it's actually running over a path with 
policing. I doubt policing will be necessary longer term, but we will 
have it if we need it. So nothing is hanging on who predicts this 
correctly.

Or are you saying that we should never allow anyone to even attempt to 
have low latency in shared queues in case it requires the complexity of 
per-flow policing?
But instead we should only ever allow per-flow queuing for low latency, 
which is somewhat more complex than per-flow policing anyway.
I hope you appreciate that avoiding the possible risk of a little 
complexity by requiring more complexity is broken logic.

> allocate a bit in the network header to identify traffic types.

[BB] I don't understand what you have in mind here. Pls explain more fully.

Bob

>
>
>
>
>
>     Bob
>
>     >
>     > Regards
>     >       Sebastian
>     >
>     >
>     > *)
>     https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19
>     <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19>:
>     https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-10
>     <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-10>:
>     <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled>
>     > I ignore the yarn spun in those drafts, why this prioritization
>     would only result in latency but not longer-time-scale throughput
>     prioritization, because Pete's data show that for quite a number
>     of realistic traffic patterns DualQ fails to arbitrate capacity
>     equitably between the two queues.
>     >
>     >
>     >
>     intention that all applications will transition, rather it is an
>     intention not to prevent any application from transitioning.
>     >>
>     >>
>     >> Bob
>     >>
>     >> On 02/09/2021 19:26, [email-masked]
>     <mailto:[email-masked]> wrote:
>     >>> Hi Gorry, tsvwg;
>     >>>
>     >>>
>     >>> See inline [AB]
>     >>>
>     >>>
>     >>>
>     >>> On Thursday, August 12, 2021, 11:23:49 AM GMT+1, Gorry
>     Fairhurst<[email-masked] <mailto:[email-masked]>> wrote:
>     >>>
>     >>> ---
>     >>> L4s-arch has at the very start the following: “The L4S
>     architecture is intended to enable   _all_ Internet applications
>     to transition away from congestion control algorithms that cause
>     queuing delay, to a new class of congestion controls that induce
>     very little queuing, aided by  explicit congestion signaling from
>     the network. “
>     >>>
>     >>>
>     >>> * An internet in which nearly all traffic has transitioned to
>     L4S, remaining non-L4S traffic being supported on a
>     backwards-compatibility basis
>     >>>
>     >>> As I understand it, if L4S is deployed, either of these could
>     be possible end states. At present it is not clear that all use
>     cases could migrate (eg, long RTT traffic)
>     >>>
>     >>>
>     >>> * What the formal position of the WG would be if the drafts
>     are accepted, IE what decision has been made (or left for later)
>     regarding the desired end state.
>     >>>
>     >>>
>     >>> regards,
>     >>>
>     >>> Alex
>     >>>
>     >> --
>     >> Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/>
>     >>
>
>     -- 
>     Bob Briscoe http://bobbriscoe.net/ <http://bobbriscoe.net/>
>

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 17 Oct 2021 05:16:57 +0300

> On 17 Oct, 2021, at 2:38 am, Bob Briscoe <[email-masked]> wrote:
> 
> [BB] This may be true in the short term for non-latency sensitive bulk traffic, especially over variable capacity (e.g. radio) links where buffered packets can fill newly available capacity while the e2e congestion control catches up.
> 
> Nonetheless, there is also the unscalable aspect of loss-based congestion control where, as flow rate scales, it takes longer and longer (minutes) to recover after each loss [RFC3649]. As was pointed out in footnote 6 of [Jac88], the noise sensitivity gets worse as flow rate scales. Meaning, even if there's a very low level of transmission losses or new flows arriving, a flow keeps getting knocked down by each noise event, while it's slowly trying to regain its previous position.
> 
> It's possible that some new approach will get out of that unscalable rut, but it's been quite resistant to solutions so far. BBRv2 has already had to reinstate a capped dependence on loss probability, in order to remain backward compatible with the legacy of loss-based congestion controls. The setting of that cap will have to keep being reduced in order to scale flow rate, so it is trapped in the same noise sensitivity problem. 

I think it's very disingenuous for you to quote only a paper from 1988, at the very dawn of TCP congestion control research, in support of this argument.  At that time, only the Reno algorithm was in use, and Reno does have widely-acknowledged scalability problems, especially with respect to random loss on high-BDP paths.  It's actually rather strange, given that fact, that TCP Prague uses only the Reno growth function while claiming to be a "scalable transport".

However the deployed state of the art today is CUBIC (as you very well know), and CUBIC is considerably more scalable than Reno (in the upward direction, at least), especially in the face of low levels of random loss.  On the research side there is also Scalable TCP, an MIMD algorithm that is even less sensitive to random loss than CUBIC.

This sensitivity to random loss is easily shown to be a function of the cycle time of the congestion-control loss-response sawtooth; iff loss events occur more frequently than this cycle time, the average cwnd will trend downwards.  In Reno (and Prague), this cycle time is proportional to RTT and to the square of throughput.  In CUBIC, the cycle time is roughly proportional to throughput and mostly independent of RTT.  In Scalable TCP, the cycle time is proportional to RTT and independent of throughput.  (These generalisations assume constant segment size, etc.)

> I don't know whether bulk flows will prefer to continue to use the classical approach with more buffering, or the scalable approach with frequent control signals. I'm just saying it's not as clear-cut as you make out.

They will likely choose the option that is universally available and yields the maximum goodput.  In the present state of L4S, that is clearly "Classic".

> As a fairly close analogy, can you imagine any individual or company gaming a PIE AQM to induce more latency on their own flow, in order to impose more latency on others in the same household?

The goal would not be to increase latency.  The goal would either be to selfishly increase throughput for their own traffic (which could be done using Relentless TCP at the sender side, exploiting exactly the same effect with a drop-only AQM as L4S would impose in a shared ECN AQM), or to increase packet loss and thus degrade service for the other traffic (which could be done using a simple flood, driving the AQM into a high dropping probability).  These scenarios are so common, and so easy to imagine, that they should be part of a standard security analysis - and neither of them would result in substantially higher latency on the path, since in both cases it is assumed that the AQM effectively controls the queue delay to its target.

DualQ offers attackers a very easy way to gain higher throughput for themselves.  With some traffic, it also offers an easy to to increase packet loss as high as 100% for a particular type of target traffic.  These are problems over and above those of PIE, which are relatively minor by comparison.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 17 Oct 2021 12:20:04 +0200

Bob,

one point below, [SM4]

> This reveals to me that you have never read this material. I think you need to justify on this list why you have not read the primary performance evaluation material about L4S, given everything you have said about L4S performance.
> 
> The DualQ Coupled AQM draft does not (experimentally) standardize a WRR scheduler. Everything in the appendix about DualPI2 (and the other appendix about Curvy RED) is informative. The aqm-dualq-coupled draft experimentally standardizes a framework for coupling AQMs across two queues (the first line of the introduction says this). The words chosen to be capitalized (i.e. normative) have been carefully selected to be as concrete as possible without limiting choice.
> 

	[SM4] From https://bobbriscoe.net/projects/latency/dctth_journal_draft20190726.pdf:
"We have also tried what we call a Time-Shifted FIFO scheduler [37]. It selects the packet with the earliest arrival timestamp, after subtracting a constant time-shift to favour L4S packets. It performs nearly as well as WRR despite its simplicity. However, it tends to allow long bursts of delay to leak from the C to the L queue, so it is not used further in this paper."

	In other words, TS-FIFO is even worse at meeting the promises in the L4S RFCs than DualQ, and was hence "abandoned*" (and not fully characterized in the paper), and yet you bring it up here as example for what exactly? IMHO this demonstrates that the K4S drafts are way too broad, instead on focussing on actual solutions to the stated challenge they offer a multitude of not-good-enough attempts. How about doing one thing/proposing one solution that works well instead?

Sebastian

*) There is a whole ensemble of solutions to L4S existing problems that where explored for some time and then abandoned without having reached a satisfying solution:

a) rfc3168 AQM detection
b) paced chirping
c) RTT de-biasing
d) L4S AQMs that actually meet the promises in the drafts

impressive how long this effort is still maintained in the light of constant failure of its design and implementation to meet its promises.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 22 Oct 2021 10:26:28 +0100

Jonathan,

On 17/10/2021 03:16, Jonathan Morton wrote:
>> On 17 Oct, 2021, at 2:38 am, Bob Briscoe<[email-masked]>  wrote:
>>

[BB] Can you please stop preceding every discussion with completely 
unnecessary and nasty accusations.

You can still read my words above. They are about "the unscalable aspect 
of loss-based congestion control", which started with TCP congestion 
control back in 1988, and how it has continued through to BBRv2 today. 
The 1988 footnote is the relevant reference for the noise sensitivity 
scaling problem. When you jumped to the conclusion that my omission of 
CUBIC from the succession was part of some disingenuous conspiracy, 
before jumping to an insult, you could have first tried to think of 
other innocent explanations - like that I was giving an email summary of 
a longer argument.

The fuller account in the L4S architecture draft quantifies how CUBIC is 
now reaching its scaling limit (search for the words: 'less unscalable').

Ironically, for the next rev, I have had to change the numbers because 
CUBIC's unscalability is ~14% worse than I thought - due to a mistake 
about the numbers in the Linux implementation of CUBIC pointed to by 
Neal on the tcpm list. (Also, if you're following tcpm, you will have 
noticed that I have written up corrections to the paper (not mine) that 
the CUBIC RFC relied on for CUBIC's Reno-friendliness formula. This is 
in support of the CUBIC RFC moving to standards track.)

Tom Kelly's Scalable TCP (capital S) turned out to be unstable due to 
synchronization effects. A number of the people on this list and in 
iccrg (including me) were involved as BIC, CUBIC, Scalable TCP, etc  
were being worked through (around the annual PFLDnet workshops). In DCs, 
CUBIC (and Compound) have been and are being superseded by DCTCP. DCTCP 
and L4S are a continuation of that work to evolve towards scalable 
congestion controls.

> The goal would not be to increase latency.  The goal would either be to selfishly increase throughput for their own traffic (which could be done using Relentless TCP at the sender side, exploiting…

[BB] Yes, you have repeatedly made this argument. And it has been 
repeatedly pointed out in response that such throughput increasing 
attacks are already possible in any of the non-FQ AQMs or FIFOs in the 
Internet. This is not L4S-specific.

There is no work in the IETF to address this more general problem, 
because it's not been a significantly /exploited/ problem for the last 
33 years. It is an advantage of FQ that it solves this /potential/ 
problem, but it is also important to understand why it has not been a 
significant /actual/ problem anyway. Most likely because inter-customer 
scheduling limits the attack to intra-customer (going right back to 
[RFC970]). This means that any attempt by a server to improve its own 
performance will be at the expense of other usage in the same household. 
And therefore likely to become unpopular and die away.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 22 Oct 2021 11:48:02 +0100

Luca

> allocate a bit in the network header to identify traffic types.

Setting aside the point about identifiers, the first half of your 
sentence made me wonder whether we have been tripping you (and others) 
up by using incorrect words for the scheduler between L4S and Classic.

For the DualPI2 Linux implementation we recommend weighted DRR, but it's 
also OK to use what we call time-shifted FIFO (TS-FIFO).
* TS-FIFO is just FIFO, but with the internal arrival time of L4S 
packets artificially shifted back.
    So I think you would call TS-FIFO 'priority'.
* However, I think you are calling weighted DRR 'scheduling' but not 
'priority' (?)

For the normative text, we tried to generalize to cover both as follows:

      "The scheduler draining the two queues MUST give L4S packets
    priority over Classic, although priority MUST be bounded in order
    not to starve Classic traffic. The scheduler SHOULD be work-conserving"

Link to this text in context: 
https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#name-functional-requirements

If you reckon that, strictly, the word 'priority' excludes WDRR, how about:

    "The scheduler draining the two queues MUST isolate the latency of
    the L4S queue from the latency of the classic queue, but the
    isolation MUST be conditional on always allowing Classic traffic to
    make some progress. The scheduler SHOULD be work-conserving."

________
Incidentally, as a follow-up to the earlier explanation of how the WDRR 
doesn't normally intervene, here's a plot of the L queue length when 
there's two flows: 1 L4S vs 1 Classic.
https://bobbriscoe.net/tmp/qs_1-1_timeline_10s-eps-converted-to.pdf
As you can see, the L4S flow keeps the L queue empty half the time 
(because of its response to the coupled ECN marking). The Classic queue 
(not shown) is always larger, so it always fills the gaps. So, it's the 
norm for C to borrow from underused L.

If the L queue left fewer or shorter gaps, the C queue would grow and 
cause the C AQM to increase ECN marking, which is coupled across to the 
L queue. So the L4S flow would open up more gaps to return to the 
position shown.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 Oct 2021 12:23:21 +0100

Ermin,

Thanks for the comments on the drafts during WGLC. See [BB] inline for 
responses. Where there is no response, I just took the comment as read.

On 16/08/2021 08:59, Ermin Sakic wrote:
> Hi Wesley, all,
>
> L4S proposals are an important step towards enabling low-latency transmissions for remote control, AR/VR, and online gaming applications.
> I support progressing the drafts to RFC status.
>
> I have attached a few comments below after re-reading the latest versions of the three drafts.
> My apologies if some of the comments came up in discussion earlier.
>
> Best wishes,
> Ermin Sakic, NVIDIA
>
> - draft-ietf-tsvwg-l4s-arch-10:
>          - Figure 1: Is there a specific reason why the “mark/drop” action is not denoted for L4S queue compared to “mark”-only? Section 5.1 of draft-ietf-tsvwg-ecn-l4s-id-19 states that “…L4S marking treatment MUST begin applying drop to L4S traffic until the overload episode has subsided…”.

[BB] On balance I think it's best for a high level figure like this to 
show only the behaviour expected under normal circumstances. This is why 
it doesn't show drop as well, because Not-ECT traffic is not classified 
into the L queue.

But, you're right that it's important to emphasize at every opportunity 
that the L queue has to apply drop if overloaded. So I've added the 
following to the text describing the figure:
     "If either queue becomes persistently overloaded, ECN marking is 
disabled, as recommended in Section 7 of [RFC3168] and Section 4.2.1 of 
[RFC7567]. Then both queues introduce the same level of drop (not shown 
in the figure)."

>          - Section 8.2: “…if it is leads…” -> “…if it leads…”
> - draft-ietf-tsvwg-ecn-l4s-id-19:
>          - Section 1 Introduction: “…responses to them result..” -> “…responses to markings result…”

[BB] Yes, 'them' wasn't clear, but this highlights another problem - 
it's not clear what the "control signals" are. So rather than use a 
different word ("markings"), how about referring back to 'these 
signals', and adding 'ECN' to the first occurrence.
     "...the much more frequent ECN control signals, and the finer 
responses to these signals, result in..."

>          - Section 1.1: “The explanation is summarised without the maths in the L4S architecture document” -> I suggest referencing to the specific section number of draft-ietf-tsvwg-l4s-arch here.
>          - Section 7.1: “TCP. QUIC…” -> “TCP, QUIC…”
>          - Appendix B: “tranport-layer” -> “transport-layer”
> - draft-ietf-tsvwg-aqm-dualq-coupled-16:
>          - Figure 1: As mentioned above, I’d expect to see an optional drop as the outcome of forwarding to the L traffic queue in addition to the “ECN marking” action.

Again, I'd rather not complicate the "opening lesson of the tutorial" 
with exceptions that come later. But it would be good to highlight that 
exceptions will be coming later. So instead of adding to the diagram, 
I've added to the first para of the section, which already warns of one 
case where the diagram might not be authoritative:

"Also, the schematic only illustrates operation under normally expected 
circumstances; behaviour under overload or with operator-specific 
classifiers is deferred to Section 2.5.1.1"

These edits will appear alongside those for the other recent comments, 
before the draft deadline on Monday.

And I've ack'd your contribution in all three drafts. Thank you again.

Bob

>
> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
> Sent: Thursday, 29 July 2021 18:18
> To: [email-masked]
> Subject: [tsvwg] start of WGLC on L4S drafts
>
> External email: Use caution opening links or attachments
>
>
> This message is starting a combined working group last call on 3 of the L4S drafts:
>
> - Architecture: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-arch%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a7V1lmfJi%2Fbs2H2mZIr3QhveYsPj3QzPguBFDWsy%2BC0%3D&amp;reserved=0
>
> - DualQ:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-coupled%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=PwO0BFTNKv2c4ZbGoQZEvxfLg7NIG9mw8oWtNqdwsjs%3D&amp;reserved=0
>
> - ECN ID: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-id%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=QpBxzuleCeXpxzoBR4oEJWANLk7v1Vvo%2Ft4H8d3kZz0%3D&amp;reserved=0
>
> The WGLC will last through 4 weeks from today, and then we'll see what to do next.  Please submit any comments you have on these to the TSVWG list in that timeframe.
>
> The chairs are considering a possible virtual interim following the close in order to work through feedback received.
>
> The work on the L4S operational guidance draft is continuing in parallel, but that draft is not being last called yet.
>
>
> -----------------------------------------------------------------------------------
> NVIDIA GmbH
> Wuerselen
> Amtsgericht Aachen
> HRB 8361
> Managing Director: Karen Theresa Burns
>
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information.  Any unauthorized review, use, disclosure or distribution
> is prohibited.  If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 Oct 2021 17:18:21 +0100

Gorry,

Thanks for your (as always) diligent review. Unless I've responded below 
with [BB], consider them accepted as written...

On 12/08/2021 11:30, Gorry Fairhurst wrote:
> I have reviewed DualQ Coupled AQMs for  L4S
>
> I re-reviewed: draft-ietf-tsvwg-aqm-dualq-coupled-16 in the WGLC. As 
> an inidvidual, I support publication as a PS, and have some comments 
> below on -16, that
>
> 1)
> The abstract is unnecessarily long. It does not need to provide this 
> level of background information. I’d suggest the editors shorten this.

[BB] I've compressed it down to the size of the first of the two paras. 
I haven't run this past my co-authors, so you might see some (hopefully 
minor) changes in the version that gets submitted.

This specification defines a framework for coupling the Active Queue 
Management (AQM) algorithms in two queues intended for flows with 
different responses to congestion. This provides a way for the Internet 
to transition from the scaling problems of standard TCP Reno-friendly 
('Classic') congestion controls to the family of 'Scalable' congestion 
controls. These achieve consistently very Low queuing Latency, very Low 
congestion Loss and Scaling of per-flow throughput (L4S) by using 
Explicit Congestion Notification (ECN) in a modified way. Until the 
Coupled DualQ, these L4S senders could only be deployed where a 
clean-slate environment could be arranged, such as in private data 
centres. The coupling acts like a semi-permeable membrane; isolating the 
sub-millisecond average queuing delay and zero congestion loss of L4S 
from Classic latency and loss; but pooling the capacity between any 
combination of Scalable and Classic flows with roughly equivalent 
throughput per flow. The DualQ achieves this indirectly, without having 
to inspect transport layer flow identifiers and without compromising the 
performance of the Classic traffic. The solution has low complexity and 
requires no configuration for the public Internet.

> ---
> 2) The 'Low Loss" part of the name denotes
> - This has mixed quote marks. In other places, double quotes are used.
> ---
> 3) There appears to be some repetition of topics in section 1.4 
> between this and the L4S arch specifications. This just gives more to 
> read. Please do consider removing duplicate examples/discussion.

[BB] The "Problem Outline" section was rewritten over Drafts 09 & 10 
(both July 2019) to be specific to the Problem the DualQ solves, and 
other material was moved to l4s-arch.

The Scope section was also rewritten. It has a couple of paras putting 
the DualQ in the wider context of scalable congestion controls, then 
refers to l4s-arch for details. The rest of the scope section is 
specific to the DualQ. Yes, it does pick some pieces out of l4s-arch 
(e.g. most likely deployment scenario for the dualQ). But the aim was to 
give implementers who are only interested in the DualQ what they need.

This might not be the way you would have chosen to decide which material 
to include or exclude, but I hope you'll understand that this means I 
really don't want to go through the exercise of rewriting these sections 
again,... except to address specific concerns.

> ---
> 4) Typo - additional space:
> “[RFC5681] .”
> ---
> 5) Unpublished Individual ID as a Ref explaining a “MUST”:
> “The two queues can be part of a larger
>    queuing hierarchy [I-D.briscoe-tsvwg-l4s-diffserv].”
> - Since this relates a to a requirement, I do not think this is 
> appropiate. The reference provided is out of date, and does not appear 
> to be progressing. Could this reference be removed?

[BB] You're right the normative section about Functional Requirements is 
not the right place for this. I've deleted the whole second sentence.

Nonetheless, an implementer will need to have an idea how this DualQ 
fits into their existing queuing structures, so this sentence is worth 
keeping somewhere. I've stuck it at the end of §2.4 "Overall DualQ 
Coupled AQM Structure". I've also kept the ref, 'cos there is no other 
place an implementer could go for this information. But it's not in a 
normative section any more, and I've added some fluff to make it clearer 
that this reference is just early ideas:
     "The two queues could optionally be part of a larger queuing 
hierarchy, such as the initial example ideas in 
[I-D.briscoe-tsvwg-l4s-diffserv]."

BTW, as I understand the chairs' position on the l4s-diffserv draft, it 
is 'on hold' while the WG focuses on the main L4S drafts. That means 
there will be no request for adoption (yet), but it also hasn't been 
refused adoption.

> ---
> 6) Unpublished Individual ID as a Ref:
> “and Voice-Admit
>    service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]),
> - I’d prefer to leave this without a REF that to refer to an 
> individual draft submission.

[BB] I've replaced all 4 refs in this para with one ref to §5.4.1 of 
ecn-l4s-id. We can then confine this discussion of which refs are 
appropriate to that doc.

CURRENT:
For instance addresses of specific applications or hosts (see 
[I-D.ietf-tsvwg-ecn-l4s-id 
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-ecn-l4s-id>]), 
specific Diffserv codepoints such as EF (Expedited Forwarding) and 
Voice-Admit service classes (see [I-D.briscoe-tsvwg-l4s-diffserv 
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.briscoe-tsvwg-l4s-diffserv>]), 
the Non-Queue-Building (NQB) per-hop behaviour [I-D.ietf-tsvwg-nqb 
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-nqb>] 
or certain protocols (e.g. ARP, DNS).

PROPOSED:
For instance addresses of specific applications or hosts; specific 
Diffserv codepoints such as EF (Expedited Forwarding), Voice-Admit or 
the Non-Queue-Building (NQB) per-hop behaviour; or certain protocols 
(e.g. ARP, DNS) (see Section 5.4.1 of [I-D.ietf-tsvwg-ecn-l4s-id 
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-ecn-l4s-id>]). 

> ---
> 7) NiT: “The scheduler SHOULD be work-conserving.”
> - This “SHOULD” introduces a design decision: Please add a one 
> sentence explanation what happens if it is not.

[BB]
PROPOSED:
The scheduler SHOULD be work-conserving, or otherwise close to work 
conserving, given Classic service will often rely on borrowing from the 
L4S service.

> ---
> 8) Dead reference (used multiple times)
> "if the packet is Not-ECT, the appropriate action depends on
>          whether some other function is protecting the L queue from
>          misbehaving flows (e.g. per-flow queue
>          protection [I-D.briscoe-docsis-q-protection] or latency
>          policing)"
> "Therefore per-flow policing
>    (e.g. [I-D.briscoe-docsis-q-protection]) needs to be separable from a
>    basic AQM, as an option under policy control."
> "However, the addition of DOCSIS per-flow
>    Queue Protection [I-D.briscoe-docsis-q-protection] turns this into
>    'delay on saturation"
> - The tracker shows draft-briscoe-docsis-q-protection has expired and 
> is no longer active. What is the current status of this ID?

[BB] The ISE has indicated it is OK in principle to proceed to 
publication. I have a couple of edits to it and I just haven't 
prioritized them. Sorry.

> ---
> 9) RFC2119 usage:
> " The following
>    parameters MAY be operator-configurable, e.g. to tune for non-
>    Internet settings:"
> - To me this does not seem like a standards interoperability 
> statement. Do we need RFC2119 language, would it be OK to just say:
> " The following
>    parameters can be operator-configurable, e.g. to tune for non-
>    Internet settings:"

[BB] I thought this was one of the few occasions where MAY is 
appropriate. But obviously you don't. If an implementer doesn't follow 
this MAY, their device won't be able to interoperate well with 
congestion controls in non-Internet environments. But whether this is a 
concern depends on the market their device is intended to address.
To me that's nearly word-for-word what RFC2119 says about the intended 
use of MAY.

I'll leave it for now, and see what you say in response.

> ---
> 10)RFC2119 usage:
> "The maximum queue
>       delay per queue per interval MAY also be recorded."
> - I'd sa surely a system can record anything? --  So this does not 
> seem to need RFC2119 language, would it be OK to just say "can"... I'd 
> also be interested to see a sentence that helps me understand *why* 
> this might be useful as a log entry.

[BB] I've added "..., to aid diagnosis of faults and anomalous events"
Again, this is really about interop - not data plane, but management 
plane - interop with typical monitoring systems. So there's the same 
argument for a 'MAY' as above.
Again, I'll leave in the MAY and see if you disagree.

> ---
> 11) NiT:
> "Curvy RED can be used as is when
>    the RTT range to be supported is limited, otherwise an adaptation
>    mechanism is required."
> /required/needed/ - for avoidance of doubt?

[BB] I'm impressed that you've found this in the depths of an appendix!
Agree with 'needed' in this first case, but...

> ---
> 12) NiT:
> "Priority of L4S is required to be conditional to avoid total
>    starvation of Classic by heavy L4S traffic."
> /required/needed/ - for avoidance of doubt?

[BB] Here, 'required' is actually correct. So I've added a pointer to 
the requirement:
     "Priority of L4S is required to be conditional (see Section 2.5.1) 
to avoid..."

Thank you very much.

Bob

> --------------
>>
>> On 29.07.21, 18:18, "tsvwg on behalf of Wesley 
>> Eddy"<[email-masked] on behalf of [email-masked]>  wrote:
>>
>>     This message is starting a combined working group last call on 3 
>> of the
>>     L4S drafts:
>>
>>     - 
>> Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>>
>>     - DualQ:
>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
>>
>>     - ECN 
>> ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>>
>>     The WGLC will last through 4 weeks from today, and then we'll see 
>> what
>>     to do next.  Please submit any comments you have on these to the 
>> TSVWG
>>     list in that timeframe.
>>
>>     The chairs are considering a possible virtual interim following the
>>     close in order to work through feedback received.
>>
>>     The work on the L4S operational guidance draft is continuing in
>>     parallel, but that draft is not being last called yet.
>>
>>
>

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 23 Oct 2021 18:30:35 +0200

Hi List,

so this post of Bob's highlights an inconsistency in the DualQ draft:
for one he writes:

"The coupling acts like a semi-permeable membrane; isolating the sub-millisecond average queuing delay and zero congestion loss of L4S from Classic latency and loss; but pooling the capacity between any combination of Scalable and Classic flows with roughly equivalent throughput per flow. The DualQ achieves this indirectly, without having to inspect transport layer flow identifiers and without compromising the performance of the Classic traffic."

but later it is stated

"Priority of L4S is required to be conditional to avoid total starvation of Classic by heavy L4S traffic."

Which really just admits that the indirect "coupling" mechanism from the introduction does not work as promised, as without such an emergency scheduling get-out-of-jail card, L4S traffic can very much compromise "the performance of the Classic traffic" when both compete in the DualQ. 

Regards
	Sebastian

> On Oct 23, 2021, at 18:18, Bob Briscoe <[email-masked]> wrote:
> 
> Gorry, 
> 
> Thanks for your (as always) diligent review. Unless I've responded below with [BB], consider them accepted as written...
> 
>> I re-reviewed: draft-ietf-tsvwg-aqm-dualq-coupled-16 in the WGLC. As an inidvidual, I support publication as a PS, and have some comments below on -16, that 
>> 
>> 1) 
>> The abstract is unnecessarily long. It does not need to provide this level of background information. I’d suggest the editors shorten this. 
> 
> [BB] I've compressed it down to the size of the first of the two paras. I haven't run this past my co-authors, so you might see some (hopefully minor) changes in the version that gets submitted.
> 
> This specification defines a framework for coupling the Active Queue Management (AQM) algorithms in two queues intended for flows with different responses to congestion. This provides a way for the Internet to transition from the scaling problems of standard TCP Reno-friendly ('Classic') congestion controls to the family of 'Scalable' congestion controls. These achieve consistently very Low queuing Latency, very Low congestion Loss and Scaling of per-flow throughput (L4S) by using Explicit Congestion Notification (ECN) in a modified way. Until the Coupled DualQ, these L4S senders could only be deployed where a clean-slate environment could be arranged, such as in private data centres. The coupling acts like a semi-permeable membrane; isolating the sub-millisecond average queuing delay and zero congestion loss of L4S from Classic latency and loss; but pooling the capacity between any combination of Scalable and Classic flows with roughly equivalent throughput per flow. The DualQ achieves this indirectly, without having to inspect transport layer flow identifiers and without compromising the performance of the Classic traffic. The solution has low complexity and requires no configuration for the public Internet.
> 
>> --- 
>> 3) There appears to be some repetition of topics in section 1.4 between this and the L4S arch specifications. This just gives more to read. Please do consider removing duplicate examples/discussion. 
> 
> [BB] The "Problem Outline" section was rewritten over Drafts 09 & 10 (both July 2019) to be specific to the Problem the DualQ solves, and other material was moved to l4s-arch.
> 
> The Scope section was also rewritten. It has a couple of paras putting the DualQ in the wider context of scalable congestion controls, then refers to l4s-arch for details. The rest of the scope section is specific to the DualQ. Yes, it does pick some pieces out of l4s-arch (e.g. most likely deployment scenario for the dualQ). But the aim was to give implementers who are only interested in the DualQ what they need.
> 
> This might not be the way you would have chosen to decide which material to include or exclude, but I hope you'll understand that this means I really don't want to go through the exercise of rewriting these sections again,... except to address specific concerns. 
> 
>> --- 
>> - Since this relates a to a requirement, I do not think this is appropiate. The reference provided is out of date, and does not appear to be progressing. Could this reference be removed? 
> 
> [BB] You're right the normative section about Functional Requirements is not the right place for this. I've deleted the whole second sentence.
> 
> Nonetheless, an implementer will need to have an idea how this DualQ fits into their existing queuing structures, so this sentence is worth keeping somewhere. I've stuck it at the end of §2.4 "Overall DualQ Coupled AQM Structure". I've also kept the ref, 'cos there is no other place an implementer could go for this information. But it's not in a normative section any more, and I've added some fluff to make it clearer that this reference is just early ideas:
>     "The two queues could optionally be part of a larger queuing hierarchy, such as the initial example ideas in [I-D.briscoe-tsvwg-l4s-diffserv]."
> 
> BTW, as I understand the chairs' position on the l4s-diffserv draft, it is 'on hold' while the WG focuses on the main L4S drafts. That means there will be no request for adoption (yet), but it also hasn't been refused adoption.
> 
>> --- 
>> - I’d prefer to leave this without a REF that to refer to an individual draft submission. 
> 
> [BB] I've replaced all 4 refs in this para with one ref to §5.4.1 of ecn-l4s-id. We can then confine this discussion of which refs are appropriate to that doc.
> 
> CURRENT:
> For instance addresses of specific applications or hosts (see [I-D.ietf-tsvwg-ecn-l4s-id]), specific Diffserv codepoints such as EF (Expedited Forwarding) and Voice-Admit service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]), the Non-Queue-Building (NQB) per-hop behaviour [I-D.ietf-tsvwg-nqb] or certain protocols (e.g. ARP, DNS). 
> 
> PROPOSED:
> For instance addresses of specific applications or hosts; specific Diffserv codepoints such as EF (Expedited Forwarding), Voice-Admit or the Non-Queue-Building (NQB) per-hop behaviour; or certain protocols (e.g. ARP, DNS) (see Section 5.4.1 of [I-D.ietf-tsvwg-ecn-l4s-id]). 
> 
>> --- 
>> - This “SHOULD” introduces a design decision: Please add a one sentence explanation what happens if it is not. 
> 
> [BB]
> PROPOSED:
> The scheduler SHOULD be work-conserving, or otherwise close to work conserving, given Classic service will often rely on borrowing from the L4S service.
> 
>> --- 
>> - The tracker shows draft-briscoe-docsis-q-protection has expired and is no longer active. What is the current status of this ID? 
> 
> [BB] The ISE has indicated it is OK in principle to proceed to publication. I have a couple of edits to it and I just haven't prioritized them. Sorry.
> 
>> --- 
>> 9) RFC2119 usage: 
>> " The following 
>> - To me this does not seem like a standards interoperability statement. Do we need RFC2119 language, would it be OK to just say: 
>> " The following 
> [BB] I thought this was one of the few occasions where MAY is appropriate. But obviously you don't. If an implementer doesn't follow this MAY, their device won't be able to interoperate well with congestion controls in non-Internet environments. But whether this is a concern depends on the market their device is intended to address. 
> To me that's nearly word-for-word what RFC2119 says about the intended use of MAY.
> 
>> - I'd sa surely a system can record anything? --  So this does not seem to need RFC2119 language, would it be OK to just say "can"... I'd also be interested to see a sentence that helps me understand *why* this might be useful as a log entry. 
> 
> Again, this is really about interop - not data plane, but management plane - interop with typical monitoring systems. So there's the same argument for a 'MAY' as above. 
> [BB] Here, 'required' is actually correct. So I've added a pointer to the requirement:
>     "Priority of L4S is required to be conditional (see Section 2.5.1) to avoid..."
> 
> Thank you very much.
> 
> 
> Bob
> 
>> -------------- 
>>> 
>>> On 29.07.21, 18:18, "tsvwg on behalf of Wesley Eddy"<[email-masked] on behalf of [email-masked]>  wrote: 
>>> 
>>>     - Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/ 
>>> 
>>>     - DualQ: 
>>>     - ECN ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/ 
>>> 
>>>     The WGLC will last through 4 weeks from today, and then we'll see what 
>>>     to do next.  Please submit any comments you have on these to the TSVWG 
>>>     list in that timeframe. 
>>>

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 Oct 2021 10:14:29 +0100

> Which really just admits that the indirect "coupling" mechanism from the introduction does not work as promised, as without such an emergency scheduling get-out-of-jail card, L4S traffic can very m…

As we move towards trhe resolution of the WGLC, I agree it's good to 
clarify text that can be potentially misinterpretted. Could the editors 
please you align the two statements please in their next revision?

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 Oct 2021 10:35:20 +0100

On 23/10/2021 17:18, Bob Briscoe wrote:
> Gorry,
>
> Thanks for your (as always) diligent review. Unless I've responded 
> below with [BB], consider them accepted as written...
>
> [BB] I've compressed it down to the size of the first of the two 
> paras. I haven't run this past my co-authors, so you might see some 
> (hopefully minor) changes in the version that gets submitted.
>
> different responses to congestion. This provides a way for the 
> Internet to transition from the scaling problems of standard TCP 
> Reno-friendly ('Classic') congestion controls to the family of 
> 'Scalable' congestion controls. These achieve consistently very Low 
> queuing Latency, very Low congestion Loss and Scaling of per-flow 
> throughput (L4S) by using Explicit Congestion Notification (ECN) in a 
> modified way. Until the Coupled DualQ, these L4S senders could only be 
> deployed where a clean-slate environment could be arranged, such as in 
> private data centres. The coupling acts like a semi-permeable 
> membrane; isolating the sub-millisecond average queuing delay and zero 
> congestion loss of L4S from Classic latency and loss; but pooling the 
> capacity between any combination of Scalable and Classic flows with 
> roughly equivalent throughput per flow. The DualQ achieves this 
> indirectly, without having to inspect transport layer flow identifiers 
> and without compromising the performance of the Classic traffic. The 
> solution has low complexity and requires no configuration for the 
> public Internet.
>
I don't think the abstract needs to claim what is achieved, only what is 
sought by the spec. Claiming what is achieved would require evidence, 
and claiming a solution is not necessary. Instead I think an RFC seeks 
to publish a spec that intends to deliver an expected outcome. From my 
perspective, I'd feel happier with changes such as:

/achieve/are designed to achieve/

/The solution has low/The DualQ AQM design has low/

> (e.g. most likely deployment scenario for the dualQ). But the aim was 
> to give implementers who are only interested in the DualQ what they need.
>
> This might not be the way you would have chosen to decide which 
> material to include or exclude, but I hope you'll understand that this 
> means I really don't want to go through the exercise of rewriting 
> these sections again,... except to address specific concerns.
>
Understood, to avoid multiple advice, I'll defer to the Shepherd and AD 
for advice here, if any is needed.
>> appropiate. The reference provided is out of date, and does not 
>> appear to be progressing. Could this reference be removed?
>
> [BB] You're right the normative section about Functional Requirements 
> is not the right place for this. I've deleted the whole second sentence.
>
> normative section any more, and I've added some fluff to make it 
> clearer that this reference is just early ideas:

OK

>
> BTW, as I understand the chairs' position on the l4s-diffserv draft, 
> it is 'on hold' while the WG focuses on the main L4S drafts. That 
> means there will be no request for adoption (yet), but it also hasn't 
> been refused adoption.
Yes - this should only be used as an example.
> <https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-dualq-coupled-17.html#I-D.ietf-tsvwg-ecn-l4s-id>]). 
OK with me.
> conserving, given Classic service will often rely on borrowing from 
> the L4S service.
>
I'm not sure the additional explanation needs to be in the RFC2119 
clause, it might actually be clearer as a separate following sentence.

The senetence seems to be going in the correct direction, but is 
"borrowing" clear in this context?

> publication. I have a couple of edits to it and I just haven't 
That's OK.
> congestion controls in non-Internet environments. But whether this is 
> a concern depends on the market their device is intended to address.
It would be nice if the sentence before says the default configuration 
parameters need to be safe for Internet deployment - I didn't check, but 
you'll know.
>> seem to need RFC2119 language, would it be OK to just say "can"... 
>> I'd also be interested to see a sentence that helps me understand 
>> *why* this might be useful as a log entry.
>
Perfect.
> Agree with 'needed' in this first case, but...
OK
>     "Priority of L4S is required to be conditional (see Section 2.5.1) 
Aha, that's good.
> Bob
Seems like you might have what you need from me to make the next revision.

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Oct 2021 13:15:30 +0100

Gorry,

Thank you again for yet another careful review.
No comment = accepted. For qualifications or push back, pls see [BB] 
inline...

On 13/08/2021 12:00, Gorry Fairhurst wrote:
> Adding to my own review, after re-reading L4S ID  ...
>
> Gorry
>
> On 12/08/2021 10:44, Gorry Fairhurst wrote:
>> I reviewed Low Latency, Low Loss, Scalable Throughput (L4S) Internet 
>> Service Architecture during the WGLC. I support publication, but I 
>> think there are issues that would need to be addressed relating to 
>> the way L4S is presented.
>>
>> Best wishes,
>>
>> Gorry
>>
>>
>> This is a review of draft-ietf-tsvwg-l4s-arch-10:
>>
>> 1a) Issue: “It is increasingly common for _all_ of a user's 
>> applications at any
>>    one time to require low delay: interactive Web, Web services, voice,
>>    conversational video, interactive video, interactive remote presence,
>>    instant messaging, online gaming, remote desktop, cloud-based
>>    applications and video-assisted remote control of machinery and
>>    industrial processes. “
>> - I generally agree on the value for a wide range of applications and 
>> sect 6.1 appears to contain a helpful list. ~ However, I disagree 
>> that the word “all” is necessary or correct in this section. I think 
>> there are cases that do not fit. Background software update, 
>> retrieval of logging data, some cases where loss protection is more 
>> important than for other traffic, etc, So I insist there are examples 
>> that do not require this. I expect I could find more examples, but 
>> I’d prefer the word “all” to be removed, to leave this as a 
>> generalisation.

[BB] This no way intends to say "all applications require low delay".
The words bracketing around the "all" are not to be overlooked... 
/Increasingly common/ and  /at any one time/.
How about this to make that point clearer:

"At any one time, it is increasingly common for /all/ of the traffic in 
a bottleneck link (e.g. a household's Internet access) to come from 
applications that prefer low delay:..."

>> ---
>> 1b) Issue: I’d also quibble with: “so that all of a user's 
>> applications can shift to it when their stack is updated.“ - again, 
>> is the word “all” really necessary or correct here?

[BB] OK. "So that a user's applications can shift..."

>> ---
>> 2) NiT
>> “The L4S
>>    approach also requires component mechanisms at the endpoints to
>>    fulfill its goal.”
>> - True, can you provide examples of what these mechanisms are (or a 
>> cross ref)? e.g., are they Protocol Mechanisms (as they are called in 
>> section 4.1)? or is this more the  to Host Mechanisms (section 4.3) ?

[BB] Yes a bit abstract isn't it. I've actually rewritten parts of the 
whole para, that were also abstract:

    L4S is designed for incremental deployment. It is possible to deploy
    the L4S service at a bottleneck link alongside the existing best
    efforts service [DualPI2Linux] so that unmodified applications can
    start using it as soon as the sender's stack is updated.  Access
    networks are typically designed with one link as the bottleneck for
    each site (which might be a home, small enterprise or mobile device),
    so deployment at either or both ends of this link should give nearly
    all the benefit in the respective direction.  With some transport
    protocols, namely TCP and SCTP, the sender has to check for suitably
    updated receiver feedback, whereas with more recent transport
    protocols such as QUIC and DCCP, all receivers have always been
    suitable.

    This document presents the L4S architecture, by describing and
    justifying the component parts and how they interact to provide the
    scalable, low latency, low loss Internet service.  It also details
    the approach to incremental deployment, as briefly summarized above.

>> ---
>> 3) Issue or reword.
>> “Although DCTCP as-is 'works' well over the public
>>       Internet, most implementations lack certain safety features that
>>       will be necessary once it is used outside controlled environments
>>       like data centres (see Section 6.4.3 and Appendix A).”
>> - If this is published by the iETF, then I think this needs to have 
>> consensus that DCTP is acceptable. I don’t now which RFC states that.
>> Later this ID states: “ DCTCP needs some safety concerns to be fixed 
>> for general use
>>    over the public Internet (see Section 4.3 of
>>    [I-D.ietf-tsvwg-ecn-l4s-id]),”
>> which seems much closer to what is needed in this section.

[BB] Ah yes, the intent was to reassure that is works well over 
wide-area RTTs. I can see how "it works over the public Internet" would 
be misconstrued.

       Although DCTCP as-is functions well over wide-area
       round trip times, most implementations lack certain safety
       features that would be necessary for use outside controlled
       environments like data centres (see Section 6.4.3 and Appendix A).

---
>> 4) NiT
>> “With
>>       Classic congestion controls, such as Reno or Cubic, because flow
>>       rate has scaled since TCP congestion control was first designed in
>>       1988, it now takes hundreds of round trips (and growing) to
>>       recover after a congestion signal (whether a loss or an ECN mark)
>>       as shown in the examples in Section 5.1 and [RFC3649].
>> “
>> - This only seems like a correct statement, if the flow is a bulk / 
>> high volume flow that is using a high capacity path? Please check and 
>> clarify.

[BB] We will certainly add
     "assuming the flow lasts long enough".
But the "hundreds of round trips and growing" statement is accurate. 
There is no assumption of a high BDP path - the first example with a 
Cubic recovery time of 4.3s (166 rounds) in Section 5.1 is *lower* than 
the the global average for a download from a CDN - at least using the 
best data I can find.

I agree that both x-references only describe the scaling, not where we 
are on the scale. Rather than putting loads of detail in the terminology 
definition, we've added the following after the example it refers to in 
§ 5.1:

    "For a feel of where the global average lone-flow download sits on
    this scale at the time of writing (2021), according to [BDPdata]
    globally averaged fixed access capacity was 103Mb/s in 2020 and
    averaged base RTT to a CDN was 25-34ms in 2019. Averaging of
    per-country data was weighted by Internet user population. So a lone
    CUBIC flow would at best take about 200 round trips (5 s) to recover
    from each of its sawtooth reductions, if the flow even lasted that
    long. This is described as 'at best' because it assume everyone uses
    an AQM, whereas In reality most users still have a bloated tail-drop
    buffer. So likely average recovery time would be at least 4x 5 s, if
    not more, because RTT under load would be at least double, and
    recovery time depends on the square of RTT."

That's why we said "hundreds". Is that reasonable?
Here's the figures:

	avg fixed downstream [Mb/s]
	avg CDN RTT under load [ms]
	avg BDP under load
	Recovery time [s]
	Recovery time [#RTTs]
Global avg
	103
	35ms
(25-34ms base RTT + avg queue - half sawtooth)
	438kB
	6.6s
	188
Example ref'd in Draft
	120
	25.5ms
(30ms max RTT - half sawtooth)
	383kB
	4.2s
	166

This assumes Cubic.
The top row has the bottom half of the sawtooth in Reno-friendly mode, 
and the top half in Cubic. There is no formula for a sawtooth in 
transition, so we've taken the average of the two: 6.8s & 6.4s.
The bottom row is fully in Reno-friendly mode still.

Reference:
[BDPdata] Briscoe, B., "PI2 Parameters", Technical Report TR-BB-2021-001 
arXiv:2107.01003 [cs.NI], July 2021, <https://arxiv.org/abs/2107.01003>.

>> ---
>> 5) NiT
>> “such as QUIC.”
>> - Reference needed for [RFC9000].

[BB] Transport protocol references like this are given when they appear 
elsewhere, but we've added this one here as well, to support the 
statement about Reno being the default CC in QUIC.

>> ---
>> 6) NiT
>> “[I-D.ietf-tsvwg-ecn-l4s-id] recommends ECT(1)”
>> draft-ietf-tsvwg-ecn-l4s-id appears to “specify” the use of ECT(1) 
>> for this purpose. Please check whether ECT(1) is only recommended.
>> ---
>> 7) Issue: TCP is a core protocol, and L4S support depends on Accurate 
>> ECN: Is accurate ECN a normative?  (see comment 22)
>> “Work to standardize and implement more
>>           accurate ECN feedback for TCP (AccECN) is in
>>           progress [I-D.ietf-tcpm-accurate-ecn],”

[BB] When I asked about this before (years ago), the WG and chairs 
agreed on it being informative.
Reasoning: AccECN should be a normative ref in a scalable CC spec that 
requires AccECN (for instance, Acc-ECN is normative in 
draft-briscoe-iccrg-prague-congestion-control).
But there is no normative statement that references AccECN in l4s-arch, 
it's always referred to informatively, like QUIC or DCCP is.
Similarly for ecn-l4s-id, but we'll discuss that in your other review.

>> ---
>> 8) NiT In section 2, ref useful:
>> “high level explanation of how FQ”
>> - Please expand and add reference for FQ.

[BB] We've expanded FQ. But there is no ref to an L4S-FQ - the only 
description is in this draft - other than the Linux patch. And where 
this draft describes it, it refers to the appropriate section of the 
FQ-CoDel RFC.

>> ---
>> 9) Typo:
>> “ The 'Low Loss” part of the name denotes”
>> 10) Issues relating to critique of diffserv:
>>
>> 10a) Making assertions about Diffserv:
>> I really don’t like this choice of words, it’s far too general a 
>> conclusion:
>> “      Also, Diffserv only works for a small subset of the traffic on a
>>       link. “
>> - Diffserv  works fine when all traffic is marked with a set of DSCPs 
>> - I hope you’d agree, but there are cases where it wouldn’t give 
>> benefit, e.g. when all traffic on a congested link/router is marked 
>> with one code point, whereas there may be benefit if all traffic on a 
>> link/router is marked with one code point, when that traffic later 
>> shares bandwidth with traffic carrying other marks.

[BB] I think this implies you'd be happier if we wrote:
      "Also, Diffserv can only provide a latency benefit if a small 
subset of the traffic on a bottleneck link requests low latency."
OK?

>> ---
>> 10b) Making assertions about Diffserv:
>> “As already explained, it is not applicable when all the
>>       applications in use at one time at a single site (home, small
>>       business or mobile device) require low latency.”
>> - See above. I really don’t like saying it is not applicable. It can 
>> be applied, and it can give benefit - I’d accept there was no benefit 
>> if the traffic was all marked the same and the resources were 
>> constrained in that part of the network.

[BB] s/is not applicable/has no effect/
OK?

>> ---
>> 10c) In security considerations, making assertions about Diffserv:
>> “Diffserv only works if some packets
>>    get less favourable treatment than others.”
>> - Again not so, the word “works” is problematic. Offers benefit is 
>> perhaps more true?

[BB] We've used "makes a difference" this time.

>> ---
>> 10d) In security considerations, making assertions about Diffserv:
>> “So Diffserv has to use
>>    traffic rate policers to limit how much traffic can be favoured.”
>> - I do not agree in this general conclusion and I think this was 
>> discussed in previous comments. This is not the place to make 
>> sweeping statements about diffserv, or how it might be used. I would 
>> strongly urge you to remove this sweeping statement.
>> ---
>> 10e) In security considerations, making assertions about Diffserv:
>> “In
>>    turn, traffic policers require traffic contracts between users and
>>    networks as well as pairwise between networks. “
>> - Does it actually need to be a contract with a user, I don’t thinks 
>> so, there are many cases where diffserv marks are applied in the 
>> network, or at an enterprise edge. This is not the document to make 
>> such statements about diffserv. I’m going to argue that you need to 
>> remove this phrase, people can decide to use L4S without having to 
>> address this.
>> ---

[BB] We've reduced all the discussion of Diffserv in the first para of 
Sec Consid's by referring back to previous discussion:

"Because the L4S service reduces delay without increasing the delay of 
Classic traffic, it should not be necessary to rate-police access to the 
L4S service. In contrast, Section 5.2 explains how Diffserv only makes a 
difference if some packets get less favourable treatment than others, 
which typically requires traffic policing, which can, in turn, lead to 
further complexity such as traffic contracts at trust boundaries."

> >>>
>
> I have now completed a review of L4S ID, I’ve now also reviewed “1.1.  
> Latency, Loss and Scaling Problems” and I saw a much more polished 
> version of the text with respect to DiffServ. To avoid doubt, here is 
> the text I saw in L4S ID::
>
>    The Diffserv architecture provides Expedited Forwarding [RFC3246], so
>    that low latency traffic can jump the queue of other traffic. If
>    growth in high-throughput latency-sensitive applications continues,
>    periods with solely latency-sensitive traffic will become
>    increasingly common on links where traffic aggregation is low. For
>    instance, on the access links dedicated to individual sites (homes,
>    small enterprises or mobile devices).  These links also tend to
>    become the path bottleneck under load.  During these periods, if all
>    the traffic were marked for the same treatment at these bottlenecks,
>    Diffserv would make no difference.  Instead, it becomes imperative to
>    remove the underlying causes of any unnecessary delay.
> ....
>
>    Active queue management (AQM) was originally developed to solve this
>    problem (and others).  Unlike Diffserv, which gives low latency to
>    some traffic at the expense of others, AQM controls latency for _all_
>    traffic in a class.
>
> The above in the L4S-ID seems to me to be an acceptable presentation 
> of the diffserv analysis and I did not find issues with this text, and 
> other mentions in L4S-ID.
>
> I wonder why the arch and L4S-ID both present this argument, ... and 
> if we really do need two sets explaining diffserv inb both documents?
>
> If this is really needed to be in both specs, could we just use the 
> same words and avoid different flavours of discussion arising from the 
> L4S ARCH document?

[BB] I hope the wording in l4s-arch is now more precise, thanks to you 
pointing out all the failings. Discussion of Diffserv in each document 
is for different reasons, one of which (the top level problem statement) 
overlaps:

l4s-arch:
§4.1 Protocol mechanism allows combination with other identifiers, 
summarising and referring to ecn-l4s-id for details
§5.2 What L4S adds to existing approaches (the problem statement, also 
summarised in a sentence in the Intro)
§8.1 Security Considerations, comparing L4S with Diffserv to illustrate 
the likelihood that L4S will need policing - now mainly refers back to §5.2

ecn-l4s-id:
§1.1: Problem statement, justifying the need for an L4S identifier (also 
summarised in a sentence in the Intro)
§5.4: Interaction of L4S with other identifiers

The main problem statement is certainly in l4s-arch (which started out 
incorporating the problem statement written for the L4S BoF).

The problem statement in ecn-l4s-id focuses in on the need for an L4S 
identifier. This needs to start with the bigger problem that L4S is 
trying to solve. I believe you're saying that you'd prefer that initial 
step of the rationale in ecn-l4s-id to be replaced by a reference to 
l4s-arch, but it then loses its logical progression. I think this is a 
question of editorial style, which I'd rather not change that at this stage.

>
> >>>
>
>> 11) Issue: Making assertions about Diffserv:
>> “   In particular, because networks tend not to trust end systems to
>>       identify which packets should be favoured over others, where
>>       networks assign packets to Diffserv classes they often use packet
>>       inspection of application flow identifiers or deeper inspection of
>>       application signatures.  Thus, nowadays, Diffserv doesn't always
>>       sit well with encryption of the layers above IP.  So users have to
>>       choose between privacy and QoS.
>> “
>> This might, or might not be true, but there are no RFCs or references 
>> sited, and I am unsure why this para is actually needed in an RFC: If 
>> it is, then it needs some specific discussio by the WG, but maybe it 
>> would be more prudent to not say this.

[BB] We have had discussion of this on the ML.

  * "Networks tend not to trust end systems to identify which
    packets..." => Diffserv bleaching discussions.
  * Which then raised the question of how operators that use Diffserv
    actually determine which packets get what.
      o You may remember I used the example of the UK ISP, PlusNet,
        which uses DPI to put traffic into different Diffserv classes.
        Also, I've pointed to home gateways that do this - I remember I
        referred to a specific AL-Lu home gateway that my
        operator-employer used to use, which communicated back to a
        regularly updated central database of traffic signatures.
      o And mobile networks use PI or DPI (see RFC8404)
  * The final logical step was discussed on the tsvwg ML during the BoF
    stages of L4S (you may recall Kevin Smith's involvement), and it's
    the inevitable conclusion of the previous steps. Also "doesn't
    always" ensures it is not overstated.
  * Also, this issue was discussed at length in the IETF more widely
    during the preparation of RFC 8404 (see §2.2.2
    <https://datatracker.ietf.org/doc/html/rfc8404#section-2.2.2>).

We've done this:
     s/they often use packet inspection of application flow identifiers/
      /they tend to use packet inspection of application flow identifiers/
And we've cited RFC8404 at the end of the sentence about encryption.

>>
>> ---
>> 12) Issue: This seems not quite true:
>>
>> “     A.  Per-flow forms of L4S like FQ-CoDel are incompatible with full
>>           end-to-end encryption of transport layer identifiers for
>>           privacy and confidentiality (e.g. IPSec or encrypted VPN
>>           tunnels), because they require packet inspection to access the
>>           end-to-end transport flow identifiers.
>> “
>> I am also unsure why this para is actually needed in this ID!
>>
>> A network flow ID can be used with FQ, and I think should be noted. I 
>> also thought this seemed slightly misleading by not mentioning that 
>> TLS over UDP still exposes the port info needed to classify in FQ. 
>> I’s suggest removing this ... it seems a well known effect of 
>> classification/scheduling on traffic aggregates.

[BB] A couple of years ago, we removed all the arguments about FQ that 
were not relevant to this doc. But we left this one because it is one of 
the main reasons why it's important to have an architecture that is a 
superset of FQ and DualQ.

Adding a network flow ID defeats the whole point of a 
confidentiality/privacy solution like IPsec ESP that deliberately 
encapsulates the transport layer flow ID to obfuscate against inferences 
made by analysing application flow behaviour.  So I'd rather not suggest 
that, as it will get thrown out again during SECDIR review.

At the end of the parenthesis we've added:
(e.g. IPSec or encrypted VPN tunnels, as opposed to TLS over UDP)

>> ---
>> 13) Ref
>> In reference to scavenger service, can you add a reference to an RFC 
>> as well as [LEDBAT_AQM]?

[BB] I've realized we should have said "e2e scavenger behaviours 
[RFC6817]".
We didn't mean "services" like the LE PHB here, which can be identified 
by FQ.

>> ---
>> 14) Query:
>> “ Indeed BBRv2 [BBRv2] uses L4S ECN...”
>> - Is this necessarily true that ECN is used, and the final spec for 
>> BBR2 specifies this? Might it be fairer to say:
>> “ Indeed BBRv2 [BBRv2] can use L4S ECN...”

[BB] "can use L4S ECN where available"

>> ---
>> 15) NiT: In 6.4.1:
>> “   L4S AQMs will not have to be deployed throughout the Internet before
>>    L4S will work for anyone.”
>> - Is this “work” for anyone? or “can benefit anyone”?

[BB] correct

>> ---
>> 16) NiT:
>> “Deployment in mesh topologies depends on how over-booked the core is.”
>> - sentence ends in “is”... which seems strange. Is “over-booked” well 
>> understood or should this be “over-provisioned” perhaps?
>> I’d suggest something more like:
>> “Deployment in a mesh topology depends on how much the core is 
>> over-provisioned.”

[BB] When I talk with network people, they use 'over-booked' quite 
frequently. I can't think of a better alternative (it's the opposite of 
over-provisioned, BTW). Searching for synonyms doesn't help, the best is 
over-subscribed which is less good. Overbooked is also in wide general 
use, e.g. the flight/hotel was overbooked. We're going to have to leave 
it as is. Except apparently it doesn't need a hyphen.

>> ---
>> 17) NiT: In 6.4.2.:
>>    “For any one L4S flow to work, it requires 3 parts to have been
>>    deployed.”
>> - what does “work” mean here? is this offer benefit or is this 
>> dysfunctional in some way unless this happens or what?

[BB] "provide benefit", particularly in the context of incentives here.

>> ---
>> 18) NiT:
>> This seems wrong from a diffserv viewpoint:
>> “they are required not to change the L4S
>>    identifier, merely treating the traffic as best efforts traffic, as
>>    they already treat traffic with ECT(1) today.”
>> - e.g. could we change to something like :
>> “they are required not to change the L4S
>>    identifier, merely treating the traffic in the same way as not-ECT 
>> traffic, as
>>    they already treat traffic with ECT(1) today.”

[BB] Good catch.
But we've changed it to: "as they might already treat ECT(1) traffic 
today." as well.

>> ---
>> 19) NiT:
>> “to police L4S traffic in order to
>>    protect competing Classic ECN traffic.”
>> - Is policing in an order - if so explain, otherwise please rephrase 
>> to “to police L4S traffic to protect competing Classic ECN traffic”

[BB] -> "so as to protect" which avoids "to ... to"

>> ---
>> 20) NiT:
>> “Their packet classifier (item 2 in Figure 1) could identify
>>    such customers against some other field (e.g. source address range)
>>    as well as ECN.”
>> /ECN/as classifying on the ECN Field/.
>> ---
>> 21) NiT:
>> “The TCP authentication option (TCP-AO [RFC5925]) can be used to
>>       detect tampering with TCP congestion feedback.”
>> -It seems strange to not also mention that QUICs use of TLS also 
>> prevent this tampering.

[BB] We've generalized as well:
Transport layer authentication such as the TCP authentication option 
(TCP-AO [RFC5925])
or QUIC's use of TLS [RFC9001] can detect any tampering with congestion 
feedback.

>> ---
>> 22)Query:  I think some references could need to be normative:
>> RFC3168
>> RFC8311
>> I-D.ietf-tcpm-accurate-ecn

[BB] We had not included any normative language at all in l4s-arch, 
being informational. We took a similar approach to the Diffserv 
architecture [RFC2475]. One could possibly argue that this sentence 
pulls in the relevant part of RFC3168 by reference:

       For L4S, the names used for the four codepoints of the 2-bit IP-
       ECN field are unchanged from those defined in [RFC3168  <https://datatracker.ietf.org/doc/html/rfc3168>]: Not ECT,
       ECT(0), ECT(1) and CE, where ECT stands for ECN-Capable Transport
       and CE stands for Congestion Experienced.

But looking at that, it needs explaining better anyway, which makes it 
clearer that the normative inclusion of RFC3168 is being done by 
ecn-l4s-id, not l4s-arch:

      "L4S uses the ECN field as an identifier [I-D.ietf-tsvwg-ecn-l4s-id]
       with the names for the four codepoints of the 2-bit IP-ECN field
       unchanged from those defined in [RFC3168]: Not ECT, ECT(0),..."

The references to RFC8311 are commentary about RFC8311, not normative. 
Whereas in ecn-l4s-id, it's definitely normative - as the authority for 
the ID to be assigned. Similarly, l4s-arch references to AccECN are 
commentary about it, not normative.

_____
Finally, this was a really useful review, which has hopefully removed 
some potential trip-wires. Thank you again.
Amazingly, I hadn't already included you in the ACKs - we have now.

Cheers

Bob

>> -----

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 25 Oct 2021 15:46:48 +0200

Hi Gorry, hi Bob,

as far as I can tell this now leaks factually wrong "averaged base RTT to a CDN" number into even more places... Even if Bob's recent opinion, that  the average RTT to the _closest_ CDN is a suitable proxy for "internet RTT" is accepted, it at least needs to be described veridically as what it is, calling that number "averaged base RTT to a CDN" does not do justice to the source that number was generated from.

Regards
	Sebastian

P.S.: I do not accept that the RTT to the closest CDN is a decent proxy for the internet, as not all content providers/ISPs work with all CDNs so it does really not matter if I sit in Lagos, Nigeria with 4ms RTT to a google CDN if the content I need sits 100ms away at the nearest CDN2 site... and even in  "rich" countries it is not guaranteed that the path to any content is equally close. If Bob would argue for average RTT to the biggest X CDNs he might have a point, but "to the closest" simply is painting too rosy a picture here.

> On Oct 25, 2021, at 14:15, Bob Briscoe <[email-masked]> wrote:
> 
> Gorry,
> 
> No comment = accepted. For qualifications or push back, pls see [BB] inline...
> 
> 
>>> I reviewed Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service Architecture during the WGLC. I support publication, but I think there are issues that would need to be addressed relating to the way L4S is presented. 
>>> 
>>> Best wishes, 
>>> 
>>> Gorry 
>>> 
>>> 
>>> 1a) Issue: “It is increasingly common for _all_ of a user's applications at any 
>>> - I generally agree on the value for a wide range of applications and sect 6.1 appears to contain a helpful list. ~ However, I disagree that the word “all” is necessary or correct in this section. I think there are cases that do not fit. Background software update, retrieval of logging data, some cases where loss protection is more important than for other traffic, etc, So I insist there are examples that do not require this. I expect I could find more examples, but I’d prefer the word “all” to be removed, to leave this as a generalisation. 
> 
> The words bracketing around the "all" are not to be overlooked... /Increasingly common/ and  /at any one time/.
> "At any one time, it is increasingly common for /all/ of the traffic in a bottleneck link (e.g. a household's Internet access) to come from applications that prefer low delay:..."
> 
> 
>>> --- 
>>> 1b) Issue: I’d also quibble with: “so that all of a user's applications can shift to it when their stack is updated.“ - again, is the word “all” really necessary or correct here? 
> 
>>> - True, can you provide examples of what these mechanisms are (or a cross ref)? e.g., are they Protocol Mechanisms (as they are called in section 4.1)? or is this more the  to Host Mechanisms (section 4.3) ? 
> 
> [BB] Yes a bit abstract isn't it. I've actually rewritten parts of the whole para, that were also abstract:
> 
>>> - If this is published by the iETF, then I think this needs to have consensus that DCTP is acceptable. I don’t now which RFC states that. 
>>> Later this ID states: “ DCTCP needs some safety concerns to be fixed for general use 
> [BB] Ah yes, the intent was to reassure that is works well over wide-area RTTs. I can see how "it works over the public Internet" would be misconstrued.
>>> - This only seems like a correct statement, if the flow is a bulk / high volume flow that is using a high capacity path? Please check and clarify. 
> 
> But the "hundreds of round trips and growing" statement is accurate. There is no assumption of a high BDP path - the first example with a Cubic recovery time of 4.3s (166 rounds) in Section 5.1 is lower than the the global average for a download from a CDN - at least using the best data I can find.
> 
> I agree that both x-references only describe the scaling, not where we are on the scale. Rather than putting loads of detail in the terminology definition, we've added the following after the example it refers to in § 5.1:
> "For a feel of where the global average lone-flow download sits on this scale at the time of writing (2021), according to [BDPdata] globally averaged fixed access capacity was 103Mb/s in 2020 and averaged base RTT to a CDN was 25-34ms in 2019. Averaging of per-country data was weighted by Internet user population. So a lone CUBIC flow would at best take about 200 round trips (5 s) to recover from each of its sawtooth reductions, if the flow even lasted that long. This is described as 'at best' because it assume everyone uses an AQM, whereas In reality most users still have a bloated tail-drop buffer. So likely average recovery time would be at least 4x 5 s, if not more, because RTT under load would be at least double, and recovery time depends on the square of RTT."
> 
> The top row has the bottom half of the sawtooth in Reno-friendly mode, and the top half in Cubic. There is no formula for a sawtooth in transition, so we've taken the average of the two: 6.8s & 6.4s.
> [BDPdata] Briscoe, B., "PI2 Parameters", Technical Report TR-BB-2021-001 arXiv:2107.01003 [cs.NI], July 2021, <https://arxiv.org/abs/2107.01003>. 
> 
> 
>>> --- 
>>> 5) NiT 
>>> “such as QUIC.” 
> [BB] Transport protocol references like this are given when they appear elsewhere, but we've added this one here as well, to support the statement about Reno being the default CC in QUIC.
> 
>>> --- 
>>> 6) NiT 
>>> draft-ietf-tsvwg-ecn-l4s-id appears to “specify” the use of ECT(1) for this purpose. Please check whether ECT(1) is only recommended. 
>>> --- 
>>> 7) Issue: TCP is a core protocol, and L4S support depends on Accurate ECN: Is accurate ECN a normative?  (see comment 22) 
> [BB] When I asked about this before (years ago), the WG and chairs agreed on it being informative.
> Reasoning: AccECN should be a normative ref in a scalable CC spec that requires AccECN (for instance, Acc-ECN is normative in draft-briscoe-iccrg-prague-congestion-control).
> But there is no normative statement that references AccECN in l4s-arch, it's always referred to informatively, like QUIC or DCCP is.
> [BB] We've expanded FQ. But there is no ref to an L4S-FQ - the only description is in this draft - other than the Linux patch. And where this draft describes it, it refers to the appropriate section of the FQ-CoDel RFC.
> 
>>> --- 
>>> 9) Typo: 
>>> I really don’t like this choice of words, it’s far too general a conclusion: 
>>> - Diffserv  works fine when all traffic is marked with a set of DSCPs - I hope you’d agree, but there are cases where it wouldn’t give benefit, e.g. when all traffic on a congested link/router is marked with one code point, whereas there may be benefit if all traffic on a link/router is marked with one code point, when that traffic later shares bandwidth with traffic carrying other marks. 
> 
>      "Also, Diffserv can only provide a latency benefit if a small subset of the traffic on a bottleneck link requests low latency."
> OK?
> 
>>> --- 
>>> - See above. I really don’t like saying it is not applicable. It can be applied, and it can give benefit - I’d accept there was no benefit if the traffic was all marked the same and the resources were constrained in that part of the network. 
> 
>>> - Again not so, the word “works” is problematic. Offers benefit is perhaps more true? 
> 
>>> - I do not agree in this general conclusion and I think this was discussed in previous comments. This is not the place to make sweeping statements about diffserv, or how it might be used. I would strongly urge you to remove this sweeping statement. 
>>> --- 
>>> - Does it actually need to be a contract with a user, I don’t thinks so, there are many cases where diffserv marks are applied in the network, or at an enterprise edge. This is not the document to make such statements about diffserv. I’m going to argue that you need to remove this phrase, people can decide to use L4S without having to address this. 
>>> --- 
> 
> [BB] We've reduced all the discussion of Diffserv in the first para of Sec Consid's by referring back to previous discussion:
> 
> "Because the L4S service reduces delay without increasing the delay of Classic traffic, it should not be necessary to rate-police access to the L4S service. In contrast, Section 5.2 explains how Diffserv only makes a difference if some packets get less favourable treatment than others, which typically requires traffic policing, which can, in turn, lead to further complexity such as traffic contracts at trust boundaries."
> 
> 
>> >>> 
>> 
>> I have now completed a review of L4S ID, I’ve now also reviewed “1.1.  Latency, Loss and Scaling Problems” and I saw a much more polished version of the text with respect to DiffServ. To avoid doubt, here is the text I saw in L4S ID:: 
>> 
>> The above in the L4S-ID seems to me to be an acceptable presentation of the diffserv analysis and I did not find issues with this text, and other mentions in L4S-ID. 
>> 
>> I wonder why the arch and L4S-ID both present this argument, ... and if we really do need two sets explaining diffserv inb both documents? 
>> 
>> If this is really needed to be in both specs, could we just use the same words and avoid different flavours of discussion arising from the L4S ARCH document? 
> 
> [BB] I hope the wording in l4s-arch is now more precise, thanks to you pointing out all the failings. Discussion of Diffserv in each document is for different reasons, one of which (the top level problem statement) overlaps:
> 
> l4s-arch: 
> §4.1 Protocol mechanism allows combination with other identifiers, summarising and referring to ecn-l4s-id for details
> §5.2 What L4S adds to existing approaches (the problem statement, also summarised in a sentence in the Intro)
> §8.1 Security Considerations, comparing L4S with Diffserv to illustrate the likelihood that L4S will need policing - now mainly refers back to §5.2
> 
> ecn-l4s-id:
> §1.1: Problem statement, justifying the need for an L4S identifier (also summarised in a sentence in the Intro)
> The main problem statement is certainly in l4s-arch (which started out incorporating the problem statement written for the L4S BoF). 
> 
> The problem statement in ecn-l4s-id focuses in on the need for an L4S identifier. This needs to start with the bigger problem that L4S is trying to solve. I believe you're saying that you'd prefer that initial step of the rationale in ecn-l4s-id to be replaced by a reference to l4s-arch, but it then loses its logical progression. I think this is a question of editorial style, which I'd rather not change that at this stage.
> 
>> 
>> >>> 
>> 
>>> This might, or might not be true, but there are no RFCs or references sited, and I am unsure why this para is actually needed in an RFC: If it is, then it needs some specific discussio by the WG, but maybe it would be more prudent to not         say this. 
> 
> 	• "Networks tend not to trust end systems to identify which packets..." => Diffserv bleaching discussions.
> 	• Which then raised the question of how operators that use Diffserv actually determine which packets get what.
> 		• You may remember I used the example of the UK ISP, PlusNet, which uses DPI to put traffic into different Diffserv classes. Also, I've pointed to home gateways that do this - I remember I referred to a specific AL-Lu home gateway that my operator-employer used to use, which communicated back to a regularly updated central database of traffic signatures.
> 		• And mobile networks use PI or DPI (see RFC8404)
> 	• The final logical step was discussed on the tsvwg ML during the BoF stages of L4S (you may recall Kevin Smith's involvement), and it's the inevitable conclusion of the previous steps. Also "doesn't always" ensures it is not overstated. 
> 	• Also, this issue was discussed at length in the IETF more widely during the preparation of RFC 8404 (see §2.2.2).
> 
> We've done this:
>>> A network flow ID can be used with FQ, and I think should be noted. I also thought this seemed slightly misleading by not mentioning that TLS over UDP still exposes the port info needed to classify in FQ. I’s suggest removing this ... it seems a well known effect of classification/scheduling on traffic aggregates. 
> 
> [BB] A couple of years ago, we removed all the arguments about FQ that were not relevant to this doc. But we left this one because it is one of the main reasons why it's important to have an architecture that is a superset of FQ and DualQ. 
> 
> Adding a network flow ID defeats the whole point of a confidentiality/privacy solution like IPsec ESP that deliberately encapsulates the transport layer flow ID to obfuscate against inferences made by analysing application flow behaviour.  So I'd rather not suggest that, as it will get thrown out again during SECDIR review.
> 
>>> In reference to scavenger service, can you add a reference to an RFC as well as [LEDBAT_AQM]? 
> 
> [BB] I've realized we should have said "e2e scavenger behaviours [RFC6817]". 
> We didn't mean "services" like the LE PHB here, which can be identified by FQ.
> 
>>> --- 
>>> 14) Query: 
>>> - Is this necessarily true that ECN is used, and the final spec for BBR2 specifies this? Might it be fairer to say: 
>>> - sentence ends in “is”... which seems strange. Is “over-booked” well understood or should this be “over-provisioned” perhaps? 
>>> “Deployment in a mesh topology depends on how much the core is over-provisioned.” 
> 
> [BB] When I talk with network people, they use 'over-booked' quite frequently. I can't think of a better alternative (it's the opposite of over-provisioned, BTW). Searching for synonyms doesn't help, the best is over-subscribed which is less good. Overbooked is also in wide general use, e.g. the flight/hotel was overbooked. We're going to have to leave it as is. Except apparently it doesn't need a hyphen. 
> 
> 
>>> --- 
>>> 17) NiT: In 6.4.2.: 
>>> - what does “work” mean here? is this offer benefit or is this dysfunctional in some way unless this happens or what? 
> 
>>>    identifier, merely treating the traffic in the same way as not-ECT traffic, as 
> But we've changed it to: "as they might already treat ECT(1) traffic today." as well.
> 
>>> --- 
>>> 19) NiT: 
>>> - Is policing in an order - if so explain, otherwise please rephrase to “to police L4S traffic to protect competing Classic ECN traffic” 
> 
>>> -It seems strange to not also mention that QUICs use of TLS also prevent this tampering. 
> 
> Transport layer authentication such as the TCP authentication option (TCP-AO [RFC5925])
> or QUIC's use of TLS [RFC9001] can detect any tampering with congestion feedback.
> 
>>> --- 
> [BB] We had not included any normative language at all in l4s-arch, being informational. We took a similar approach to the Diffserv architecture [RFC2475]. One could possibly argue that this sentence pulls in the relevant part of RFC3168 by reference:
>       ECN field are unchanged from those defined in [
> RFC3168
> ]: Not ECT,
> But looking at that, it needs explaining better anyway, which makes it clearer that the normative inclusion of RFC3168 is being done by ecn-l4s-id, not l4s-arch:
> 
> The references to RFC8311 are commentary about RFC8311, not normative. Whereas in ecn-l4s-id, it's definitely normative - as the authority for the ID to be assigned. Similarly, l4s-arch references to AccECN are commentary about it, not normative.
> 
> _____
> Finally, this was a really useful review, which has hopefully removed some potential trip-wires. Thank you again.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Oct 2021 21:05:12 +0100

Phil,

On 26/08/2021 09:08, philip.eardley=[email-masked] wrote:
> Some comments on the Architecture draft.
>
> It is a nice document, thanks for all the work.
> Some minor suggestions that I think are easy to resolve.
>
> Section 1 Intro
> Para 1- maybe some refs could be added, eg "spikes of hundreds of milliseconds are common"

[BB] That comes from a slow start of a short RTT flow in fast links with 
a state of the art AQM, which is generally tuned to respond over a max 
RTT (e.g. 100ms). Or multiple parallel flow starts. I've added a couple 
of references, which are the evaluations of the two main families of 
solutions prepared by those who developed them:
[COBALT] (which also compares with FQ_CoDel) and [DOCSIS3AQM] which 
evaluates CoDel, SFQ-CoDel and PIE.

    [COBALT]   Palmei, J., Gupta, S., Imputato, P., Morton, J.,
               Tahiliani, M., Avallone, S., and D. Taeht, "Design and
               Evaluation of COBALT Queue Discipline", In Proc. IEEE
               Int'l Symp. Local and Metropolitan Area Networks
               (LANMAN'19) 2019:1-6, July 2019,
               <https://ieeexplore.ieee.org/abstract/document/8847054>.

    [DOCSIS3AQM]
               White, G., "Active Queue Management Algorithms for DOCSIS
               3.0; A Simulation Study of CoDel, SFQ-CoDel and PIE in
               DOCSIS 3.0 Networks", CableLabs Technical Report , April
               2013, <{http://www.cablelabs.com/wp-
               content/uploads/2013/11/
               Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf>.

I think we'll need to temper "common" to "not uncommon" though, 'cos 
it's difficult to be sure how common the test scenarios are in practice.

> and "queuing is typically configured to cause overall network delay to roughly double relative to expected base (unloaded) path delay."

[BB] "Roughly double" refers to the 1 BDP rule of thumb for buffer 
sizing in access links with low stat mux. I've cited the well-known 
SIGCOMM'04 paper on 'Sizing Router Buffers'. Altho this is primarily 
about why the rule does /not/ apply where flows are highly multiplexed 
(core networks), it starts by explaining the 1 BDP rule for access networks.

> Section 2 L4S Architecture Overview
> I think you could expand this section a little to provide a 'protocol model' (RFC4101: "to answer three basic questions:   1. What problem is the protocol trying to achieve?     2. What messages are being transmitted and what do they mean?     3. What are the important, but unobvious, features of the protocol?")
> The intro does a good job of answering question 1.
> For me, S2 should lay out (at a high-level) the solution. The current Section gives the 'components' without spelling out how they fit together, and also includes justification and background (about state of the art), which I think would be better to discuss separately. Also, it's stated that ECT(1) and CE are used, but not how. I think it would be good to include a figure here (Fig 1, or some version)

[BB] OK I've caved to your demands, esp. given I've batted off a number 
of similar comments in the past.

But first, I've added a Document Roadmap section at the end of the 
Introduction to explain that the way the document is written 
deliberately starts with assertion not explanation. Then the rationale 
follows later. Some readers prefer to just be told. Others prefer to 
know why. This serves both. It's done this way, because, if the reader 
doesn't have the whole picture, they can get lost in all the 
justification before the big picture has become clear.

Then I've started the Architecture Overview with a para that gives the 
1-minute explanation based around the 3 parts.
Then I've changed the order of the 3 parts, to put the host first, 'cos 
scalable CC is the thing that creates the low latency. And I've moved 
the explanation of the problem with Classic and why scalable CC solves 
it from the terminology section to here.

I won't paste it all here, I'll post it in a few hours time anyway.

I haven't moved Fig 1 to here. Excuse: This is a symptom of the document 
having originally been written with the DualQ as /the/ architecture. 
Then we made FQ and DualQ equally important, but still had Fig 1 in this 
section to illustrate the DualQ side. Then we were asked to explain the 
DualQ and FQ solutions more fully (in §4.2) rather than relying on the 
dualq draft, and we were asked to move the DualQ figure down into §4.1.

> Section 4.2
> Fig 1
> The '1' between the functions Mark and Scheduler could be explained?

[BB]
s/Then the scheduler can serve the L4S queue with priority/
   /Then the scheduler can serve the L4S queue with priority (denoted by 
the '1' on the priority input)/

That's common scheduler notation apparently.

>
> I found S4.2 a bit hard to parse. I think my issue is that the intro para implies that bullets are about different per flow options. Bullets b and c are about this, but bullet a is (I think) a solution without per flow, and is actually the preferred approach. Bullet a also includes explanation of the dual queue that would apply to any solution.
> Perhaps: re-write the intro para, especially delete the "Nevertheless" sentence, make bullet (a) main text and explain dual q etc. Then say there are different per flow options, and start bullet (a) by stating it is the option without per-flow.

[BB] OK, we had obviously confused you (so presumably others as well): 
I've added this sentence to introduce the table, and I've added a 
heading at the start of each bullet:

"The following bullets describe the known arrangements: a) the DualQ 
Coupled AQM with an L4S AQM in one queue coupled from a Classic AQM in 
the other; b) Per-Flow Queues with an instance of a Classic and an L4S 
AQM in each queue; c) Dual queues with per-flow AQMs, but no per-flow 
queues:"

I've also added a reference to the Linux patch that adds the necessary 
classification to the FQ_CoDel qdisc in order to implement (b).

BTW, there isn't a "preferred approach", from the IETF's point of view.

> Section 8.3
> Is "catastrophically" the best word?

[BB] Yes ;)

Thank you for all these. Some were quite hard, but necessary.

Cheers

Bob

> Best wishes
> Phil
>> Sent: 29 July 2021 17:18
>> - Architecture:
>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf
>> .org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-
>> arch%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed467
>> 1142f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637
>> 631723313409962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=fATwA9c
>> bixSYOkh3vM2IugJ29oPEpvdgEIxbhAx6lMw%3D&amp;reserved=0
>>
>> - DualQ:
>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf
>> .org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-
>> coupled%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed
>> 4671142f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C
>> 637631723313419949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
>> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PR8Rj
>> 3%2BmElcld7NBD7TxOgdkkvEyvbjda2qeUANy%2FAY%3D&amp;reserved=0
>>
>> - ECN ID:
>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf
>> .org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-
>> id%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed46711
>> 42f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C63763
>> 1723313419949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj
>> oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=oAg1Pi24Z
>> Uc4lZK9zwtXA95tm%2F%2B5Y9V2Fz8pEmBMv9g%3D&amp;reserved=0
>>
>> The WGLC will last through 4 weeks from today, and then we'll see what to do
>> next.  Please submit any comments you have on these to the TSVWG list in that
>> timeframe.
>>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 26 Oct 2021 00:12:55 +0100

Pete,

Replying to your points inline [BB] on each draft one at a time, 
starting with dualq-coupled-16...

On 24/08/2021 21:27, Pete Heist wrote:
> # Comments on dualq-coupled-16
>
>> dualq-coupled Abstract:
>>
>> Analytical study and implementation testing of the coupled aqm have
>> shown that scalable and classic flows competing under similar
>> conditions run at roughly the same rate.
> Testing DualPI2 with the defaults showed about a 2x advantage in
> throughput for L4S flows vs non-L4S flows, at the same RTT (see #2
> above). "Roughly the same rate" isn't quantified, but the results seem
> different from that.

[BB] I've changed it to "roughly equivalent", which I think is the 
briefest way of summarizing loads of results for words in an abstract.

2:1 cannot cause a significantly negative impact [RFC5033] and is well 
within the "order of magnitude" required in the most recent RFC on this 
subject [RFC8085] (altho' that was about bulk UDP transfers, I can't see 
how fairness would depend on which transport protocol is in use).

>> dualq-coupled Abstract:
>>
>> When tested in a residential broadband setting, DCTCP also achieves
>> sub-millisecond average queuing delay and zero congestion loss under
>> a wide range of mixes of DCTCP and `Classic' broadband Internet
>> traffic, without compromising the performance of the Classic traffic.
>> The solution has low complexity and requires no configuration for the
>> public Internet.
> Bursty traffic or link layers can raise the limits on low queueing
> delay:
>
> https://github.com/heistp/l4s-tests/#between-flow-induced-delay

[BB] Gorry asked me to cut down the abstract, so I've removed all the 
stuff about testing in a residential broadband setting anyway.
See the mail I just sent him for the new wording. However, I can see now 
that changing it is probably going to lead to further haggling over 
different wording...

Nonetheless, let's move from wording to your technical point itself. 
Before I was hired by CableLabs to work on Low Latency DOCSIS I wrote a 
tech report about why bursty effects like this might happen and how to 
deal with it if they were a problem [SigQ-Dyn]. It's about how sojourn 
time doesn't measure a burst even once the whole burst is queued. You'll 
find this is already referenced from the dualq draft. We implemented the 
alternative metric ('scaled sojourn time'), but didn't use it in the end 
because we couldn't find a good reason for it being necessary. Since 
that time, I've thought of a simpler way of implementing an improved 
variant of the metric, which I wrote into the next rev of the draft 
recently.

But before considering replacing sojourn time we need to bear in mind 
that there's only one run of one experiment that looks odd. So we need 
to set ourselves up with a lot more solid evidence. If there is a 
performance degradation in more typical scenarios, we need to know 
whether it's big enough to worry about. If it is, the modified metric 
should improve the performance again.

[SigQ-Dyn] Briscoe, B., "Rapid Signalling of Queue Dynamics", Technical 
Report TR-BB-2017-001 arXiv:1904.07044 [cs.NI], September 2017, 
<https://arxiv.org/abs/1904.07044>.

>> dualq-coupled Section 1.1:
>>
>> In contrast, dualq coupled aqms addresses the root cause of the
>> latency problem --- they are an enabler for the smooth low latency
>> scalable behaviour of scalable congestion controls, so that every
>> packet in every flow can enjoy very low latency, then there is no
>> need to isolate each flow into a separate queue.
> I would like to be sure that this is true, not just with bursty traffic
> and link layers, but also cross-flow traffic (tests needed on the
> latter).

[BB] Which part do you mean by 'this'? "...every flow can enjoy..."? or 
"No need to isolate..."?

I've edited the former to "...can potentially enjoy...".

>
>
>> dualq-coupled Section 1.4:
>>
>> Thousands of tests have been conducted in a typical fixed residential
>> broadband setting.  experiments used a range of base round trip
>> delays up to 100ms and link rates up to 200 mb/s between the data
>> centre and home network, with varying amounts of background traffic
>> in both queues.
> It sounds like these tests were performed on a certain type of link
> (residential cable?), in a certain RTT range (below 100ms).

[BB] The draft says "below 100ms". And it gives the ref to the paper 
with the spec of the tests - it says they were over DSL, then emulated 
with an ethernet testbed for higher speeds than the DSL could achieve, 
after comparing the validity of the emulation.

> Bursty
> links and higher RTTs should also be tested if L4S is to be used for
> general purpose congestion control (see #3 above). We should end up
> with a system that works well under most all conditions, otherwise it's
> not clear how users and admins will know to change CCAs from one
> activity to the next.

See my posting on the list to Sebastian on 31 Aug 21:
"[tsvwg] tests of Prague utilization with bursty competition in L and C 
[more burst sources]"
https://mailarchive.ietf.org/arch/msg/tsvwg/LfcOHNupIusPHyjVV4vFWFwS8bI/

There's a lot in that conversation, and you'll see some results of it 
appear in the ecn-l4s-id draft shortly.
At the risk of over-simplifying, two key sentences are:
* "Updated link designs are not going to appear any time soon. In the 
mean time, the parameters of any flaky links and the L4S target Q delay 
can be set to levels that trade off between utilization and delay."
* "Whichever technology enables reduced queuing delay variability, the 
world would want to change to reduce burst delays as well. "

>> dualq-coupled Section 4.1.3:
>>
>> Experiments with the DualPI2 AQM (Appendix A) have shown that
>> introducing 'drop on saturation' at 100% L4S marking addresses this
>> problem with unresponsive ECN as well as addressing the saturation
>> problem.  It leaves only a small range of congestion levels where
>> unresponsive traffic gains any advantage from using the ECN
>> capability, and the advantage is hardly detectable [DualQ-Test].
> I might be misunderstanding the text, but two-flow tests of traffic
> that is unresponsive to CE show that, as with RFC3168 ECN, there is an
> advantage to setting ECT(0) or ECT(1), then not responding to CE,
> that's easy to detect:

[BB] That section is about unresponsive traffic gaining advantage *from 
using* the ECN capability.
I think you're talking about gaining advantage from being unresponsive, 
which is not in doubt.

To be clearer, I'll add "... unresponsive traffic gains any advantage 
from using the ECN capability (relative to being unresponsive without ECN)"

> https://sce.dnsmgr.net/results/ect1-2020-05-20-s11-ce-unresponsive/l4s-s11-ce-unresponsive/

[BB] What am I looking at here? Sorry, I'm not familiar with the framing 
of these tests.

___________
[BB] Whatever, thank you v much for your continued testing, reviewing 
and comments.

Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 26 Oct 2021 00:38:07 +0100

Sebastian,

> P.S.: I do not accept that the RTT to the closest CDN is a decent proxy for the internet, as not all content providers/ISPs work with all CDNs so it does really not matter if I sit in Lagos, Nigeri…

[BB] I know ISPs usually work with one or two CDNs, as opposed to all of 
them. I said the figures produced in that study are a workable estimate. 
They are not perfect. But most CDNs that serve a country are going to 
have pretty similar RTTs.

Whatever, this data is miles better than a guess. That's all I'm trying 
to achieve. I can certainly describe the source of the data with more 
circumspection. But, I can't both write and read emails simultaneously, 
and I'm afraid I've now missed the deadline for submitting the drafts 
again - for a few days. I'll updated my editor's copy.

Bob

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 26 Oct 2021 08:42:27 +0100

A little late in replying, but see below.

On 25/10/2021 13:15, Bob Briscoe wrote:
>>> - I generally agree on the value for a wide range of applications 
>>> and sect 6.1 appears to contain a helpful list. ~ However, I 
>>> disagree that the word “all” is necessary or correct in this 
>>> section. I think there are cases that do not fit. Background 
>>> software update, retrieval of logging data, some cases where loss 
>>> protection is more important than for other traffic, etc, So I 
>>> insist there are examples that do not require this. I expect I could 
>>> find more examples, but I’d prefer the word “all” to be removed, to 
>>> leave this as a generalisation.
>
> "At any one time, it is increasingly common for /all/ of the traffic 
> in a bottleneck link (e.g. a household's Internet access) to come from 
Looks like a big improvment, thanks.
> [BB] OK. "So that a user's applications can shift..."
:-)
>>> cross ref)? e.g., are they Protocol Mechanisms (as they are called 
>>> in section 4.1)? or is this more the  to Host Mechanisms (section 
>>> 4.3) ?
>
This shorter text seems also a great improvement, thank you for working 
on this.
> wide-area RTTs. I can see how "it works over the public Internet" 
> would be misconstrued.
I think this reflects what we previously discussed months ago, thanks.
>>> high volume flow that is using a high capacity path? Please check 
>>> and clarify.
>
> Cubic recovery time of 4.3s (166 rounds) in Section 5.1 is *lower* 
> than the the global average for a download from a CDN - at least using 
> the best data I can find.
>
> are on the scale. Rather than putting loads of detail in the 
> terminology definition, we've added the following after the example it 
> refers to in § 5.1:
>
>     per-country data was weighted by Internet user population. So a
>     lone CUBIC flow would at best take about 200 round trips (5 s) to
>     recover from each of its sawtooth reductions, if the flow even
>     lasted that long. This is described as 'at best' because it assume
>     everyone uses an AQM, whereas In reality most users still have a
>     bloated tail-drop buffer. So likely average recovery time would be
>     at least 4x 5 s, if not more, because RTT under load would be at
>     least double, and recovery time depends on the square of RTT."
>
>
> [BDPdata] Briscoe, B., "PI2 Parameters", Technical Report 
> TR-BB-2021-001 arXiv:2107.01003 [cs.NI], July 2021, 
> <https://arxiv.org/abs/2107.01003>.
>
>
:-)
> [BB] Transport protocol references like this are given when they 
> appear elsewhere, but we've added this one here as well, to support 
> the statement about Reno being the default CC in QUIC.
>
:-)
>>> 7) Issue: TCP is a core protocol, and L4S support depends on 
>>> Accurate ECN: Is accurate ECN a normative?  (see comment 22)
> But there is no normative statement that references AccECN in 
> l4s-arch, it's always referred to informatively, like QUIC or DCCP is.
I recall the conversation. I'd also agree the argument still holds that 
this is for all transports that wish to use it and as you note there is 
no normative statement that references.This should not be taken as a 
blocking comment in any way.  If draft-ietf-tcpm-accurate-ecn is already 
ahead in the publication schedule.
>>> draft-ietf-tcpm-accurate-ecn
:-)
>>> - Diffserv  works fine when all traffic is marked with a set of 
>>> DSCPs - I hope you’d agree, but there are cases where it wouldn’t 
>>> give benefit, e.g. when all traffic on a congested link/router is 
>>> marked with one code point, whereas there may be benefit if all 
>>> traffic on a link/router is marked with one code point, when that 
>>> traffic later shares bandwidth with traffic carrying other marks.
>
Much better for me.
>>> be applied, and it can give benefit - I’d accept there was no 
>>> benefit if the traffic was all marked the same and the resources 
>>> were constrained in that part of the network.
>
:-)
> [BB] We've used "makes a difference" this time.
Vague, but the proposed text wouldn't have raised my comment.
> Classic traffic, it should not be necessary to rate-police access to 
> the L4S service. In contrast, Section 5.2 explains how Diffserv only 
> makes a difference if some packets get less favourable treatment than 
> others, which typically requires traffic policing, which can, in turn, 
> lead to further complexity such as traffic contracts at trust boundaries."
>
>
:-)
>> >>>
>>
>> I have now completed a review of L4S ID, I’ve now also reviewed 
>> “1.1.  Latency, Loss and Scaling Problems” and I saw a much more 
>> polished version of the text with respect to DiffServ. To avoid 
>> doubt, here is the text I saw in L4S ID::
>>
>> of the diffserv analysis and I did not find issues with this text, 
>> and other mentions in L4S-ID.
>>
>> same words and avoid different flavours of discussion arising from 
>> the L4S ARCH document?
>
> is for different reasons, one of which (the top level problem 
> statement) overlaps:
>
> l4s-arch:
> §8.1 Security Considerations, comparing L4S with Diffserv to 
> illustrate the likelihood that L4S will need policing - now mainly 
> refers back to §5.2
>
> ecn-l4s-id:
> §1.1: Problem statement, justifying the need for an L4S identifier 
> (also summarised in a sentence in the Intro)
> trying to solve. I believe you're saying that you'd prefer that 
> initial step of the rationale in ecn-l4s-id to be replaced by a 
> reference to l4s-arch, but it then loses its logical progression. I 
> think this is a question of editorial style, which I'd rather not 
> change that at this stage.
>
This sounds better, will read final version and let you know later if I 
have further comments.
>>> This might, or might not be true, but there are no RFCs or 
>>> references sited, and I am unsure why this para is actually needed 
>>> in an RFC: If it is, then it needs some specific discussio by the 
>>> WG, but maybe it would be more prudent to not say this.
>
>         Also, I've pointed to home gateways that do this - I remember
>         I referred to a specific AL-Lu home gateway that my
>   * The final logical step was discussed on the tsvwg ML during the
>     BoF stages of L4S (you may recall Kevin Smith's involvement), and
>     it's the inevitable conclusion of the previous steps. Also
>     "doesn't always" ensures it is not overstated.
:-)
>>> A network flow ID can be used with FQ, and I think should be noted. 
>>> I also thought this seemed slightly misleading by not mentioning 
>>> that TLS over UDP still exposes the port info needed to classify in 
>>> FQ. I’s suggest removing this ... it seems a well known effect of 
> were not relevant to this doc. But we left this one because it is one 
> of the main reasons why it's important to have an architecture that is 
> a superset of FQ and DualQ.
>
> encapsulates the transport layer flow ID to obfuscate against 
> inferences made by analysing application flow behaviour.  So I'd 
> rather not suggest that, as it will get thrown out again during SECDIR 
> review.
>
:-)
> We didn't mean "services" like the LE PHB here, which can be 
Thanks.
> [BB] "can use L4S ECN where available"
:-)
>>> - sentence ends in “is”... which seems strange. Is “over-booked” 
>>> well understood or should this be “over-provisioned” perhaps?
> frequently. I can't think of a better alternative (it's the opposite 
> of over-provisioned, BTW). Searching for synonyms doesn't help, the 
> best is over-subscribed which is less good. Overbooked is also in wide 
> general use, e.g. the flight/hotel was overbooked. We're going to have 
> to leave it as is. Except apparently it doesn't need a hyphen.
>
>
Thanks - yes overbooked is the opposite of over-provisioned, sorry! 
Maybe, if we really wanted to unpick the sentence, we could just say 
this.... if others don't have my comment, nothing might be needed.
> or QUIC's use of TLS [RFC9001] can detect any tampering with 
Thanks
> being informational. 
Aha. I see.
> We took a similar approach to the Diffserv architecture [RFC2475]. One 
> could possibly argue that this sentence pulls in the relevant part of 
> RFC3168 by reference:
Right.
> Whereas in ecn-l4s-id, it's definitely normative - as the authority 
> for the ID to be assigned. Similarly, l4s-arch references to AccECN 
> are commentary about it, not normative.
>
OK. This explanation seems good to me.
> Amazingly, I hadn't already included you in the ACKs - we have now.
Thanks for the careful consideration,

Gorry

>>>     parallel, but that draft is not being last called yet.

Great to see this document working it's way forward through the WGLC 
comments.

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 26 Oct 2021 10:45:19 +0200

Hi Bob,

> On Oct 26, 2021, at 01:38, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> [BB] I know ISPs usually work with one or two CDNs, as opposed to all of them.

	[SM] Okay, so clearly the single one country-average-closest CDN is not a good match here, if e.g. the second closest CDN is ~90ms further out....

> I said the figures produced in that study are a workable estimate. They are not perfect. But most CDNs that serve a country are going to have pretty similar RTTs.

	[SM] Look, the RIPE ATLAS numbers are of wildly different quality per country, mostly as a function of how many measurement nodes exist per country. And due to your scaling by population size, countries like Nigeria become an issue.

> Whatever, this data is miles better than a guess.

	[SM] I disagree, without proper scrutiny and pruning these data as used by you are actively misleading. Claiming a typical base RTT to CDNs of 4ms for Nigeria is not justified by those numbers at all. And I have issues with you still calling this "average RTT to a countries fastest/closest CDN" a "typical base RTT between Internet users and CDNs". This claim a level of generalization to all internet users of a country and also to more than a single CDN...

	GiGo applies here, without sanity checking the inputs, the output might appear "better than a guess" but as I demonstrated for the case of Nigeria this is not actually a honest assessment of the true relevance of such numbers.

> That's all I'm trying to achieve. I can certainly describe the source of the data with more circumspection. But, I can't both write and read emails simultaneously, and I'm afraid I've now missed the deadline for submitting the drafts again - for a few days. I'll updated my editor's copy.

	[SM] I have not opened that issue just yesterday (and I also did post my bjection several times with increasing level of detail), there has been ample time for you to incorporate that information into v18 of the DualQ draft. 

P.S.: IMHO the whole approach is flawed, not just the actual numbers, a population weighted average per-country RTT to the "closest" CDN is not a good proxy for general internet RTT... this is similar with queueing delay, where an average or median is not telling the full story either...

Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 Oct 2021 23:01:35 +0100

Gorry,

Thank you for the rapid confirmations.

Going through all your responses, I think the only one that didn't get a 
100% thumbs up was whether readers will understand what 'overbooked' 
means in the para quoted below.

You said to leave it as-is unless someone else also has a problem 
understanding overbooked. I'll leave this at the top of this email, to 
see if anyone squeals.

I don't want to replace "overbooked" with "under-provisioned", which is 
more about insufficient capacity for the /typical/ busy-hour traffic, 
whereas overbooked is the /worst-case/ contention if all the ingress 
links to an interface were full, which is very different. I think the 
subsequent sentences define what overbooked means, even if the reader 
didn't know at first:

    Deployment in mesh topologies depends on how overbooked the core is.
    If the core is non-blocking, or at least generously provisioned so
    that the edges are nearly always the bottlenecks, it would only be
    necessary to deploy an L4S AQM at the edge bottlenecks.  For example,
    some data-centre networks are designed with the bottleneck in the
    hypervisor or host NICs, while others bottleneck at the top-of-rack
    switch (both the output ports facing hosts and those facing the
    core).

Bob

On 26/10/2021 08:42, Gorry Fairhurst wrote:
>>>> I reviewed Low Latency, Low Loss, Scalable Throughput (L4S) 
>>>> Internet Service Architecture during the WGLC. I support 
>>>> publication, but I think there are issues that would need to be 
>>>> addressed relating to the way L4S is presented.
>>>>
>>>> Best wishes,
>>>>
>>>> Gorry
>>>>
>>>>
>>>> insist there are examples that do not require this. I expect I 
>>>> could find more examples, but I’d prefer the word “all” to be 
>>>> removed, to leave this as a generalisation.
>>
>> in a bottleneck link (e.g. a household's Internet access) to come 
>> from applications that prefer low delay:..."
>>
> This shorter text seems also a great improvement, thank you for 
> working on this.
>>
>>>> ---
>>>> 3) Issue or reword.
>> than the the global average for a download from a CDN - at least 
>> using the best data I can find.
>>
>> I agree that both x-references only describe the scaling, not where 
>> we are on the scale. Rather than putting loads of detail in the 
>> terminology definition, we've added the following after the example 
>> it refers to in § 5.1:
>>
>>     "For a feel of where the global average lone-flow download sits
>>     on this scale at the time of writing (2021), according to
>>     [BDPdata] globally averaged fixed access capacity was 103Mb/s in
>>     2020 and averaged base RTT to a CDN was 25-34ms in 2019.
>>     Averaging of per-country data was weighted by Internet user
>>     population. So a lone CUBIC flow would at best take about 200
>>     round trips (5 s) to recover from each of its sawtooth
>>     reductions, if the flow even lasted that long. This is described
>>     as 'at best' because it assume everyone uses an AQM, whereas In
>>     reality most users still have a bloated tail-drop buffer. So
>>     likely average recovery time would be at least 4x 5 s, if not
>>     more, because RTT under load would be at least double, and
>> The top row has the bottom half of the sawtooth in Reno-friendly 
>> mode, and the top half in Cubic. There is no formula for a sawtooth 
>> in transition, so we've taken the average of the two: 6.8s & 6.4s.
>> Reasoning: AccECN should be a normative ref in a scalable CC spec 
>> that requires AccECN (for instance, Acc-ECN is normative in 
> I recall the conversation. I'd also agree the argument still holds 
> that this is for all transports that wish to use it and as you note 
> there is no normative statement that references.This should not be 
> taken as a blocking comment in any way.  If 
> draft-ietf-tcpm-accurate-ecn is already ahead in the publication schedule.
>>>> ---
>>>> - See above. I really don’t like saying it is not applicable. It 
>>>> can be applied, and it can give benefit - I’d accept there was no 
>>>> sweeping statements about diffserv, or how it might be used. I 
>>>> would strongly urge you to remove this sweeping statement.
>>>> ---
>>>> - Does it actually need to be a contract with a user, I don’t 
>>>> thinks so, there are many cases where diffserv marks are applied in 
>>>> the network, or at an enterprise edge. This is not the document to 
>>>> make such statements about diffserv. I’m going to argue that you 
>>>> need to remove this phrase, people can decide to use L4S without 
>>>> having to address this.
>>>> ---
>>
>> [BB] We've reduced all the discussion of Diffserv in the first para 
>> of Sec Consid's by referring back to previous discussion:
>>
>> "Because the L4S service reduces delay without increasing the delay 
>> of Classic traffic, it should not be necessary to rate-police access 
>> to the L4S service. In contrast, Section 5.2 explains how Diffserv 
>> only makes a difference if some packets get less favourable treatment 
>> than others, which typically requires traffic policing, which can, in 
>> turn, lead to further complexity such as traffic contracts at trust 
>> boundaries."
>>
>>
> :-)
>>> >>>
>>>
>> [BB] I hope the wording in l4s-arch is now more precise, thanks to 
>> you pointing out all the failings. Discussion of Diffserv in each 
>> document is for different reasons, one of which (the top level 
>> problem statement) overlaps:
>>
>> l4s-arch:
>> §5.2 What L4S adds to existing approaches (the problem statement, 
>> also summarised in a sentence in the Intro)
>> The main problem statement is certainly in l4s-arch (which started 
>> out incorporating the problem statement written for the L4S BoF).
>>
> This sounds better, will read final version and let you know later if 
> I have further comments.
>>>
>>> >>>
>>>
>>         which uses DPI to put traffic into different Diffserv
>>         classes. Also, I've pointed to home gateways that do this - I
>>         remember I referred to a specific AL-Lu home gateway that my
>> [BB] A couple of years ago, we removed all the arguments about FQ 
>> that were not relevant to this doc. But we left this one because it 
>> is one of the main reasons why it's important to have an architecture 
>> that is a superset of FQ and DualQ.
>>
>> rather not suggest that, as it will get thrown out again during 
>> SECDIR review.
>>
>> best is over-subscribed which is less good. Overbooked is also in 
>> wide general use, e.g. the flight/hotel was overbooked. We're going 
>> to have to leave it as is. Except apparently it doesn't need a hyphen.
>>
>>
>>>> - Is policing in an order - if so explain, otherwise please 
>>>> rephrase to “to police L4S traffic to protect competing Classic ECN 
>>>> traffic”
>>
>> We took a similar approach to the Diffserv architecture [RFC2475]. 
>> One could possibly argue that this sentence pulls in the relevant 
>> part of RFC3168 by reference:
>> But looking at that, it needs explaining better anyway, which makes 
>> it clearer that the normative inclusion of RFC3168 is being done by 
>> The references to RFC8311 are commentary about RFC8311, not 
>> normative. Whereas in ecn-l4s-id, it's definitely normative - as the 
>> authority for the ID to be assigned. Similarly, l4s-arch references 
>> to AccECN are commentary about it, not normative.
>>
>>>>     This message is starting a combined working group last call on 
>>>> 3 of the
>>>>     L4S drafts:
>>>>
>>>>     -

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 28 Oct 2021 00:23:28 +0100

Sebastian, Gorry,

On 25/10/2021 10:14, Gorry Fairhurst wrote:
>> "The coupling acts like a semi-permeable membrane; isolating the 
>> throughput per flow. The DualQ achieves this indirectly, without 
>> having to inspect transport layer flow identifiers and without 
>> compromising the performance of the Classic traffic."
>>
>> but later it is stated
>>
>> Which really just admits that the indirect "coupling" mechanism from 
>> the introduction does not work as promised, as without such an 
>> emergency scheduling get-out-of-jail card, L4S traffic can very much 
>> compromise "the performance of the Classic traffic" when both compete 
>> in the DualQ.

[BB] To the end of the first sentences from the abstract we've added ", 
relative to a single queue"

Incidentally, the co-authors were looking at the scheduling words on 
Tuesday, separately from Sebastian's email, and we decided the start of 
the main explanation of the conditional priority in Section 2.4 made it 
sound like it was only about overload, when it's also about small 
amounts of C traffic pushing in. So we've also improved the explanation 
in Section 2.4 as follows:

    Priority has to be conditional
    in some way to prevent the C queue being starved by excessive
    unresponsive L traffic (see Section 4.1) and to give C traffic a
    means to push in, as explained next.  With normal responsive L
    traffic, the coupled ECN marking gives C traffic the ability to push
    back against even strict priority, by congestion marking the L
    traffic to make it yield some space.  However, if there is just a
    small finite amount of C traffic (e.g. a DNS request or an initial
    window of data) it cannot induce enough queuing to cause any ECN
    marking in the L queue.  Then, if the L queue happens to remain busy,
    the C traffic would never get a scheduling opportunity from a strict
    priority scheduler - hence the need for L priority to be conditional.
    Giving a small weight or limited waiting time for C traffic improves
    response times for short Classic messages, such as DNS requests and
    improves Classic flow startup because immediate capacity is
    available.

We've also added "(see Section 2.4 and Section 2.5.1)" to the second 
sentence Sebastian quotes, to refer back to the above explanation in 
Section 2.4.

Finally, for completeness, we've also forward referenced from the 
following sentence in 2.2, which now says:

    Priority
    is conditional to prevent starvation of Classic traffic in certain
    conditions (see Section 2.4).

Bob

> clarify text that can be potentially misinterpretted. Could the 
> editors please you align the two statements please in their next 
> revision?
>
> Gorry
>

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 Oct 2021 09:20:16 +0200

Hi Bob, Gorry,

[SM] below.

> On Oct 28, 2021, at 01:23, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian, Gorry,
> 
> [BB] To the end of the first sentences from the abstract we've added ", relative to a single queue"

	[SM] That appears to still gloss over the massive starvation level rate imbalance between L- and C-queue that DualQ induces when the true path RTT is short, see https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/l4s-s1-2/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_fixed.png and https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/l4s-s1-2/batch-l4s-s1-2-cubic-vs-prague-50Mbit-10ms_fixed.png as example.s Since L4S explicitly is tailored for short paths (given TCP pragues/DCTCPS abyssmal RTT-bias see https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667) this is not an obscure corner-case. This really documents that the coupling is just a heuristic and NOT a mechanism to base class separation on...
So in short, adding "relative to a single queue" does not fix the inconsistency in the draft. I do not want to hear about how this imbalance is fine since at short RTTs will experience significantly different queueing delay, as it is the DualQ that inflicts this different queueing delay. The description in the draft should clear enough that readers can make decent predictions of how the syste is going to behave under expected conditions whithout having to put every word on a fine-scale. 

> 
> Incidentally, the co-authors were looking at the scheduling words on Tuesday, separately from Sebastian's email, and we decided the start of the main explanation of the conditional priority in Section 2.4 made it sound like it was only about overload, when it's also about small amounts of C traffic pushing in. So we've also improved the explanation in Section 2.4 as follows:
>    traffic to make it yield some space.  

	[SM] That clearly is a description that ignores what happens at short RTTs. Can we please be frank here and tell readers upfront where the dual queue coupling's pain points are? Truth in advertising and all?

> However, if there is just a

[SM] Please add an explicit action about what happens at short RTTs, as well. Since this deficiency has been documented and discussed for years now, it is clear that it is not a pure implementation issue and hence should be clearly documented.

> 
> 
> We've also added "(see Section 2.4 and Section 2.5.1)" to the second sentence Sebastian quotes, to refer back to the above explanation in Section 2.4.
> 
> Finally, for completeness, we've also forward referenced from the following sentence in 2.2, which now says:
>    Priority

	[SM] To sum up, please include an explicit description of the short RTT case, as that is going to be relevant here as the scheduler clearly triggers in that condition and the condition is well in the expected use-cases of L4S.

Regards
	Sebastian

P.S.: IMHO all of this shows that the coupling is not fi for purpose, it might be fine as an additional heuistic on top of a robust underlaying scheduler, but unlike what the draft makes believe it is not a robust and reliable solution to the issue it pretends to solve. It might well be an acceptable trade-off for many deployers, but the draft should be explicit and open about this and hence let users make informed decisions....

> 
> 
> 
> Bob
> 
>>> 
>>> 
>>> Regards 
>>>     Sebastian 
>>> 
>> 
>> As we move towards trhe resolution of the WGLC, I agree it's good to clarify text that can be potentially misinterpretted. Could the editors please you align the two statements please in their next revision? 
>> 
>> Gorry 
>> 
> 
> --

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 Oct 2021 16:42:37 +0200

Hi list,

one question below, prefixed [SM].

> On Oct 26, 2021, at 01:12, Bob Briscoe <[email-masked]> wrote:
> 
> Pete,
> 
> Replying to your points inline [BB] on each draft one at a time, starting with dualq-coupled-16...
> 
> [BB] I've changed it to "roughly equivalent", which I think is the briefest way of summarizing loads of results for words in an abstract.
> 
> 2:1 cannot cause a significantly negative impact [RFC5033] and is well within the "order of magnitude" required in the most recent RFC on this subject [RFC8085] (altho' that was about bulk UDP transfers, I can't see how fairness would depend on which transport protocol is in use).

	[SM] Is there consensus, that a systematic rate-bias of 2:1 is equivalent? I was under the impression that if rates randomly vary in the 2:1 to 1:2 range that would be equivalent, but a robust and reliable advantage of one type over the other would not qualify as equivalent. What am I missing here?

Regards
	Sebastian

> [BB] Gorry asked me to cut down the abstract, so I've removed all the stuff about testing in a residential broadband setting anyway. 
> See the mail I just sent him for the new wording. However, I can see now that changing it is probably going to lead to further haggling over different wording...
> 
> Nonetheless, let's move from wording to your technical point itself. Before I was hired by CableLabs to work on Low Latency DOCSIS I wrote a tech report about why bursty effects like this might happen and how to deal with it if they were a problem [SigQ-Dyn]. It's about how sojourn time doesn't measure a burst even once the whole burst is queued. You'll find this is already referenced from the dualq draft. We implemented the alternative metric ('scaled sojourn time'), but didn't use it in the end because we couldn't find a good reason for it being necessary. Since that time, I've thought of a simpler way of implementing an improved variant of the metric, which I wrote into the next rev of the draft recently.
> 
> But before considering replacing sojourn time we need to bear in mind that there's only one run of one experiment that looks odd. So we need to set ourselves up with a lot more solid evidence. If there is a performance degradation in more typical scenarios, we need to know whether it's big enough to worry about. If it is, the modified metric should improve the performance again.
> 
> [SigQ-Dyn] Briscoe, B., "Rapid Signalling of Queue Dynamics", Technical Report TR-BB-2017-001 arXiv:1904.07044 [cs.NI], September 2017, <https://arxiv.org/abs/1904.07044>. 
> 
> [BB] Which part do you mean by 'this'? "...every flow can enjoy..."? or "No need to isolate..."? 
> 
> [BB] The draft says "below 100ms". And it gives the ref to the paper with the spec of the tests - it says they were over DSL, then emulated with an ethernet testbed for higher speeds than the DSL could achieve, after comparing the validity of the emulation.
> 
>> Bursty
> "[tsvwg] tests of Prague utilization with bursty competition in L and C [more burst sources]"
> There's a lot in that conversation, and you'll see some results of it appear in the ecn-l4s-id draft shortly. 
> * "Updated link designs are not going to appear any time soon. In the mean time, the parameters of any flaky links and the L4S target Q delay can be set to levels that trade off between utilization and delay."
> * "Whichever technology enables reduced queuing delay variability, the world would want to change to reduce burst delays as well. "
> 
> 
> [BB] That section is about unresponsive traffic gaining advantage from using the ECN capability.
> I think you're talking about gaining advantage from being unresponsive, which is not in doubt.
> 
> To be clearer, I'll add "... unresponsive traffic gains any advantage from using the ECN capability (relative to being unresponsive without ECN)"
> 
> [BB] What am I looking at here? Sorry, I'm not familiar with the framing of these tests.
> 
> ___________
> [BB] Whatever, thank you v much for your continued testing, reviewing and comments.
> 
> Cheers
> 
> 
> Bob
> 
> --

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 28 Oct 2021 16:42:42 +0100

I'll quote some words from RFC805.

This notes that transport need to "employ suitable mechanisms to prevent congestion collapse and
establish a degree of fairness".

It also makes this recommendation:
  
    "Bulk-transfer applications that choose not to implement TFRC or TCP-
    like windowing SHOULD implement a congestion control scheme that
    results in bandwidth (capacity) use that competes fairly with TCP
    within an order of magnitude."

- Which at least to me seems like a starting point for any conversation.

Gorry