Subject: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-ecn-l4s-id-13.txt

From: Bob Briscoe <[email-masked]>
Date: Tue, 2 Mar 2021 13:53:34 +0000

Asad and Ingemar,

@Asad. Thank you v much for reminding me of that section in RFC3168. The 
text in ecn-l4s-id wasn't trying to allow a higher min window than 
existing RFCs. Indeed, it was trying to encourage implementers the other 
way - lower. But without forcing anyone, because implementing a lower 
minimum than 1  requires a step-change in complexity (as your thesis on 
this subject taught me).

I don't think the first sentence with the SHOULD contradicts RFC3168, 
but it seems the second sentence makes it look that way to some people's 
eyes. So let's reword it. Also, I notice that it was only applicable to 
TCP, not all transports. How about...

       A scalable congestion control SHOULD remain responsive to
       congestion when typical RTTs over the public Internet are
       significantly smaller because they are no longer inflated by
       queuing delay.  It would be preferable for the minimum window of a
       scalable congestion control to be lower than the 1 segment minimum
       for TCP mentioned in S.6.1.2 of [RFC3168] (or an equivalent for other
       transports). However, a lower minimum is not set as a formal requirement
       for L4S experiments (see Appendix A.1.6 for rationale)."

@Ingemar, you said your min in SCReAM is 3 datagrams, but you could 
configure it lower. Are SCReAM's datagrams smaller than the MTU? Is the 
text above clear enough for a non-TCP transport? Whatever, would you set 
a different limit?

Bob

On 25/02/2021 20:06, Asad Sajjad Ahmed wrote:
> Bob,
>
> Thanks for updating and for have put in a lot of effort into this draft.
> I have a minor comment regarding the cwnd reduction over ECN when cwnd
> is already very low:
>
> Current draft gives a formal requirement:
>
> §4.3:
>       "A scalable congestion control SHOULD remain responsive to
>        congestion when typical RTTs over the public Internet are
>        significantly smaller because they are no longer inflated by
>        queuing delay.  It would be preferable for the minimum window of a
>        scalable congestion control to be lower than the 2 segment minimum
>        of TCP Reno [RFC5681] but this is not set as a formal requirement
>        for L4S experiments (see Appendix A.1.6 for rationale)."
>
> §A.1.6:
>      "However, the requirement in Section 4.3 is worded as a "SHOULD"
>       because the existence of a minimum window is not all bad.  When
>       competing with an unresponsive flow, a minimum window naturally
>       protects the flow from starvation by at least keeping some data
>       flowing."
>
> But, this contradict with RFC3168 (PS):
>
> §6.1.2:
>      "[...] Therefore, the sending TCP MUST reset the
>       retransmit timer on receiving the ECN-Echo packet when the congestion
>       window is one.  The sending TCP will then be able to send a new
>       packet only when the retransmit timer expires."
>
> The contradiction here is "SHOULD" vs. "MUST". Also TCP CC, as given out by
> RFC5681, does NOT mandate a minimum transmission rate. TCP's loss recovery at
> low cwnd relies on RTO to recover lost segments and TCP using ECN is supposed
> to mimic this behavior [floyd98]. Otherwise, you risk starving the Not-ECT set
> of flow, driving up the queue and increasing the marking probability by going
> unresponsive at cwnd of two segments. This is about harm toward other traffic
> so it make no sense to relax a formal "MUST". Of couse, it is desireable
> that the AQM take action and evict packets from non-responsive flows, but
> this is not true for all AQM, one example being CoDel.
>
> But, the minimum cwnd of 2 does not originate from L4S, but has been
> the current practice for a very long time at least under the Linux TCP stack.
> This, obviously, also needs to be addressed.
>
> Asad
>
> [floyd98] http://www.icir.org/floyd/papers/ecnsims.pdf (§8)
>
> On 21/02/22 14:01:03, [email-masked] wrote:
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>> This draft is a work item of the Transport Area Working Group WG of the IETF.
>>
>>          Title           : Identifying Modified Explicit Congestion Notification (ECN) Semantics for Ultra-Low Queuing Delay (L4S)
>>          Authors         : Koen De Schepper
>>                            Bob Briscoe
>> 	Filename        : draft-ietf-tsvwg-ecn-l4s-id-13.txt
>> 	Pages           : 58
>> 	Date            : 2021-02-22
>>
>> Abstract:
>>     This specification defines the identifier to be used on IP packets
>>     for a new network service called low latency, low loss and scalable
>>     throughput (L4S).  L4S uses an Explicit Congestion Notification (ECN)
>>     scheme that is similar to the original (or 'Classic') ECN approach.
>>     'Classic' ECN marking was required to be equivalent to a drop, both
>>     when applied in the network and when responded to by a transport.
>>     Unlike 'Classic' ECN marking, for packets carrying the L4S
>>     identifier, the network applies marking more immediately and more
>>     aggressively than drop, and the transport response to each mark is
>>     reduced and smoothed relative to that for drop.  The two changes
>>     counterbalance each other so that the throughput of an L4S flow will
>>     be roughly the same as a non-L4S flow under the same conditions.
>>     Nonetheless, the much more frequent control signals and the finer
>>     responses to them result in much more fine-grained adjustments, so
>>     that ultra-low and consistently low queuing delay (typically sub-
>>     millisecond on average) becomes possible for L4S traffic without
>>     compromising link utilization.  Thus even capacity-seeking (TCP-like)
>>     traffic can have high bandwidth and very low delay at the same time,
>>     even during periods of high traffic load.
>>
>>     The L4S identifier defined in this document distinguishes L4S from
>>     'Classic' (e.g. TCP-Reno-friendly) traffic.  It gives an incremental
>>     migration path so that suitably modified network bottlenecks can
>>     distinguish and isolate existing traffic that still follows the
>>     Classic behaviour, to prevent it degrading the low queuing delay and
>>     low loss of L4S traffic.  This specification defines the rules that
>>     L4S transports and network elements need to follow to ensure they
>>     neither harm each other's performance nor that of Classic traffic.
>>     Examples of new active queue management (AQM) marking algorithms and
>>     examples of new transports (whether TCP-like or real-time) are
>>     specified separately.
>>
>>
>> The IETF datatracker status page for this draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>>
>> There are also htmlized versions available at:
>> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-13
>> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-13
>>
>> A diff from the previous version is available at:
>> https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-13
>>
>>
>> Please note that it may take a couple of minutes from the time of submission
>> until the htmlized version and diff are available at tools.ietf.org.
>>
>> Internet-Drafts are also available by anonymous FTP at:
>> ftp://ftp.ietf.org/internet-drafts/
>>
>>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 3 Mar 2021 14:31:40 +0000

Hi Bob, Asad + others

Answer to the @Ingemar question inline below [IJ1] :

/Ingemar

> -----Original Message-----
> From: Bob Briscoe <[email-masked]>
> Sent: den 2 mars 2021 14:54
> To: Asad Sajjad Ahmed <[email-masked]>; [email-masked]; Ingemar
> Johansson S <[email-masked]>
> Subject: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-ecn-l4s-id-13.txt
> 
> Asad and Ingemar,
> 
> text in ecn-l4s-id wasn't trying to allow a higher min window than existing
> RFCs. Indeed, it was trying to encourage implementers the other way -
> lower. But without forcing anyone, because implementing a lower minimum
> than 1  requires a step-change in complexity (as your thesis on this subject
> taught me).
> 
> I don't think the first sentence with the SHOULD contradicts RFC3168, but it
> seems the second sentence makes it look that way to some people's eyes.
> So let's reword it. Also, I notice that it was only applicable to TCP, not all
> transports. How about...
> 
> text above clear enough for a non-TCP transport? Whatever, would you set a
> different limit?

[IJ1] Yes, you can probably set it lower. In practice however I have not seen much need to try that extreme as SCReAM is mostly to be used with services that require edge user access, where RTTs still are in the order of milliseconds at best. This means that the BDP will be a few kilobytes at a minimum. I probably need to look closer into this if necessary.. The RTP packets are delivered by the video encoder pipeline where the MTU is set in the RTP payload packetize. One can perhaps change the MTU on the fly to lower values of the BDP of the links become very low, not sure however if that is possible for instance in gstreamer.

> > The contradiction here is "SHOULD" vs. "MUST". Also TCP CC, as given
> > out by RFC5681, does NOT mandate a minimum transmission rate. TCP's
> > loss recovery at low cwnd relies on RTO to recover lost segments and
> > TCP using ECN is supposed to mimic this behavior [floyd98]. Otherwise,
> > you risk starving the Not-ECT set of flow, driving up the queue and
> > increasing the marking probability by going unresponsive at cwnd of
> > two segments. This is about harm toward other traffic so it make no
> > sense to relax a formal "MUST". Of couse, it is desireable that the
> > AQM take action and evict packets from non-responsive flows, but this is
> not true for all AQM, one example being CoDel.
> >
> > [floyd98]
> > https://protect2.fireeye.com/v1/url?k=b9da725e-e6414b1d-b9da32c5-
> 861fc
> > b972bfc-e0a4204086eb23c8&q=1&e=417868d5-deeb-4c4f-9d1e-
> debe95d3e33c&u=
> > http%3A%2F%2Fwww.icir.org%2Ffloyd%2Fpapers%2Fecnsims.pdf (§8)
> >
> >>          Title           : Identifying Modified Explicit Congestion Notification (ECN)
> Semantics for Ultra-Low Queuing Delay (L4S)
> >> Please note that it may take a couple of minutes from the time of
> >> submission until the htmlized version and diff are available at
> tools.ietf.org.
> >>
> Bob Briscoe                               https://protect2.fireeye.com/v1/url?k=2a6f2e59-
> 75f4171a-2a6f6ec2-861fcb972bfc-e28ad1342845f12e&q=1&e=417868d5-
> deeb-4c4f-9d1e-debe95d3e33c&u=http%3A%2F%2Fbobbriscoe.net%2F

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 3 Mar 2021 17:17:48 +0000

Ingemar,
I obviously didn't phrase my question clearly. Pls see inline [BB]...

> [IJ1] Yes, you can probably set it lower. In practice however I have not seen much need to try that extreme as SCReAM is mostly to be used with services that require edge user access, where RTTs st…

[BB] I was trying to ask what you meant the typical min window would be 
in bytes.
That's the only reason I was trying to ask what MTU you were thinking 
of, not because I was thinking you could reduce the MTU to force more 
packets per window (that's a bad idea IMO).

Also, what about this question: Is the text above clear enough for a 
non-TCP transport?

Bob

PS. See Fig 1 here https://arxiv.org/abs/1904.07598 for back-of-envelope 
calculations to work out if you could have a min-window problem once you 
reduce Qdelay. When we first started testing L4S we found we were 
hitting TCP's min window when we ran more than half a dozen flows in 
parallel.

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 4 Mar 2021 07:15:29 +0000

😊, quite likely that the misunderstanding was on my side
Please see inline [IJ2]
/Ingemar

> Sent: den 3 mars 2021 18:18
> To: Ingemar Johansson S <[email-masked]>; Asad Sajjad
> Ahmed <[email-masked]>; [email-masked]
> >> @Asad. Thank you v much for reminding me of that section in RFC3168.
> >> The text in ecn-l4s-id wasn't trying to allow a higher min window
> >> than existing RFCs. Indeed, it was trying to encourage implementers
> >> the other way - lower. But without forcing anyone, because
> >> implementing a lower minimum than 1  requires a step-change in
> >> complexity (as your thesis on this subject taught me).
> >>
> >> but it seems the second sentence makes it look that way to some
> people's eyes.
> >> So let's reword it. Also, I notice that it was only applicable to
> >> configure it lower. Are SCReAM's datagrams smaller than the MTU? Is
> >> the text above clear enough for a non-TCP transport? Whatever, would
> >> you set a different limit?
> > [IJ1] Yes, you can probably set it lower. In practice however I have not seen
> much need to try that extreme as SCReAM is mostly to be used with services
> that require edge user access, where RTTs still are in the order of
> milliseconds at best. This means that the BDP will be a few kilobytes at a
> minimum. I probably need to look closer into this if necessary.. The RTP
> packets are delivered by the video encoder pipeline where the MTU is set in
> the RTP payload packetize. One can perhaps change the MTU on the fly to
> lower values of the BDP of the links become very low, not sure however if
> that is possible for instance in gstreamer.
> 
> [BB] I was trying to ask what you meant the typical min window would be in
> bytes.
> That's the only reason I was trying to ask what MTU you were thinking of, not
> because I was thinking you could reduce the MTU to force more packets per
> window (that's a bad idea IMO).

[IJ2] OK, understood. SCReAMs window is in bytes, but the RTP packets of course need to be transmitted as whole units. The background is that SCReAM is desiged to be able to congestion control VoIP which typically has much smaller packets than video (e.g. 200bytes istf 1400bytes). A window size in bytes, rather than packets was therefore easier to implement. SCReAM will have problem to go below 1 RTP packet per window. I picked 3 packets as some kind of minimum but it can perhaps be reduced to one packet but below one packet per window that can be problematic. I believe that it is doable but then one need to redesign a bit and I have never really tried out SCReAM over links with very short RTT. 

> 
> Also, what about this question: Is the text above clear enough for a non-TCP
> transport?
[IJ2] Yes, I don't see any difference for e.g. RTP/UDP in this respect. 

> >>>> This draft is a work item of the Transport Area Working Group WG of
> the
> >> IETF.
> >>>>      Examples of new active queue management (AQM) marking
> algorithms
> >> and
> https://protect2.fireeye.com/v1/url?k=2a6f2e59-
> Bob Briscoe                               https://protect2.fireeye.com/v1/url?k=88a239c2-
> d73900a3-88a27959-86d8a30ca42b-6abcf6465bd4a19d&q=1&e=e8dae484-
> 35ef-4d79-b1ee-6fe94beff7ce&u=http%3A%2F%2Fbobbriscoe.net%2F

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 4 Mar 2021 07:33:33 +0000

Ingemar,
OK. I think we've converged.
Thank you

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 4 Mar 2021 18:21:22 +0000

Ingemar,

Sorry, you probably thought this thread had ended this morning, but...

I've had someone suggest (off list), that the conclusion that RTCP 
datagrams can be ECN-capable ought to be written (informatively) into 
ecn-l4s-id. It would go in Appendix A.2.1, which would have to be 
retitled: "Setting ECT in -TCP- Control Packets and Retransmissions".

Before doing that, I checked RFC6679. It says
"RTCP reports MUST NOT be ECT marked, since ECT-marked traffic may be 
dropped if the path is not ECN compliant."

This seems rather paranoid, given the path is first checked for ECN 
support before using it on RTP media. It justifies this by saying the 
path might change to one that black-holes ECN-capable traffic. Whatever, 
if RFC6679 (proposed standard) says this, I don't want to contradict it. 
Do you think it would be OK to say:

"RFC6679 precludes the use of ECT on RTCP datagrams, in case the path 
changes after it has been checked for ECN traversal. L4S experiments 
will need to observe this rule. Nonetheless, as ECN usage becomes more 
widespread, it would be useful to gather data on whether such caution is 
necessary."

Does that seem reasonable?

Bob

On 04/03/2021 07:33, Bob Briscoe wrote:
>> [IJ2] OK, understood. SCReAMs window is in bytes, but the RTP packets 
>> of course need to be transmitted as whole units. The background is 
>> that SCReAM is desiged to be able to congestion control VoIP which 
>> typically has much smaller packets than video (e.g. 200bytes istf 
>> 1400bytes). A window size in bytes, rather than packets was therefore 
>> easier to implement. SCReAM will have problem to go below 1 RTP 
>> packet per window. I picked 3 packets as some kind of minimum but it 
>> can perhaps be reduced to one packet but below one packet per window 
>> that can be problematic. I believe that it is doable but then one 
>> need to redesign a bit and I have never really tried out SCReAM over 
>> links with very short RTT.
>>
>>> non-TCP
>>>>>> the current practice for a very long time at least under the 
>>>>>> Linux TCP
>>> stack.
>>>>>>>            Title           : Identifying Modified Explicit 
>>>>>>> Congestion Notification
>>> (ECN)

---

From: Asad Sajjad Ahmed <[email-masked]>
Date: Fri, 5 Mar 2021 00:39:41 +0100

Bob,

(see [ASA] inline for comments, your original reply got lost for me (not 
in the spam folder, so I had to reconstruct the thread from a later reply.)

> > > than 1  requires a step-change in complexity (as your thesis on this subject

[ASA] Sorry, that was what I meant. There is no need, I hope, to reinforce the
same requirement in the L4S experiment, as long as RFC3168
requirement about being responsive to congestion at low cwnd stands.
It's also good that the problem and the requirement of RFC3168 is mentioned
here as with very low queuing delay there is significantly higher likelihood 
for the problem to occur.

> > >         for L4S experiments (see Appendix A.1.6 for rationale)."

[ASA] Thanks, this new text is good. But, Appendix A.1.6 needs to reflect the
same view (TCP as is does NOT allow a minimum other the maximum RTO you
are allowed to set [RFC6298], i.e., 60 seconds). For Appendix A.1.6, I would 
also rephrase the paragraph I quoted to be more explicit about, when competing 
with unresponsive flow it's a trade off between being fully responsive and 
going back to behaving same as one Not-ECT flow.

This got me thinking, should an unresponsive ECN flow be allowed to keep using 
the ECT(1) code-point? I would say no because when you decide to go 
unresponsive, you made the decision to harm other traffic (!), you are not 
interested in keeping the queue low (!!), you are gambling that the AQM /drops/ 
your packets with the same probability as for Not-ECT flows (and not mark
them) (!!!). This means that you are, essentially, aiming for no benefits
from L4S so why bother signaling to the network that you are responsive.

This would give a more appropriate trade-off, on one side you keep the
benefits of L4S by being responsive while on the other side you withdraw
your benefits, but at the same time be more "protected from starvation". In 
quotes, because you are now as much protected as the Not-ECT flows you once 
tried to starve.

Asad

-- 
Asad

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 5 Mar 2021 09:48:08 +0000

Hi

Please see inline

> Sent: den 4 mars 2021 19:21
> To: Ingemar Johansson S <[email-masked]>;
> [email-masked]
> I've had someone suggest (off list), that the conclusion that RTCP datagrams
> can be ECN-capable ought to be written (informatively) into ecn-l4s-id. It
> would go in Appendix A.2.1, which would have to be
> Before doing that, I checked RFC6679. It says "RTCP reports MUST NOT be
> ECT marked, since ECT-marked traffic may be dropped if the path is not ECN
> compliant."
> 
> This seems rather paranoid, given the path is first checked for ECN support
> before using it on RTP media. It justifies this by saying the path might change
> to one that black-holes ECN-capable traffic. Whatever, if RFC6679 (proposed
> standard) says this, I don't want to contradict it.
> changes after it has been checked for ECN traversal. L4S experiments will
> need to observe this rule. Nonetheless, as ECN usage becomes more
[IJ] Yes. it makes sense. I guess the text was written at a time when it was uncertain how likely ECT traffic is to become black holed. I understand that today these cases are quite rate. 
Still, if it happens, then real time streaming applications should implement circuit brakers  (https://tools.ietf.org/html/rfc8083) . This should make it safe to use ECT for RTCP traffic as well. 

> >> 😊, quite likely that the misunderstanding was on my side Please see
> >> inline [IJ2] /Ingemar
> >>
> >>> To: Ingemar Johansson S <[email-masked]>; Asad
> >>> Sajjad Ahmed <[email-masked]>; [email-masked]
> >>>>> than existing RFCs. Indeed, it was trying to encourage
> >>>>> implementers the other way - lower. But without forcing anyone,
> >>>>> because implementing a lower minimum than 1  requires a
> >>>>> step-change in complexity (as your thesis on this subject taught me).
> >>>>>
> >>>>> I don't think the first sentence with the SHOULD contradicts
> >>>>> RFC3168, but it seems the second sentence makes it look that way
> >>>>> to some
> >>> people's eyes.
> >>>>> @Ingemar, you said your min in SCReAM is 3 datagrams, but you
> >>>>> could configure it lower. Are SCReAM's datagrams smaller than the
> >>>>> MTU? Is the text above clear enough for a non-TCP transport?
> >>>>> Whatever, would you set a different limit?
> >>> much need to try that extreme as SCReAM is mostly to be used with
> >>> services that require edge user access, where RTTs still are in the
> >>> order of milliseconds at best. This means that the BDP will be a few
> >>> kilobytes at a minimum. I probably need to look closer into this if
> >>> necessary.. The RTP packets are delivered by the video encoder
> >>> pipeline where the MTU is set in the RTP payload packetize. One can
> >>> perhaps change the MTU on the fly to lower values of the BDP of the
> >>> links become very low, not sure however if that is possible for
> >>> instance in gstreamer.
> >>>
> >>> [BB] I was trying to ask what you meant the typical min window would
> >>> be in bytes.
> >>> That's the only reason I was trying to ask what MTU you were
> >>> thinking of, not because I was thinking you could reduce the MTU to
> >>> force more packets per window (that's a bad idea IMO).
> >>> PS. See Fig 1 here https://arxiv.org/abs/1904.07598 for
> >>> back-of-envelope calculations to work out if you could have a
> >>> min-window problem once you reduce Qdelay. When we first started
> >>> testing L4S we found we were hitting TCP's min window when we ran
> >>> more than half a dozen flows in parallel.
> >>>
> >>>>> Bob
> >>>>>
> >>>>>> I have a minor comment regarding the cwnd reduction over ECN
> when
> >>>>> cwnd
> >>>>>> is already very low:
> >>>>>>
> >>>>>>          scalable congestion control to be lower than the 2
> >>>>>> segment
> >>> minimum
> >>>>>>         retransmit timer on receiving the ECN-Echo packet when
> >>>>>> the
> >>> congestion
> >>>>>> The contradiction here is "SHOULD" vs. "MUST". Also TCP CC, as
> >>>>>> given out by RFC5681, does NOT mandate a minimum transmission
> >>>>>> rate. TCP's loss recovery at low cwnd relies on RTO to recover
> >>>>>> lost segments and TCP using ECN is supposed to mimic this behavior
> [floyd98].
> >>>>>> Otherwise,
> >>>>>> you risk starving the Not-ECT set of flow, driving up the queue
> >>>>>> and increasing the marking probability by going unresponsive at
> >>>>>> cwnd of two segments. This is about harm toward other traffic so
> >>>>>> it make no sense to relax a formal "MUST". Of couse, it is
> >>>>>> desireable that the AQM take action and evict packets from
> >>>>>> non-responsive flows, but this is
> >>>>>> But, the minimum cwnd of 2 does not originate from L4S, but has
> >>>>>> been the current practice for a very long time at least under the
> >>>>>> Linux TCP
> >>> stack.
> >>>>>>> A New Internet-Draft is available from the on-line
> >>>>>>> Internet-Drafts
> >>>>> directories.
> >>>>>>> Please note that it may take a couple of minutes from the time
> >>>>>>> of submission until the htmlized version and diff are available
> >>>>>>> at
> >>>>> tools.ietf.org.
> >>>>> deeb-4c4f-9d1e-
> debe95d3e33c&u=http%3A%2F%2Fbobbriscoe.net%2F
> >>> --
> >>>
> Bob Briscoe                               https://protect2.fireeye.com/v1/url?k=beec6770-
> e1775e33-beec27eb-861fcb972bfc-cc8f2339192b7cc3&q=1&e=1a89d039-
> 56b3-4ff5-acdf-759bba57d3b4&u=http%3A%2F%2Fbobbriscoe.net%2F

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 5 Mar 2021 10:33:27 +0000

Ingemar, see [BB] inline

> Still, if it happens, then real time streaming applications should implement circuit brakers  (https://tools.ietf.org/html/rfc8083) . This should make it safe to use ECT for RTCP traffic as well.

[BB] In a 2017 measurement study, we found ECN was wiped on 0.23% of 
paths from residential fixed access vantage points. But with mobile 
access, we found 10.5 / 18 of the EU mobile networks we tested wipe the 
ECN field at the first IP hop incoming to their network (the 0.5 of an 
operator was solely wiping on port 80).
     See https://bobbriscoe.net/pubs.html#measure-generalized-ecn
Some follow-up contacts with the operator found this was probably not 
deliberate wiping; most likely bugs in kit wiping the DS field.

So we're probably not out of the woods yet.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 5 Mar 2021 10:42:37 +0000

Asad, see [BB] inline

> quotes, because you are now as much protected as the Not-ECT flows you once

[BB]

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 5 Mar 2021 11:42:58 +0000

Asad, see [BB] inline

On 04/03/2021 23:39, Asad Sajjad Ahmed wrote:
>>>> existing
>>>> RFC3168, but it

[BB] I would rather encourage L4S flows to do what the SHOULD says, than 
"give up". They can do this either by implementing a min window <1 or by 
doing what RFC3168 requires. So I've changed the wording again to make 
it clearer that the RFC3168 requirement is still relevant to L4S as a 
(rather drastic) alternative to window < 1.

    o  A scalable congestion control SHOULD remain responsive to
       congestion when typical RTTs over the public Internet are
       significantly smaller because they are no longer inflated by
       queuing delay.  It would be preferable for the minimum window of a
       scalable congestion control to be lower than 1 segment rather than
       use the timeout approach described for TCP in S.6.1.2 of [RFC3168]
       (or an equivalent for other transports).  However, a lower minimum
       is not set as a formal requirement for L4S experiments (see
       Appendix A.1.6 for rationale).

Also thx for pointing out the Appendix is now inconsistent. I've also 
updated that. Rather than try to explain the diffs here, I'll shall be 
posting the new rev when the IETF servers open on Monday. Then everyone 
can use the usual diff tools.

Cheers

Bob

>>>>>> This draft is a work item of the Transport Area Working Group WG 
>>>>>> of the
>>>> IETF.
>> -- ________________________________________________________________

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Fri, 5 Mar 2021 04:12:44 -0800 (PST)

Bob, see [RWG] inline

>        queuing delay.

I would think it does not matter what causes the decrease.  Simplify
by ending at "smaller"?  Or better yet, can that just be expressed
as "SHOULD remain responsive to congestion independent of RTT"?

>                        It would be preferable for the minimum window of a

lower than 1?  That is 0, and that is window colapse which in most
stacks would be a deadlock, that is why the minimum window is 1.

Also what is "segment" in this context?    Or is what your saying
is that the minimum window can be lower than a TCP segment in the
sense of MSS?

> >>>>> ?4.3:
> >>>>> ?A.1.6:
> >>>>> ?6.1.2:
> >>>>> http%3A%2F%2Fwww.icir.org%2Ffloyd%2Fpapers%2Fecnsims.pdf (?8)
> >>>>>

-- 
Rod Grimes                                                 [email-masked]

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 5 Mar 2021 13:51:51 +0000

Hi

Inline [IJ2]

/Ingemar

> Sent: den 5 mars 2021 11:33
> >> Before doing that, I checked RFC6679. It says "RTCP reports MUST NOT
> >> be ECT marked, since ECT-marked traffic may be dropped if the path is
> >> not ECN compliant."
> >>
> >> path might change to one that black-holes ECN-capable traffic.
> >> Whatever, if RFC6679 (proposed
> >> will need to observe this rule. Nonetheless, as ECN usage becomes
> >> more widespread, it would be useful to gather data on whether such
> >> caution is necessary."
> > [IJ] Yes. it makes sense. I guess the text was written at a time when it 
> > was
> uncertain how likely ECT traffic is to become black holed. I understand that
> today these cases are quite rate.
> > Still, if it happens, then real time streaming applications should 
> > implement
> circuit brakers  (https://tools.ietf.org/html/rfc8083) . This should make it 
> safe
> to use ECT for RTCP traffic as well.
>
> paths from residential fixed access vantage points. But with mobile access,
> we found 10.5 / 18 of the EU mobile networks we tested wipe the ECN field
> at the first IP hop incoming to their network (the 0.5 of an operator was
> solely wiping on port 80).
>      See https://protect2.fireeye.com/v1/url?k=6d2c03bd-32b73afe-6d2c4326-
> 861fcb972bfc-b249fdfe321805e2&q=1&e=908b3683-d82e-493c-a2bc-
> 159f5fe358e2&u=https%3A%2F%2Fbobbriscoe.net%2Fpubs.html%23measur
> e-generalized-ecn
[IJ2] I guess by "wiping" you refer to the case that the ECN bits are set to 
zero ?. Yes, that seems to happen when packets are marked ECT in UL, at least 
in one experiment I ran in a controlled lab setup. I suspect that this is a 
configuration matter but it could be bugs. In any case I believe that this 
will sort itself out for the better when L4S is on the table in 3GPP, 
hopefully mid-year.

> https://protect2.fireeye.com/v1/url?k=170b85e9-
> 4890bcaa-170bc572-861fcb972bfc-6795715d08611dd8&q=1&e=908b3683-
> d82e-493c-a2bc-159f5fe358e2&u=http%3A%2F%2Fbobbriscoe.net%2F

---

From: Asad Sajjad Ahmed <[email-masked]>
Date: Sat, 6 Mar 2021 16:19:37 +0100

Bob, see [ASA2] inline,

On 21/03/05 11:42:58, Bob Briscoe wrote:
> doing what RFC3168 requires. So I've changed the wording again to make it
> clearer that the RFC3168 requirement is still relevant to L4S as a (rather
> drastic) alternative to window < 1.
> 

[ASA2] Thanks, this new text works for me.

Asad 

> posting the new rev when the IETF servers open on Monday. Then everyone can
> use the usual diff tools.
> 
> Cheers
> 
> 
> Bob
> 
> 
> > 
> > Asad
> > 
> > > > > 
> > > > > > > This draft is a work item of the Transport Area
> > > > > > > Working Group WG of the
> > > > > IETF.

-- 
Asad

---

From: Asad Sajjad Ahmed <[email-masked]>
Date: Sat, 6 Mar 2021 16:16:20 +0100

Hi Rod, see [ASA2] inline

> is that the minimum window can be lower than a TCP segment in the

[ASA2] Less than 1 segment per RTT means that you send out your segment with a
local computed delay (possible with pacing), so if you want to keep a cwnd of 
0.5 segment per RTT you would need to send out one segment every other RTT. 
Of course, you could send out smaller segments, in this case half a segment per 
RTT, but you would then have more overhead per packet which would make the
network useless as less data is transferred.

The idea is that if every sender can defer their segment with a small delay 
then the AQM can continue to keep a shallow queue. This would benefit 
everyone at the bottleneck as they would only traverse the shallow
queue. If the sender instead keep at a minimum 1 or 2 segments in flight
per RTT, then the AQM have no way to slow down the sender and as a result the 
queue would get out of control. This is current practice under Linux at
least, where a ECN mark is merely ignored when cwnd is as low as 2.

Asad

-- 
Asad

---

From: Tom Henderson <[email-masked]>
Date: Sun, 7 Mar 2021 12:24:08 -0800

Koen and Bob,

Below are some comments on draft-ietf-tsvwg-ecn-l4s-id-13 for your and 
the working group's consideration.

Title

The title of this draft suggests that the scope is narrowly defining the 
identifier of L4S semantics, but the draft covers much more than this; 
in fact, it perhaps it could more accurately be described as an L4S 
protocol specification.  At the end of the abstract, the draft states 
"This specification defines the rules that L4S transports and network 
elements need to follow...", i.e. a protocol.  It also gets into 
operational considerations and open questions for experimentation.

Perhaps a broader title such as "Explicit Congestion Notification (ECN) 
Protocol for Ultra-Low Queuing Delay (L4S)" would better match the contents.

Abstract

1) If the title is changed to reflect a broader scope, the first 
sentence of the abstract should also be changed accordingly.

2) "L4S uses an Explicit Congestion Notification (ECN)
    scheme that is similar to the original (or 'Classic') ECN approach."

Would it be clearer to state that it uses an ECN scheme similar to the 
scheme described in RFC 8257 DCTCP (which IMO is distinct from the 
original ECN approach)?

1. Introduction

1) See comment 1 and 2 of Abstract; applicable here as well.

2) The goal of less than 1 ms average queuing delay probably should be 
qualified; e.g. something like "on networks not subject to multiple 
access delays above these thresholds"

3) s/transport wire protocol/transport protocol/

4) s/prevent it from/prevent such queues from

1.1 Latency, Loss and Scaling Problem

1) "Latency is becoming the critical performance factor for many (most?)
    applications on the public Internet"

perhaps soften this to "Latency control is becoming increasingly 
important for applications on the public Internet"

2) It seems to me that the paragraph starting with "The DualQ solution 
was developed..." could be deleted.  Perhaps instead in the paragraph 
before, shorten to state "L4S isolation can be achieved with a queue per 
flow (e.g. [RFC8290]) or a DualQ [I-D.ietf-tsvwg-aqm-dualq-coupled]; 
both approaches are addressed in this document."  My rationale for 
suggesting this is that pros and cons of different AQMs seems out of 
scope for this document.

1.2 Terminology

1) The sentence "So it takes longer" is a fragment that could be 
appended to the previous.

2) the second paragraph of Classic Congestion Control definition doesn't 
seem to be about terminology.  Perhaps consider to move commentary that 
isn't about clarifying and distinguishing terms out of this section.

3) For Reno-friendly, rather than defining it as what it is not ('subset 
of traffic that excludes...'), define what it is (e.g. 'consistent with 
fairness goals, as described in RFC 2914, in the presence of other flows 
using congestion control based on RFC 5681').

2. Consensus Choice of L4S Packet Identifier:  Requirements

IMO, this entire section about rationale probably belongs in a separate 
appendix or merged with existing Appendix B.  I would suggest to keep 
the main sections in this draft about specification as much as possible.

3. L4S Packet Identification at Run-Time

1) Consider to change the title to 'L4S Packet Identification' (i.e. 
drop 'at Run-Time').

2) Consider to change (for more clarity here):

    "A network node that implements the L4S service normally classifies
    arriving ECT(1) and CE packets for L4S treatment."

to

    "A network node that implements the L4S service always classifies
    arriving ECT(1) packets as belonging to the L4S service, and, by 
default, classifies arriving CE packets as belonging to the L4S service, 
unless other heuristics as described below in Section 5.3 are employed."

4.  Prerequisite Transport Layer Behavior (the "Prague Requirements")

I suggest to change most, if not all, uses of 'prerequisite' to 
'requirement' or else avoid the term.  For instance, the title could be 
"Transport Layer Requirements".

4.3 Prerequisite Congestion Response

In general, in the first paragraph, I wonder if the congestion response 
could be defined more directly in a prescriptive manner for endpoints 
(and also for compliance detection in the network), such as stating that 
the endpoint must track its base RTT, and it must reduce its sending 
rate when it observes more than two reported CE marks per RTT, possibly 
averaged over some time scale.

5.1. Prerequisite Classification and Re-Marking Behavior

paragraph 3:  The term 'Classic drop' is undefined earlier and probably 
should be defined in the terminology section or clarified here.

5.2. The Meaning of L4S CE Relative to Drop

I wonder whether this section could be clarified by stating first that 
in a dual queue structure with FIFO scheduling, the stated relationship 
must be observed, but in a flow queue scheduled AQM, each flow queue 
operates independently in this regard?  The title could also perhaps 
substitute 'Relationship' for 'Meaning'.

----

I did not have time to review the appendices today.

- Tom

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 8 Mar 2021 01:54:59 +0000

Tom,

On 07/03/2021 20:24, Tom Henderson wrote:
> The title of this draft suggests that the scope is narrowly defining 
> the identifier of L4S semantics, but the draft covers much more than 
> this; in fact, it perhaps it could more accurately be described as an 
> L4S protocol specification.  At the end of the abstract, the draft 
> states "This specification defines the rules that L4S transports and 
> network elements need to follow...", i.e. a protocol.  It also gets 
> into operational considerations and open questions for experimentation.
>
> Perhaps a broader title such as "Explicit Congestion Notification 
> (ECN) Protocol for Ultra-Low Queuing Delay (L4S)" would better match 
> the contents.

[BB] That's a good idea. I like that. Unless anyone shouts, I'll change 
it to that in my local copy.
I intend to post it in the morning, when the IETF servers open.
It contains some queued up changes that resulted from Koen's Prague 
Requirements survey - at least those that seemed non-controversial.
And it will contain the changes from your review below.

> scheme described in RFC 8257 DCTCP (which IMO is distinct from the 

[BB] This is probably nearly as true, but the purpose of this sentence 
was to help readers who might already know the RFC3168 scheme. I think 
that's likely to be a much larger set than those who know DCTCP. (Think 
of all the training courses and Web pages that describe the IP header, 
and the function of the fields).

RFC3168 defines the codepoints and the valid transitions of the ECN 
protocol. The L4S ECN protocol keeps all that, and the semantics of the 
codepoints. The main difference is that ECT(1) is no longer equivalent 
to ECT(0), although they still both have the meaning of ECN-capable 
transport and unmarked.

So I think that warrants the word "similar".

> 1) See comment 1 and 2 of Abstract; applicable here as well.

[BB] Yup to 1 again, but same comment about 2.

> access delays above these thresholds"

[BB] OK. How about "queuing delay... due to e2e congestion control"
Meaning if no possible alteration to the behaviour of the e2e CC can 
reduce a certain type of queuing delay (e.g. queuing for medium access) 
it's not due to e2e CC.

> 3) s/transport wire protocol/transport protocol/

[BB] OK

> 4) s/prevent it from/prevent such queues from

[BB] Don't agree here.
There's been no mention yet of any queues that "such queues" could refer 
back to.

The full clause was:
"...isolate existing Classic traffic from L4S traffic to prevent it from 
degrading"

The 'it' here means the existing Classic traffic. If this was unclear 
I'd happily change it, but I think it's OK as it is, isn't it?

[BB] I'll continue tomorrow... need sleep.
But thx for taking the time to do this.

Cheers

Bob

> before, shorten to state "L4S isolation can be achieved with a queue 
> per flow (e.g. [RFC8290]) or a DualQ 
> [I-D.ietf-tsvwg-aqm-dualq-coupled]; both approaches are addressed in 
> this document."  My rationale for suggesting this is that pros and 
> cons of different AQMs seems out of scope for this document.
>
> 1.2 Terminology
>
> 2) the second paragraph of Classic Congestion Control definition 
> doesn't seem to be about terminology.  Perhaps consider to move 
> commentary that isn't about clarifying and distinguishing terms out of 
> this section.
>
> 3) For Reno-friendly, rather than defining it as what it is not 
> ('subset of traffic that excludes...'), define what it is (e.g. 
> 'consistent with fairness goals, as described in RFC 2914, in the 
> presence of other flows using congestion control based on RFC 5681').
>
> IMO, this entire section about rationale probably belongs in a 
> separate appendix or merged with existing Appendix B.  I would suggest 
> to keep the main sections in this draft about specification as much as 
> possible.
>
> default, classifies arriving CE packets as belonging to the L4S 
> service, unless other heuristics as described below in Section 5.3 are 
> employed."
>
> 'requirement' or else avoid the term.  For instance, the title could 
> be "Transport Layer Requirements".
>
> In general, in the first paragraph, I wonder if the congestion 
> response could be defined more directly in a prescriptive manner for 
> endpoints (and also for compliance detection in the network), such as 
> stating that the endpoint must track its base RTT, and it must reduce 
> its sending rate when it observes more than two reported CE marks per 
> RTT, possibly averaged over some time scale.
>
> paragraph 3:  The term 'Classic drop' is undefined earlier and 
> probably should be defined in the terminology section or clarified here.
>
> in a dual queue structure with FIFO scheduling, the stated 
> relationship must be observed, but in a flow queue scheduled AQM, each 
> flow queue operates independently in this regard?  The title could 
> also perhaps substitute 'Relationship' for 'Meaning'.
>
> ----
>

---

From: Tom Henderson <[email-masked]>
Date: Sun, 7 Mar 2021 19:52:26 -0800

Bob, responding to a couple points below.

On 3/7/21 5:54 PM, Bob Briscoe wrote:
>> Would it be clearer to state that it uses an ECN scheme similar to 
>> the scheme described in RFC 8257 DCTCP (which IMO is distinct from 
>> the original ECN approach)?
>
> that's likely to be a much larger set than those who know DCTCP. 
> (Think of all the training courses and Web pages that describe the IP 
> header, and the function of the fields).
>
> protocol. The L4S ECN protocol keeps all that, and the semantics of 
> the codepoints. The main difference is that ECT(1) is no longer 
> equivalent to ECT(0), although they still both have the meaning of 
> ECN-capable transport and unmarked.
>

In that case, perhaps just call out the differences as follows?

s/is similar to the original (or 'Classic') ECN approach/differs from 
the original (or 'Classic') ECN approach in the following ways/

>>
>> 2) The goal of less than 1 ms average queuing delay probably should 
>> be qualified; e.g. something like "on networks not subject to 
>> multiple access delays above these thresholds"
>
> reduce a certain type of queuing delay (e.g. queuing for medium 
> access) it's not due to e2e CC.

OK.

> There's been no mention yet of any queues that "such queues" could 
> refer back to.
>
> The full clause was:
> "...isolate existing Classic traffic from L4S traffic to prevent it 
> from degrading"
>

I made the comment because upon first read, I didn't know which traffic 
'it' referred to.  So perhaps change 'it' to 'the former'?

- Tom

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 8 Mar 2021 03:43:24 -0800 (PST)

Bob,
	Comments inline [RWG].
Regards,
Rod

> > L4S protocol specification.? At the end of the abstract, the draft 
> > network elements need to follow...", i.e. a protocol.? It also gets 

 [RWG] I like this title change.

> > ?? scheme that is similar to the original (or 'Classic') ECN approach."
> >

 [RWG] I would consider the redefinition of the meaning of a CE mark
to be NOT keeping the semantics of the codepoints as defined by RFC 3168.
I disagree on what the main difference is, it is not just that
ECT(0) and (1) now have different meanings, it is that CE is redefined
when ECT(1) is present.  Further more that difference is lost to
the rest of the network, except the receiving end, once a CE mark has
been applied.

> So I think that warrants the word "similar".

 [RWG] Either word is valid, it is similar in some ways and different
in others.  Perhaps best to cover both?

> I'd happily change it, but I think it's OK as it is, isn't it?

 [RWG] No.  Better to be as clear as possible, these "clear to us" native
english speakers often become unclear when read by non-native speakers.

> > ?? applications on the public Internet"
> >
> > was developed..." could be deleted.? Perhaps instead in the paragraph 
> > this document."? My rationale for suggesting this is that pros and 
> > doesn't seem to be about terminology.? Perhaps consider to move 
> > 2. Consensus Choice of L4S Packet Identifier:? Requirements
> >
> > separate appendix or merged with existing Appendix B.? I would suggest 
> > ?? "A network node that implements the L4S service normally classifies
> > ?? arriving ECT(1) and CE packets for L4S treatment."
> >
> > to
> >
> > ?? "A network node that implements the L4S service always classifies
> > ?? arriving ECT(1) packets as belonging to the L4S service, and, by 
> > 4.? Prerequisite Transport Layer Behavior (the "Prague Requirements")
> >
> > 'requirement' or else avoid the term.? For instance, the title could 
> > paragraph 3:? The term 'Classic drop' is undefined earlier and 
> > flow queue operates independently in this regard?? The title could

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 8 Mar 2021 22:07:52 +0000

Tom,

I've stitched in your responses to the points I got to last night, then 
continued inline, tagged [BB2]...

On 08/03/2021 01:54, Bob Briscoe wrote:
>> Below are some comments on draft-ietf-tsvwg-ecn-l4s-id-13 for your 
>> and the working group's consideration.
>>
>> Title
>>
> [BB] That's a good idea. I like that. Unless anyone shouts, I'll 
> change it to that in my local copy.

[TH] In that case, perhaps just call out the differences as follows?

[BB] That loses the notion that it is largely the same scheme, with 
exceptions.

I've done this, both at the start of the abstract, and the Intro:
Old:

New:

    This specification defines the protocol to be used for a new network
    service called low latency, low loss and scalable throughput (L4S).
    L4S uses an Explicit Congestion Notification (ECN) scheme at the IP
    layer that is similar to the original (or 'Classic') ECN approach,
    except as specified within.

> access) it's not due to e2e CC.

[TH] OK.

>> 3) s/transport wire protocol/transport protocol/

[BB] Actually, I take that back. Re-reading the sentence, it was 
specifically  distinguishing behaviour from the wire protocol, and 
concluding the wire protocol could not be used as the identifier.

> I'd happily change it, but I think it's OK as it is, isn't it?

[TH] I made the comment because upon first read, I didn't know which 
traffic 'it' referred to.  So perhaps change 'it' to 'the former'?

[BB] Done.

- Tom

[BB] Continuing to address the remainder...

>> important for applications on the public Internet"

[BB] The critical performance factor means the limiting factor. Why do 
you think that is not true for the long list of applications that 
follows this statement (justifying the use of 'many')? The the 
subsequent sentence justifies this by putting it in the context of more 
bandwidth offering diminishing returns. Again, is that not true?

>>
>> 2) It seems to me that the paragraph starting with "The DualQ 
>> solution was developed..." could be deleted.  Perhaps instead in the 
>> paragraph before, shorten to state "L4S isolation can be achieved 
>> with a queue per flow (e.g. [RFC8290]) or a DualQ 

[BB] We decided to delete all the anti FQ stuff in the original draft. 
But this is the critical point that justifies the need for an identifier 
at the IP layer. If FQ did not involve the compromises given here, there 
would be no need for an L4S identifier.

>> appended to the previous.

[BB] Grammar: It's a sentence with subject and predicate, not just a 
fragment.

>> commentary that isn't about clarifying and distinguishing terms out 
>> of this section.

[BB] I'm happy to leave this where it is. The first para defines Classic 
CC with arm-wavy adjectives. The second para puts example numbers 
against the adjectives. I agree it seems incongruous, but it is really 
just a richer definition of the term. If this had been a research paper, 
I would have defined the maths of the response functions. But I would 
avoid that in an I-D, where each formula halves the readership (I think 
Stephen Hawking said that).

>> presence of other flows using congestion control based on RFC 5681').

[BB] RFC2914 was written before the concept of 'TCP-friendly' (or 
Reno-Friendly) was coined to be distinct from 'TCP-compatible'.

Whatever, some of the responses to the Prague Requirements Survey seem 
not to believe that a flow should have to fall back to Reno-Friendly anwyay.
So, I'll start a separate thread on this.

>> separate appendix or merged with existing Appendix B.  I would 
>> suggest to keep the main sections in this draft about specification 
>> as much as possible.

[BB] I'd like to hear the views of others on this.
I think it's important to highlight that there was a process to find the 
least worst choice.

>> 1) Consider to change the title to 'L4S Packet Identification' (i.e. 

[BB] Done

>> service, unless other heuristics as described below in Section 5.3 
>> are employed."

[BB] Good. Done.
I wasn't sure about "as belonging to the L4S service." So I kept "for 
L4S treatment", which seems more precise.

>> be "Transport Layer Requirements".

[BB] Thanks. I've removed 'Prerequisite' from all the headings except 
4.2 & 4.3, which are structured as prerequisites for the sender setting 
ECT(1).

>> RTT, possibly averaged over some time scale.

[BB] Nooo! The last thing anyone wants is to be prescriptive about CC 
design. The 2 marks per RTT is an emergent and average outcome of the 
AIMD. It would be very jumpy and unstable if it actually tried to go 
straight to it. No known scalable CC directly aims for 2 marks per RTT. 
Just as there's nothing in Reno that takes the square root of the loss 
probability then aims for it. Also, I don't know why you're saying the 
endpoint must track its base RTT?

The whole idea is to specify only the most important desired outcome, 
then allow freedom in how implementers gets there.
The desired outcome is to keep delay very low, utilization very high and 
controllability very high. And to make sure they stay that way into the 
future. No one wants higher delay, lower utilization or less 
controllability.

>> probably should be defined in the terminology section or clarified here.

[BB] I've reworded so the phrase 'Classic drop' isn't used or needed.

    Under persistent overload an L4S marking treatment MUST begin
    applying drop to L4S traffic until the overload episode has subsided,
    as recommended for all AQM methods in [RFC7567] (Section 4.2.1),
    which follows the similar advice in RFC 3168 (Section 7).  During
    overload, it MUST apply the same drop probability to L4S traffic as
    it would to Classic traffic.

    Where an L4S AQM is transport-aware, this requirement could be
    satisfied by using drop in only the most overloaded individual per-
    flow AQMs.  In a DualQ with flow-aware queue protection (e.g.
    [I-D.briscoe-docsis-q-protection]), this could be achieved by
    redirecting packets in those flows contributing most to the overload
    out of the L4S queue so that they are subjected to drop in the
    Classic queue.

>> I wonder whether this section could be clarified by stating first 
>> that in a dual queue structure with FIFO scheduling, the stated 
>> relationship must be observed, but in a flow queue scheduled AQM, 
>> each flow queue operates independently in this regard? 

[BB] Thanks, yes that's better. However, I haven't said FIFO scheduling 
'cos it's not. Instead I have had to define applicability by what it's not.

    Unless an AQM node schedules application flows explicitly, the
    likelihood that the AQM drops ...

>> The title could also perhaps substitute 'Relationship' for 'Meaning'.

[BB] Now you've made me think about this, I'd prefer "The Strength of 
L4S CE Marking Relative to Drop".

>> I did not have time to review the appendices today.

[BB] They were rather dated, but I have reviewed them myself (twice) in 
the last few months.

Thanks. This review has helped update quite a bit of the early but 
outdated thought framework  that had been embedded in this draft when it 
was first written.

Bob