Subject: Re: [tsvwg] start of WGLC on L4S drafts

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 3 Aug 2021 07:42:13 +0000

Hi Wesley + group

I support that the L4S drafts advance to RFC status. 
I (we = me + others in my company doing L4S implementations and experiments) see L4S as a key technology component to enable wide adoption of XR/gaming and remote control over 5G networks. We have demonstrated the potential of L4S as implemented in real radio base station hardware in the white paper linked to by Rüdiger below. A master thesis report by Davide Brunello + a VTC paper shows promising simulation results. We also plan more experiments with L4S later this year. 
The goal is to include L4S into a work item for latency sensitive services in 3GPP SA2 after the vacation period. 

Regards
Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Smith, Kevin,
> Vodafone
> Sent: den 30 juli 2021 10:59
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: Re: [tsvwg] start of WGLC on L4S drafts
> 
> HI Wes,
> 
> I also support advancing the L4S drafts to RFC, both to facilitate the
> introduction of L4S to 3GPP 5G architecture, and to consolidate the use of L4S
> in fixed access standards (DOCSIS).
> 
> All best,
> Kevin
> 
> Kevin Smith, Vodafone
> 
> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of
> [email-masked]
> Sent: 30 July 2021 07:50
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [tsvwg] start of WGLC on L4S drafts
> 
> CYBER SECURITY WARNING: This email is from an external source - be careful
> of attachments and links. Please follow the Cyber Code and report suspicious
> emails.
> 
> Hi Wes,
> 
> Thanks. I support advancing these drafts to RFC level to foster broader
> community support.
> 
> Deutsche Telekom started experimentation with L4S, see
> 
> https://www.ericsson.com/en/reports-and-papers/white-papers/enabling-
> time-critical-applications-over-5g-with-rate-adaptation
> 
> Regards,
> 
> Ruediger
> 
> -----Ursprüngliche Nachricht-----
> Von: tsvwg <[email-masked]> Im Auftrag von Wesley Eddy
> Gesendet: Donnerstag, 29. Juli 2021 18:18
> An: [email-masked]
> Betreff: [tsvwg] start of WGLC on L4S drafts
> 
> This message is starting a combined working group last call on 3 of the L4S
> drafts:
> 
> - Architecture: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
> 
> - DualQ:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
> 
> - ECN ID: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
> 
> The WGLC will last through 4 weeks from today, and then we'll see what to
> do next.  Please submit any comments you have on these to the TSVWG list
> in that timeframe.
> 
> The chairs are considering a possible virtual interim following the close in
> order to work through feedback received.
> 
> The work on the L4S operational guidance draft is continuing in parallel, but
> that draft is not being last called yet.
>

---

From: Vidhi Goel <[email-masked]>
Date: Tue, 03 Aug 2021 09:42:47 -0700

Hi Wesley,

I support adoption and moving L4S drafts to RFC status.

Thanks,
Vidhi

> On Jul 29, 2021, at 9:18 AM, Wesley Eddy <[email-masked]> wrote:
> 
> This message is starting a combined working group last call on 3 of the L4S drafts:
> 
> - DualQ: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
> 
> The WGLC will last through 4 weeks from today, and then we'll see what to do next.  Please submit any comments you have on these to the TSVWG list in that timeframe.
> 
> The chairs are considering a possible virtual interim following the close in order to work through feedback received.
> 
> The work on the L4S operational guidance draft is continuing in parallel, but that draft is not being last called yet.
> 
>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 3 Aug 2021 18:38:44 +0100

Wes, chairs,

I'm a co-author of all three. But if I didn't think they were ready I 
would say so. Nonetheless, I believe all three are ready to progress.

Having said that, the WGLC process is expected to pick up issues within 
the documents. So of course we'll discuss how to handle any comments 
contributed during the WGLC process.

Bob

On 29/07/2021 17:18, Wesley Eddy wrote:
> - Architecture: 
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>
> - DualQ: 
> The WGLC will last through 4 weeks from today, and then we'll see what 
> to do next.  Please submit any comments you have on these to the TSVWG 
> list in that timeframe.
>
> The chairs are considering a possible virtual interim following the 
> close in order to work through feedback received.
>
> The work on the L4S operational guidance draft is continuing in 
> parallel, but that draft is not being last called yet.
>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 11 Aug 2021 13:57:53 +0000

Hi all,

I re-reviewed the drafts and support publication. A few editorial comments below.

Mirja

draft-ietf-tsvwg-l4s-arch - This purely editorial:
For me the main contribution of this work is nicely shown in Figure 1.  I recommend to add a similar, maybe slightly simplified figure at the beginning of section 2.

draft-ietf-tsvwg-ecn-l4s-id - nits:
- In the Intro there seems to be a missing word at the end of this sentence...?
" L4S uses an Explicit Congestion Notification (ECN) scheme at the IP
   layer that is similar to the original (or 'Classic') ECN approach,
   except as specified within."
- Section 5.1 has this todo still:
" Once disabled, all packets of all ECN codepoints will
   receive Classic treatment and ECT(1) packets MUST be treated as if
   they were {ToDo: Not-ECT / ECT(0) ?}."
- In section 7.1 there is a superfluous bullet point (without text)

Also question on Section 7.2 (open issues): Are these supposed to stay or can they also be converted into a experiment question/goal in section 7.1?

For both draft-ietf-tsvwg-ecn-l4s-id and draft-ietf-tsvwg-aqm-dualq-coupled again one purely editorial comment:
I feel that section 1, in both docs respectively, could be much shorter and point the reader to the arch document instead.

draft-ietf-tsvwg-aqm-dualq-coupled:
Should Figure 1 rather say L and C queue instead of L4S and Classic queue?

On 29.07.21, 18:18, "tsvwg on behalf of Wesley Eddy" <[email-masked] on behalf of [email-masked]> wrote:

    This message is starting a combined working group last call on 3 of the 
    L4S drafts:

    - DualQ:

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 12 Aug 2021 10:44:14 +0100

I reviewed Low Latency, Low Loss, Scalable Throughput (L4S) Internet 
Service Architecture during the WGLC. I support publication, but I think 
there are issues that would need to be addressed relating to the way L4S 
is presented.

Best wishes,

Gorry

This is a review of draft-ietf-tsvwg-l4s-arch-10:

1a) Issue: “It is increasingly common for _all_ of a user's applications 
at any
    one time to require low delay: interactive Web, Web services, voice,
    conversational video, interactive video, interactive remote presence,
    instant messaging, online gaming, remote desktop, cloud-based
    applications and video-assisted remote control of machinery and
    industrial processes. “
- I generally agree on the value for a wide range of applications and 
sect 6.1 appears to contain a helpful list. ~ However, I disagree that 
the word “all” is necessary or correct in this section. I think there 
are cases that do not fit. Background software update, retrieval of 
logging data, some cases where loss protection is more important than 
for other traffic, etc, So I insist there are examples that do not 
require this. I expect I could find more examples, but I’d prefer the 
word “all” to be removed, to leave this as a generalisation.
---
1b) Issue: I’d also quibble with: “so that all of a user's applications 
can shift to it when their stack is updated.“ - again, is the word “all” 
really necessary or correct here?
---
2) NiT
“The L4S
    approach also requires component mechanisms at the endpoints to
    fulfill its goal.”
- True, can you provide examples of what these mechanisms are (or a 
cross ref)? e.g., are they Protocol Mechanisms (as they are called in 
section 4.1)? or is this more the  to Host Mechanisms (section 4.3) ?
---
3) Issue or reword.
“Although DCTCP as-is 'works' well over the public
       Internet, most implementations lack certain safety features that
       will be necessary once it is used outside controlled environments
       like data centres (see Section 6.4.3 and Appendix A).”
- If this is published by the iETF, then I think this needs to have 
consensus that DCTP is acceptable. I don’t now which RFC states that.
Later this ID states: “ DCTCP needs some safety concerns to be fixed for 
general use
    over the public Internet (see Section 4.3 of
    [I-D.ietf-tsvwg-ecn-l4s-id]),”
which seems much closer to what is needed in this section.
---
4) NiT
“With
       Classic congestion controls, such as Reno or Cubic, because flow
       rate has scaled since TCP congestion control was first designed in
       1988, it now takes hundreds of round trips (and growing) to
       recover after a congestion signal (whether a loss or an ECN mark)
       as shown in the examples in Section 5.1 and [RFC3649].
“
- This only seems like a correct statement, if the flow is a bulk / high 
volume flow that is using a high capacity path? Please check and clarify.
---
5) NiT
“such as QUIC.”
- Reference needed for [RFC9000].
---
6) NiT
“[I-D.ietf-tsvwg-ecn-l4s-id] recommends ECT(1)”
draft-ietf-tsvwg-ecn-l4s-id appears to “specify” the use of ECT(1) for 
this purpose. Please check whether ECT(1) is only recommended.
---
7) Issue: TCP is a core protocol, and L4S support depends on Accurate 
ECN: Is accurate ECN a normative?  (see comment 22)
“Work to standardize and implement more
           accurate ECN feedback for TCP (AccECN) is in
           progress [I-D.ietf-tcpm-accurate-ecn],”
---
8) NiT In section 2, ref useful:
“high level explanation of how FQ”
- Please expand and add reference for FQ.
---
9) Typo:
“ The 'Low Loss” part of the name denotes”
- This has mixed quote marks. In other places, double quotes are used.

---

10) Issues relating to critique of diffserv:

10a) Making assertions about Diffserv:
I really don’t like this choice of words, it’s far too general a conclusion:
“      Also, Diffserv only works for a small subset of the traffic on a
       link. “
- Diffserv  works fine when all traffic is marked with a set of DSCPs - 
I hope you’d agree, but there are cases where it wouldn’t give benefit, 
e.g. when all traffic on a congested link/router is marked with one code 
point, whereas there may be benefit if all traffic on a link/router is 
marked with one code point, when that traffic later shares bandwidth 
with traffic carrying other marks.
---
10b) Making assertions about Diffserv:
“As already explained, it is not applicable when all the
       applications in use at one time at a single site (home, small
       business or mobile device) require low latency.”
- See above. I really don’t like saying it is not applicable. It can be 
applied, and it can give benefit - I’d accept there was no benefit if 
the traffic was all marked the same and the resources were constrained 
in that part of the network.
---
10c) In security considerations, making assertions about Diffserv:
“Diffserv only works if some packets
    get less favourable treatment than others.”
- Again not so, the word “works” is problematic. Offers benefit is 
perhaps more true?
---
10d) In security considerations, making assertions about Diffserv:
“So Diffserv has to use
    traffic rate policers to limit how much traffic can be favoured.”
- I do not agree in this general conclusion and I think this was 
discussed in previous comments. This is not the place to make sweeping 
statements about diffserv, or how it might be used. I would strongly 
urge you to remove this sweeping statement.
---
10e) In security considerations, making assertions about Diffserv:
“In
    turn, traffic policers require traffic contracts between users and
    networks as well as pairwise between networks. “
- Does it actually need to be a contract with a user, I don’t thinks so, 
there are many cases where diffserv marks are applied in the network, or 
at an enterprise edge. This is not the document to make such statements 
about diffserv. I’m going to argue that you need to remove this phrase, 
people can decide to use L4S without having to address this.
---
11) Issue: Making assertions about Diffserv:
“   In particular, because networks tend not to trust end systems to
       identify which packets should be favoured over others, where
       networks assign packets to Diffserv classes they often use packet
       inspection of application flow identifiers or deeper inspection of
       application signatures.  Thus, nowadays, Diffserv doesn't always
       sit well with encryption of the layers above IP.  So users have to
       choose between privacy and QoS.
“
This might, or might not be true, but there are no RFCs or references 
sited, and I am unsure why this para is actually needed in an RFC: If it 
is, then it needs some specific discussio by the WG, but maybe it would 
be more prudent to not say this.

---
12) Issue: This seems not quite true:

“     A.  Per-flow forms of L4S like FQ-CoDel are incompatible with full
           end-to-end encryption of transport layer identifiers for
           privacy and confidentiality (e.g. IPSec or encrypted VPN
           tunnels), because they require packet inspection to access the
           end-to-end transport flow identifiers.
“
I am also unsure why this para is actually needed in this ID!

A network flow ID can be used with FQ, and I think should be noted. I 
also thought this seemed slightly misleading by not mentioning that TLS 
over UDP still exposes the port info needed to classify in FQ. I’s 
suggest removing this ... it seems a well known effect of 
classification/scheduling on traffic aggregates.
---
13) Ref
In reference to scavenger service, can you add a reference to an RFC as 
well as [LEDBAT_AQM]?
---
14) Query:
“ Indeed BBRv2 [BBRv2] uses L4S ECN...”
- Is this necessarily true that ECN is used, and the final spec for BBR2 
specifies this? Might it be fairer to say:
“ Indeed BBRv2 [BBRv2] can use L4S ECN...”
---
15) NiT: In 6.4.1:
“   L4S AQMs will not have to be deployed throughout the Internet before
    L4S will work for anyone.”
- Is this “work” for anyone? or “can benefit anyone”?
---
16) NiT:
“Deployment in mesh topologies depends on how over-booked the core is.”
- sentence ends in “is”... which seems strange. Is “over-booked” well 
understood or should this be “over-provisioned” perhaps?
I’d suggest something more like:
“Deployment in a mesh topology depends on how much the core is 
over-provisioned.”
---
17) NiT: In 6.4.2.:
    “For any one L4S flow to work, it requires 3 parts to have been
    deployed.”
- what does “work” mean here? is this offer benefit or is this 
dysfunctional in some way unless this happens or what?
---
18) NiT:
This seems wrong from a diffserv viewpoint:
“they are required not to change the L4S
    identifier, merely treating the traffic as best efforts traffic, as
    they already treat traffic with ECT(1) today.”
- e.g. could we change to something like :
“they are required not to change the L4S
    identifier, merely treating the traffic in the same way as not-ECT 
traffic, as
    they already treat traffic with ECT(1) today.”
---
19) NiT:
“to police L4S traffic in order to
    protect competing Classic ECN traffic.”
- Is policing in an order - if so explain, otherwise please rephrase to 
“to police L4S traffic to protect competing Classic ECN traffic”
---
20) NiT:
“Their packet classifier (item 2 in Figure 1) could identify
    such customers against some other field (e.g. source address range)
    as well as ECN.”
/ECN/as classifying on the ECN Field/.
---
21) NiT:
“The TCP authentication option (TCP-AO [RFC5925]) can be used to
       detect tampering with TCP congestion feedback.”
-It seems strange to not also mention that QUICs use of TLS also prevent 
this tampering.
---
22)Query:  I think some references could need to be normative:
RFC3168
RFC8311
I-D.ietf-tcpm-accurate-ecn

-----

On 29.07.21, 18:18, "tsvwg on behalf of Wesley Eddy"<[email-masked] on behalf of [email-masked]>  wrote:

     - Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/

     - DualQ:

     - ECN ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 12 Aug 2021 11:30:44 +0100

I have reviewed DualQ Coupled AQMs for  L4S

I re-reviewed: draft-ietf-tsvwg-aqm-dualq-coupled-16 in the WGLC. As an 
inidvidual, I support publication as a PS, and have some comments below 
on -16, that

1)
The abstract is unnecessarily long. It does not need to provide this 
level of background information. I’d suggest the editors shorten this.
---
2) The 'Low Loss" part of the name denotes
- This has mixed quote marks. In other places, double quotes are used.
---
3) There appears to be some repetition of topics in section 1.4 between 
this and the L4S arch specifications. This just gives more to read. 
Please do consider removing duplicate examples/discussion.
---
4) Typo - additional space:
“[RFC5681] .”
---
5) Unpublished Individual ID as a Ref explaining a “MUST”:
“The two queues can be part of a larger
    queuing hierarchy [I-D.briscoe-tsvwg-l4s-diffserv].”
- Since this relates a to a requirement, I do not think this is 
appropiate. The reference provided is out of date, and does not appear 
to be progressing. Could this reference be removed?
---
6) Unpublished Individual ID as a Ref:
“and Voice-Admit
    service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]),
- I’d prefer to leave this without a REF that to refer to an individual 
draft submission.
---
7) NiT: “The scheduler SHOULD be work-conserving.”
- This “SHOULD” introduces a design decision: Please add a one sentence 
explanation what happens if it is not.
---
8) Dead reference (used multiple times)
"if the packet is Not-ECT, the appropriate action depends on
          whether some other function is protecting the L queue from
          misbehaving flows (e.g. per-flow queue
          protection [I-D.briscoe-docsis-q-protection] or latency
          policing)"
"Therefore per-flow policing
    (e.g. [I-D.briscoe-docsis-q-protection]) needs to be separable from a
    basic AQM, as an option under policy control."
"However, the addition of DOCSIS per-flow
    Queue Protection [I-D.briscoe-docsis-q-protection] turns this into
    'delay on saturation"
- The tracker shows draft-briscoe-docsis-q-protection has expired and is 
no longer active. What is the current status of this ID?
---
9) RFC2119 usage:
" The following
    parameters MAY be operator-configurable, e.g. to tune for non-
    Internet settings:"
- To me this does not seem like a standards interoperability statement. 
Do we need RFC2119 language, would it be OK to just say:
" The following
    parameters can be operator-configurable, e.g. to tune for non-
    Internet settings:"
---
10)RFC2119 usage:
"The maximum queue
       delay per queue per interval MAY also be recorded."
- I'd sa surely a system can record anything? --  So this does not seem 
to need RFC2119 language, would it be OK to just say "can"... I'd also 
be interested to see a sentence that helps me understand *why* this 
might be useful as a log entry.
---
11) NiT:
"Curvy RED can be used as is when
    the RTT range to be supported is limited, otherwise an adaptation
    mechanism is required."
/required/needed/ - for avoidance of doubt?
---
12) NiT:
"Priority of L4S is required to be conditional to avoid total
    starvation of Classic by heavy L4S traffic."
/required/needed/ - for avoidance of doubt?
--------------
>
> On 29.07.21, 18:18, "tsvwg on behalf of Wesley 
> Eddy"<[email-masked] on behalf of [email-masked]>  wrote:
>
>     - 
> Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>
>     - DualQ:
>     - ECN 
> ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 13 Aug 2021 12:00:21 +0100

Adding to my own review, after re-reading L4S ID  ...

Gorry

On 12/08/2021 10:44, Gorry Fairhurst wrote:
> Service Architecture during the WGLC. I support publication, but I 
> think there are issues that would need to be addressed relating to the 
> way L4S is presented.
>
> Best wishes,
>
> Gorry
>
>
> 1a) Issue: “It is increasingly common for _all_ of a user's 
> applications at any
> 1b) Issue: I’d also quibble with: “so that all of a user's 
> applications can shift to it when their stack is updated.“ - again, is 
> the word “all” really necessary or correct here?
> ---
> 2) NiT
> “The L4S
> Later this ID states: “ DCTCP needs some safety concerns to be fixed 
> for general use
> - This only seems like a correct statement, if the flow is a bulk / 
> high volume flow that is using a high capacity path? Please check and 
> clarify.
> ---
> 5) NiT
> “such as QUIC.”
> - Diffserv  works fine when all traffic is marked with a set of DSCPs 
> - I hope you’d agree, but there are cases where it wouldn’t give 
> benefit, e.g. when all traffic on a congested link/router is marked 
> with one code point, whereas there may be benefit if all traffic on a 
> link/router is marked with one code point, when that traffic later 
> shares bandwidth with traffic carrying other marks.
> ---
> - See above. I really don’t like saying it is not applicable. It can 
> be applied, and it can give benefit - I’d accept there was no benefit 
> if the traffic was all marked the same and the resources were 
> constrained in that part of the network.
> ---
> - Does it actually need to be a contract with a user, I don’t thinks 
> so, there are many cases where diffserv marks are applied in the 
> network, or at an enterprise edge. This is not the document to make 
> such statements about diffserv. I’m going to argue that you need to 
> remove this phrase, people can decide to use L4S without having to 
> address this.
> ---

I have now completed a review of L4S ID, I’ve now also reviewed “1.1.  
Latency, Loss and Scaling Problems” and I saw a much more polished 
version of the text with respect to DiffServ. To avoid doubt, here is 
the text I saw in L4S ID::

    The Diffserv architecture provides Expedited Forwarding [RFC3246], so
    that low latency traffic can jump the queue of other traffic. If
    growth in high-throughput latency-sensitive applications continues,
    periods with solely latency-sensitive traffic will become
    increasingly common on links where traffic aggregation is low. For
    instance, on the access links dedicated to individual sites (homes,
    small enterprises or mobile devices).  These links also tend to
    become the path bottleneck under load.  During these periods, if all
    the traffic were marked for the same treatment at these bottlenecks,
    Diffserv would make no difference.  Instead, it becomes imperative to
    remove the underlying causes of any unnecessary delay.
....

    Active queue management (AQM) was originally developed to solve this
    problem (and others).  Unlike Diffserv, which gives low latency to
    some traffic at the expense of others, AQM controls latency for _all_
    traffic in a class.

The above in the L4S-ID seems to me to be an acceptable presentation of 
the diffserv analysis and I did not find issues with this text, and 
other mentions in L4S-ID.

I wonder why the arch and L4S-ID both present this argument, ... and if 
we really do need two sets explaining diffserv inb both documents?

If this is really needed to be in both specs, could we just use the same 
words and avoid different flavours of discussion arising from the L4S 
ARCH document?

> sited, and I am unsure why this para is actually needed in an RFC: If 
> it is, then it needs some specific discussio by the WG, but maybe it 
> would be more prudent to not say this.
>
> ---
> also thought this seemed slightly misleading by not mentioning that 
> TLS over UDP still exposes the port info needed to classify in FQ. I’s 
> In reference to scavenger service, can you add a reference to an RFC 
> as well as [LEDBAT_AQM]?
> ---
> 14) Query:
> - Is this necessarily true that ECN is used, and the final spec for 
> BBR2 specifies this? Might it be fairer to say:
> - Is policing in an order - if so explain, otherwise please rephrase 
> to “to police L4S traffic to protect competing Classic ECN traffic”
> ---
> 20) NiT:
> -It seems strange to not also mention that QUICs use of TLS also 
> prevent this tampering.
> ---

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 13 Aug 2021 12:11:10 +0100

I have reviewed draft-ietf-tsvwg-ecn-l4s-id-19 in WGLC.

I think this draft provides methods that address an important need - to 
provide techniques to control end to end latency. I provided a detailed 
review of a recent previous version of this ID, and the issues I found 
then have been addressed. Thanks editors. I have checked this version 
and found the three additional issues below.

I therefore would support *as an individual* progression of this draft 
for publication as EXP.

Best wishes,

Gorry
(as individual in tsvwg)

P.S. I also have a small number of editorial NiTs that I will send 
separately to the list.
---

ISSUE 1: To-Do
   “Once disabled, all packets of all ECN codepoints will
    receive Classic treatment and ECT(1) packets MUST be treated as if
    they were {ToDo: Not-ECT / ECT(0) ?}.”
- This RFC-2119 statement is incomplete.
- This issue needs to be discussed.

---
ISSUE 2: Appendix A.
This appendix is informative, and as such I think we should be careful 
to not imply this as expressing requirements.

However, the current text contains several lines mentioning 
“requirements”, including the title!

  - is it possible to express this differently and avoid any ambiguity 
as the document receives more review?
I could suggest a few small changes which would seem to me resolve this 
issue:
The title likely needs to be renamed, because it creates most of the 
problem: “The 'Prague L4S Recommendations’ or “The 'Prague L4S Design 
Recommendations’, or similar.
Also these small number of specific lines seem to need careful 
rewording, e.g.:
/safety improvements (requirements)/safety improvements ()/
/implementation of the requirements./implementation of these 
recommendations./
/Requirements for Scalable Transport Protocols/Features for Scalable 
Transport Protocols/
/The requirement is written/This is written/
/Similarly to the requirement in Appendix A.1.4 this 
requirement/Similarly to Appendix A.1.4, this/
/Having said all the above, the approach recommended in the requirements 
is to monitor/Having said all the above, the approach recommended is to 
monitor/
---

ISSUE 3: References
-Please confirm the intended status of:
[I-D.briscoe-docsis-q-protection]

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 13 Aug 2021 12:11:13 +0100

I have reviewed draft-ietf-tsvwg-ecn-l4s-id-19 in WGLC and have some 
comments that I expect are editorial.

(I seperately sent a couple of WGLC issues concerning this draft.)

Best wishes,

Gorry

(as individual)
---
NiT: Terminology could be cross-referenced.
- I’ve just tried to compare the difference between the terminology in 
L4S arch and L4S ID, and I found them to be identical, with the 
exception that L4S arch also defines “site”.

So:
(a) Could we add “site” also as a definition to L4S-ID, the term is also 
used in this ID.
(b) Could you consider adding a one line statement to L4S arch that says 
the two sets of definitions are identical, so that a reader doesn’t need 
to check them when they read each document. I expect that L4S-ID as an 
EXP Spec provides the normative text.
---
NIT: In Sect 1.1:
- Can you expand FQ or provide a reference?
----
NIT: I think this:
“in the IP headers even in
       one direction of a TCP connection will imply that both ends must
       be supporting accurate ECN feedback.”
... Could perhaps be clearer as:
      “in the IP headers even in
       one direction of a TCP connection will imply that both ends
       support accurate ECN feedback.”
---
NIT: I think this:
“in which AQMs filter out
    variations before signalling congestion.”
... Could perhaps be clearer as (removing “out”):
“in which an AQM filters
    variations (i.e. smooths over time) before signalling congestion.”
---
NiT:
In 5.4.1.1. : “In a typical case for”... followed by “In this case it”.
- To me these seem like the same case, should therefore these both be in 
one single paragraph, rather than two?
---
NiT:
“ To include additional traffic with L4S, a network element only reads
    identifiers such as those itemized above.  It MUST NOT alter these
    non-ECN identifiers, so that they survive for any potential use later
    on the network path.”

- Actually I think this MUST statemen t only applies to the L4S 
processing within the device, and I think it doesn’t mean L4S is unable 
to work across a policy enforcement point or some other device that 
otherwise modifies the other packet headers.

.... so is this better as:

“ To include additional traffic with L4S, a network element only reads
    identifiers such as those itemized above.  The L4S forwarding
   MUST NOT alter these
    non-ECN identifiers, so that they survive for any potential use later
    on the network path.”
----
NiT: To avoid future discussion of whether this is ever used in other 
cases, it might be better not to assert the deployment extent
“But they cannot
    easily satisfy this loss recovery requirement.  However, it is
    believed they do not need to, because such implementations are
    believed to solely exist in controlled environments, where the
    network technology keeps reordering extremely low anyway.”

.... so is this better as:

“But they cannot
    easily satisfy this loss recovery requirement.  When such 
implementations are
    used in controlled environments, the network technology can keep 
reordering extremely low.”

---

NiT: Would it be OK to tweak this a little?
“Setting ECT in Control Packets and Retransmissions

    Description: This item concerns TCP and its derivatives (e.g. SCTP)
    as well as RTP/RTCP [RFC6679].  The original specification of ECN for
    TCP precluded the use of ECN on control packets and retransmissions.
    To improve performance, scalable transport protocols ought to enable
    ECN at the IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs,
    etc.) and in retransmitted packets.  The same is true for derivatives
    of TCP, e.g. SCTP.”
... I think this might be better related to transports in general, and 
rephrase the SCTP text - there is no IETF spec for ECN in SCTP (yet). I 
suggest something like:
“Setting ECT in Control Packets and Retransmissions

    Description: This item concerns TCP and other transport protocols 
(e.g. SCTP)
    as well as RTP/RTCP [RFC6679].  The original specification of ECN for
    TCP precluded the use of ECN on control packets and retransmissions.
    To improve performance, scalable transport protocols ought to enable
    ECN at the IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs,
    etc.) and in retransmitted packets.  The same is true for other

    transports, e.g. QUIC, SCTP.

----

NiT: - Add Reference?
“TCP Cubic was
    designed to solve this problem”
- please site the TCPM work item: draft-ietf-tcpm-rfc8312bis-03

=====

---

From: Ermin Sakic <[email-masked]>
Date: Mon, 16 Aug 2021 07:59:50 +0000

Hi Wesley, all,

L4S proposals are an important step towards enabling low-latency transmissions for remote control, AR/VR, and online gaming applications.
I support progressing the drafts to RFC status.

I have attached a few comments below after re-reading the latest versions of the three drafts.
My apologies if some of the comments came up in discussion earlier.

Best wishes,
Ermin Sakic, NVIDIA

- draft-ietf-tsvwg-l4s-arch-10:
        - Figure 1: Is there a specific reason why the “mark/drop” action is not denoted for L4S queue compared to “mark”-only? Section 5.1 of draft-ietf-tsvwg-ecn-l4s-id-19 states that “…L4S marking treatment MUST begin applying drop to L4S traffic until the overload episode has subsided…”.
        - Section 8.2: “…if it is leads…” -> “…if it leads…”
- draft-ietf-tsvwg-ecn-l4s-id-19:
        - Section 1 Introduction: “…responses to them result..” -> “…responses to markings result…”
        - Section 1.1: “The explanation is summarised without the maths in the L4S architecture document” -> I suggest referencing to the specific section number of draft-ietf-tsvwg-l4s-arch here.
        - Section 7.1: “TCP. QUIC…” -> “TCP, QUIC…”
        - Appendix B: “tranport-layer” -> “transport-layer”
- draft-ietf-tsvwg-aqm-dualq-coupled-16:
        - Figure 1: As mentioned above, I’d expect to see an optional drop as the outcome of forwarding to the L traffic queue in addition to the “ECN marking” action.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
Sent: Thursday, 29 July 2021 18:18
To: [email-masked]
Subject: [tsvwg] start of WGLC on L4S drafts

External email: Use caution opening links or attachments

- Architecture: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-arch%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a7V1lmfJi%2Fbs2H2mZIr3QhveYsPj3QzPguBFDWsy%2BC0%3D&amp;reserved=0

- DualQ:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-coupled%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=PwO0BFTNKv2c4ZbGoQZEvxfLg7NIG9mw8oWtNqdwsjs%3D&amp;reserved=0

- ECN ID: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-id%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=QpBxzuleCeXpxzoBR4oEJWANLk7v1Vvo%2Ft4H8d3kZz0%3D&amp;reserved=0

-----------------------------------------------------------------------------------
NVIDIA GmbH
Wuerselen
Amtsgericht Aachen
HRB 8361
Managing Director: Karen Theresa Burns

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Aug 2021 16:04:24 +0100

Ermin, Gorry, Mirja,

Just to let you know, I'm collecting all your review comments, but I 
won't respond until the call is over.

Cheers

Bob

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 17 Aug 2021 11:22:09 -0400

Hi Wesley and TSVWG folks,

I support advancing all three of these L4S drafts to RFC status.

(I have posted some minor editorial comments on the
draft-ietf-tsvwg-l4s-arch draft in a separate thread.)

Thanks,
neal

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Aug 2021 17:00:23 +0100

Mirja,

Having said I would hold back replies, yours are fairly brief, so I'll 
reply now...

> For me the main contribution of this work is nicely shown in Figure 1.  I recommend to add a similar, maybe slightly simplified figure at the beginning of section 2.

[BB] Until draft-07, this figure was in Section 2. But we moved it to 
the more detailed section on the DualQ when we also 'rebalanced' the 
draft to make it clear that FQ-L4S is as much a part of the L4S 
architecture as DualQ.

Was your comment given with that in mind?

>     except as specified within."

[BB] That's OK in English. It just means similar to "as specified 
below", but "below" tends to mean "immediately below".

>     they were {ToDo: Not-ECT / ECT(0) ?}."

[BB] Yeah, I noticed that after I had posted the drafts. This is 
actually a technical point, so I'll start a separate thread on it, in 
case some people don't read emails starting 'editorial'.

> - In section 7.1 there is a superfluous bullet point (without text)

[BB] That's a slip of the thumbs - meant to be sub-bullets of the 
previous bullet. Sorted.

> Also question on Section 7.2 (open issues): Are these supposed to stay or can they also be converted into a experiment question/goal in section 7.1?

[BB] I think they should stay. The first (coexistence) was meant to have 
a pointer to l4sops (now added).
For the second I've added:

    "...Originally, this was due to a bug in the initialization of the
    congestion EWMA in the Linux implementation of TCP Prague. That was
    quickly fixed, which removed the main performance impact, but
    further improvement would be useful (either by modifying CoDel,
    Scalable congestion controls, or both)."

> I feel that section 1, in both docs respectively, could be much shorter and point the reader to the arch document instead.

[BB] Understood. Personally I've wavered between doing as you suggest 
and keeping them so that each document stands on its own. If you're not 
rabidly against, on balance I'd leave them as they are. But if there are 
multiple strong opinions, I'd be happy to relent.

> Should Figure 1 rather say L and C queue instead of L4S and Classic queue?

[BB] I think it's useful to give the names in full in this figure, isn't 
it? However, given it talks about the L queue and C queue below that, 
I'll say
     L4S (L) queue
     Classic (C) queue
OK?

Bob

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 17 Aug 2021 09:31:35 -0700

I support the L4S drafts as well, with the concern that AccECN is the best
simple protocol for L4S. I think this part will evolve and does not need to
be a blocker for the rest of the good work done.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 17 Aug 2021 18:18:32 +0100

Mirja,

> On 11/08/2021 14:57, Mirja Kuehlewind wrote:
[snip]
>> - Section 5.1 has this todo still:
See last para of: 
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19#section-5.1 
starting:
>> "  In case unforeseen problems arise with the L4S experiment, it MUST be
>>     possible to configure an L4S implementation to disable the L4S
>>     treatment.  Once disabled, all packets of all ECN codepoints will

I propose we choose Not-ECT for the following reason:

·Redirecting ECT(1) to ECT(0) treatment would be foolish because the 
sender would still behave as L4S

·Redirecting ECT(1) to Not-ECT treatment would be effective, by inducing 
a drop response at the sender

I could also paste in a long treatise on what all the requirements might 
be, what circumstances might lead to disabling L4S, etc, etc. But 
ultimately, the above consideration trumped every other aspect of the 
discussion.

Another view has been expressed (off list) that we should not decide at 
this point, but instead wait until the problem needs a solution. I would 
argue that this 'MUST' needs to be implemented from the start, and 
asking implementers to include options for future flexibility needs to 
be tempered by whether ECT(0) treatment would ever be useful in any 
future scenario.

Bob

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 18 Aug 2021 09:01:44 +0100

On 17/08/2021 18:18, Bob Briscoe wrote:
>> Having said I would hold back replies, yours are fairly brief, so 
>> I'll reply now...
>>
> ·Redirecting ECT(1) to Not-ECT treatment would be effective, by 
> inducing a drop response at the sender
>
> I could also paste in a long treatise on what all the requirements 
> might be, what circumstances might lead to disabling L4S, etc, etc. 
> But ultimately, the above consideration trumped every other aspect of 
> the discussion.
>
> Another view has been expressed (off list) that we should not decide 
> at this point, but instead wait until the problem needs a solution. I 
> would argue that this 'MUST' needs to be implemented from the start, 
> and asking implementers to include options for future flexibility 
> needs to be tempered by whether ECT(0) treatment would ever be useful 
> in any future scenario.
>
>
> Bob
>
>
I also had the same review comment.

If one were to simply remove L4S support, rather than disable it, then a 
router might treat ECT(1) and ECT(0) the same. However, I agree if we do 
need to specify a behaviour for diabling L4S, this argument seems like a 
valid reason to treat ECT(1) marked traffic the same as Not-ECT.That 
seems to provide the most obvious way forward to deploying L4S.

Gorry (as an individual)

> Bob Briscoehttp://bobbriscoe.net/

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 18 Aug 2021 08:41:02 +0000

Hi Bob,

see three replies below.

From: tsvwg <[email-masked]> on behalf of Bob Briscoe <[email-masked]>
Date: Tuesday, 17. August 2021 at 18:01
To: Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: [tsvwg] L4S Editorial Reviews (was: start of WGLC on L4S drafts)

Mirja,

Having said I would hold back replies, yours are fairly brief, so I'll reply now...
On 11/08/2021 14:57, Mirja Kuehlewind wrote:

Hi all,

Mirja

[BB] Until draft-07, this figure was in Section 2. But we moved it to the more detailed section on the DualQ when we also 'rebalanced' the draft to make it clear that FQ-L4S is as much a part of the L4S architecture as DualQ.

[MK] This rebalance is fine but now it feels like there is a figure missing in section 2. Therefore I proposed to _also_ add the figure to section 2 but maybe a slightly simplified version without the details of dualQ, e.g. just have some box that says “L4S-aware AQM” or something. The important part would be to show the different components (Scalable CC in the endpoint, Identification/Classification, and AQM) and explain how the “work” together.

[BB] That's OK in English. It just means similar to "as specified below", but "below" tends to mean "immediately below".

[MK] Actually looking at this sentence again, the whole sentence is not super clear. I think rather than saying “it’s similar to classic ECN except it’s not”, it would be better to say what is different. In other words I would actually remove this sentence and rather say (at the end of this paragraph) something like: “This draft specifies the use of the ECN ECT(1) code point as an identifier for L4S traffic.”

[BB] Yeah, I noticed that after I had posted the drafts. This is actually a technical point, so I'll start a separate thread on it, in case some people don't read emails starting 'editorial'.

[BB] That's a slip of the thumbs - meant to be sub-bullets of the previous bullet. Sorted.

[BB] I think they should stay. The first (coexistence) was meant to have a pointer to l4sops (now added).
For the second I've added:
"...Originally, this was due to a bug in the initialization of the congestion EWMA in the Linux implementation of TCP Prague. That was quickly fixed, which removed the main performance impact, but further improvement would be useful (either by modifying CoDel, Scalable congestion controls, or both)."

[BB] Understood. Personally I've wavered between doing as you suggest and keeping them so that each document stands on its own. If you're not rabidly against, on balance I'd leave them as they are. But if there are multiple strong opinions, I'd be happy to relent.

[BB] I think it's useful to give the names in full in this figure, isn't it? However, given it talks about the L queue and C queue below that, I'll say
    L4S (L) queue
    Classic (C) queue
OK?

[MK] Section 2.3 says:

„Thus, the L queue is not
   solely an L4S queue, it can be consider more generally as a low
   latency queue.“

So my understanding was that L is actually an abbreviation for “low latency” and “L4S”. However looking at the terminology it says:

“L: Abbreviation for L4S, e.g. when used as a subscript.“

I think this might need some clear up to get this straight.

Mirja

On 29.07.21, 18:18, "tsvwg on behalf of Wesley Eddy" <[email-masked] on behalf of [email-masked]><mailto:[email-masked]@mti-systems.com> wrote:

    L4S drafts:

    - DualQ:

    list in that timeframe.

--

Bob Briscoe                               http://bobbriscoe.net/<https://protect2.fireeye.com/v1/url?k=83fd01ee-dc66393c-83fd4175-866132fe445e-2ea15d5d62688d61&q=1&e=8f894785-ae8d-45d8-a094-22557c4afbba&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: Kuhn Nicolas <[email-masked]>
Date: Wed, 18 Aug 2021 10:02:52 +0000

Hello, 

I had reviewed previous versions of these drafts and support their publication.
Thanks for the good work ! 

Nicolas

-----Message d'origine-----
De : tsvwg <[email-masked]> De la part de Wesley Eddy
Envoyé : jeudi 29 juillet 2021 18:18
À : [email-masked]
Objet : [tsvwg] start of WGLC on L4S drafts

- DualQ:

---

From: "Livingood, Jason" <[email-masked]>
Date: Wed, 18 Aug 2021 13:02:52 +0000

I also support advancing the L4S drafts to RFC status.

Jason

From: tsvwg <[email-masked]> on behalf of Neal Cardwell <ncardwell=[email-masked]>
Date: Tuesday, August 17, 2021 at 11:23
To: Wesley Eddy <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

(I have posted some minor editorial comments on the draft-ietf-tsvwg-l4s-arch draft in a separate thread.)

Thanks,
neal

On Thu, Jul 29, 2021 at 12:18 PM Wesley Eddy <[email-masked]<mailto:[email-masked]>> wrote:
This message is starting a combined working group last call on 3 of the
L4S drafts:

- Architecture: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/__;!!CQl3mcHX2A!VB0yMsv9q3RiRpViaz0JV9YKMLmrg0KM_QaFD-zMSeMvflPpO_1oBev-1nIuwil8BKnpmw$>

- DualQ:
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/__;!!CQl3mcHX2A!VB0yMsv9q3RiRpViaz0JV9YKMLmrg0KM_QaFD-zMSeMvflPpO_1oBev-1nIuwinTWGa-xQ$>

- ECN ID: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!CQl3mcHX2A!VB0yMsv9q3RiRpViaz0JV9YKMLmrg0KM_QaFD-zMSeMvflPpO_1oBev-1nIuwimVr4x4uw$>

---

From: "Glenn Deen" <[email-masked]>
Date: Thu, 19 Aug 2021 10:29:55 -0700

I support moving these drafts forward.

-Glenn Deen

On 29 Jul 2021, at 9:18, Wesley Eddy wrote:

> - ECN ID: 
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>
> to do next.  Please submit any comments you have on these to the 
> TSVWG list in that timeframe.
>

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 21 Aug 2021 20:11:47 +0000

Overall:
The documents are in much better shape than the last time I reviewed
them, thanks for all the improvements.

I'm reviewing l4s-id and l4s-arch but didn't have time to get to
dualq-aqm yet.  But I wanted to make sure these got posted, and
might not have more time before the deadline.

----

draft-ietf-tsvwg-ecn-l4s-id-19:
There's a few major issues.  These really should be fixed before
sending it to the IESG, IMO, but with these fixed I'd be happy to
see it shipped.

Major:

1. This should be proposed standard, not experimental.

This assigns a new and incompatible meaning to an IP header codepoint
in a way that can't be rolled back or kept isolated, with declared
intent to roll it out widely.  This should be considered a proposed
standard action and make an update to RFCs 3168 and 8311 that clearly
states that any existing marking queues should be changed where
possible to treat ECT(1) as either L4S or Not-ECT, even if the device
is not participating in an L4S-related experiment.

Such an update would align the specs with the stated expectations
that monitoring requirements should be in place mainly as a legacy
workaround, and would provide appropriate and potentially actionable
standards track mitigations that are available to on-path devices
implicated in any problems that are observed, whereas the specs as
currently written have none.

I understand that RFC 8311 was originally intended to cover this, but
that was published back when the L4S proposal still intended to make
the ECT(1)-marking compatible with classic marking via a robust
detection scheme, and RFC 8311 embeds that assumption throughout.  Now
that the proposal has changed to rolling this out without solving the
coexistence problem, RFC 8311's provisions are no longer adequate to
cover this IMO, and must be updated if we intend to have a self-
consistent set of specs published.

(Alternately, a separate standards-track doc could recommend changing
ECT(1) handling to Not-ECT, but I really think there's no point
separating them, and such a doc would still have to be a prerequisite
for moving this forward IMO.  I suggest that the most expedient
path as well as the better choice here is to make l4s-id standards
track and to add the Not-ECT option for ECT(1) here as a reasonable
problem mitigation strategy available to marking bottlenecks.)

2. This text from Section 4.3 should be strengthened to recommend
detection of paths that might be classic-marking L4S traffic (shared
queue or not):

   o  In uncontrolled environments, monitoring MUST be implemented to
      support detection of problems with an ECN-capable AQM at the path
      bottleneck that appears not to support L4S and might be in a
      shared queue.

Even in fq, the standing queue that will be built by a scalable cc with
classic marking is contrary to "1-2ms latency at 99th percentile" goal
of L4S, even leaving aside the harm to colliding flows (though of course
the harm to colliding flows, including from hash collisions in fq, is
also another reason to make this change).

3. This text from Section 4.3 should be strengthened to require that
sending L4S traffic in uncontrolled environments does not happen when
classic marking of L4S traffic is detected for a shared queue, and at
least recommend that it not happen even for fq:

   o  In uncontrolled environments, monitoring MUST be implemented to
      support detection of problems with an ECN-capable AQM at the path
      bottleneck that appears not to support L4S and might be in a
      shared queue.  Such monitoring SHOULD be applied to live traffic
      that is using Scalable congestion control.  Alternatively,
      monitoring need not be applied to live traffic, if monitoring has
      been arranged to cover the paths that live traffic takes through
      uncontrolled environments. 

The current text requires monitoring, but only gives a single SHOULD for
live traffic, plus non-normatively permits one alternative.  This allows
operators to monitor but not cut off (so this requirement as currently
written would be satisfied by write-only logging for instance, with the
SHOULD easily dismissable with an "implementation complexity" hand-wave
while still following the spec).

Suggested alternative, feel free to edit:

   o  In uncontrolled environments, L4S traffic MUST NOT be sent without
      monitoring to detect marking of L4S traffic by non-L4S bottlenecks.
      This monitoring can for example be performed on live traffic, or
      can rely on monitoring that covers the paths live traffic takes
      through uncontrolled environments.  Where non-L4S bottlenecks are
      observed marking L4S traffic, L4S sending MUST be disabled if the
      bottleneck is a shared queue, and SHOULD be disabled if it is FQ.

4. Although l4s-arch claims that l4s-id satisfies the RFC 4774
requirements, it's hard to tell whether it does so.  Specifically:

4.a. From section 7 of RFC 4774:
   Specifications of alternate ECN semantics must clearly state how they
   address the issues raised in this document, particularly the issues
   discussed in Section 2.

I don't see how issues 2 and 3 from section 2 are covered in l4s-id.
From section 4 of RFC 4774:
   (2) How does the possible presence of old routers affect the
       performance of the alternate-ECN connections?

   (3) How does the possible presence of old routers affect the
       coexistence of the alternate-ECN traffic with competing traffic
       on the path?

   When alternate semantics are defined for the ECN field, it is
   necessary to ensure that there are no problems caused by old routers
   along the path that don't understand the alternate ECN semantics.

4.b. Section 4 goes on to describe 3 options for how alternate ECN
semantics should be treated.  I don't see a claim in the L4S docs
specifying which of the 3 options the L4S spec for alternate ECN semantics
matches, but it implicitly appears to be trying to say it's option 3
(unsafe) and asserting that the detection and adaptive response satisfies
the requirement for isolation on option 3, I think?

Maybe there are sections in l4s-id that intend to cover these points,
and there just needs to be text listing what they are, but I don't think
the link is obvious enough to satisfy the "clearly state" requirement
from RFC 4774.  So it would be very helpful to add a list of references
to sections that are intended to address these RFC 4774 requirements.

Nits:
- the list in section 7.1 has a weird formatting problem for the sub-
  list:

   o  Did use of L4S over the Internet result in improvements to the
      following metrics:

   o

      *  queue delay (mean and 99th percentile) under various loads;

----

draft-ietf-tsvwg-l4s-arch-10:
Summary: Almost ready

+1 to Gorry's comments here, especially regarding the use of "all traffic":
https://mailarchive.ietf.org/arch/msg/tsvwg/vMMsQpXs65lk1E7NpV5RlmpyqdI/

Minor:
1. l4s-arch section 1:
   It has been demonstrated that, once access network bit rates reach
   levels now common in the developed world, increasing capacity offers
   diminishing returns if latency (delay) is not addressed.
- This needs a reference.

Editorial:
1. l4s-arch 4.2 section a:
- This is a confusing wall of text.  I think it would be better to
  give a much briefer summary here with a reference.  Exposition
  like "the obvious part" and "the less obvious part" are a minus
  here--I don't think the obviousness claims made here generalize
  well.

2. All the uses of underlining for emphasis are a minus.  The
  places where it seems necessary or useful are a good hint that
  the text on its own is not adequately capturing the intended
  meaning.
  Leaning on this kind of toned emphasis makes assumptions about
  connotations that don't hold very well even for native English
  speakers and break down entirely for non-native speakers, so they
  are generally out of place in a technical document that will need
  to be correctly interpreted by an international audience with many
  non-native speakers, IMO.

----

I'm going to be too pressed for time to do a more detailed review,
but I wanted to get the above comments in.

As a final aside, I'd like to see this happen.  The only goal I'm
pursuing at this point wrt this work is avoiding preventable harm
from pushing this out in a way that's likely to cause confusion
when and if problems are encountered in production.  (In particular,
I have given up on efforts to improve the signaling design, since
the authors have rejected all such suggestions and this work needs
to get moved out of the wg one way or another.)

Best,
Jake

On 07-29, 9:18 AM, "Wesley Eddy" <[email-masked]> wrote:

- DualQ:

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 21 Aug 2021 20:12:38 +0000

From: Bob Briscoe <[email-masked]>
> Date: Tue,2021-08-17 at 10:19 AM
> •         Redirecting ECT(1) to ECT(0) treatment would be foolish because the sender would still behave as L4S
> •         Redirecting ECT(1) to Not-ECT treatment would be effective, by inducing a drop response at the sender

+1, Not-ECT seems like the clear choice here.

> Another view has been expressed (off list) that we should not decide at this point, but instead wait until the problem needs a solution. I would argue that this 'MUST' needs to be implemented from the start, and asking implementers to include options for future flexibility needs to be tempered by whether ECT(0) treatment would ever be useful in any future scenario.

+1 again, this should be decided up front IMO.

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 22 Aug 2021 17:56:28 -0700

> 1. This should be proposed standard, not experimental.

+1.

Jake's words of explanation are much more eloquent than any I could come up
with so I won't repeat them. I don't really have a dog in the fight; those
who do may disagree; but for the good of the Internet, I do not believe
that L4S should go forward as an experiment. It is not compatible with RFC
3168, and it should either obsolete RFC 3168 or not go forward at all.

Mike Heard

---

From: Steven Blake <[email-masked]>
Date: Sun, 22 Aug 2021 22:19:01 -0400

On Sun, 2021-08-22 at 17:56 -0700, C. M. Heard wrote:
> Jake's words of explanation are much more eloquent than any I could
> come up with so I won't repeat them. I don't really have a dog in the
> fight; those who do may disagree; but for the good of the Internet, I
> do not believe that L4S should go forward as an experiment. It is not
> compatible with RFC 3168, and it should either obsolete RFC 3168 or
> not go forward at all.
> 
> Mike Heard

I fully agree as well.

Regards,

// Steve

---

From: Dave Taht <[email-masked]>
Date: Mon, 23 Aug 2021 17:54:19 -0700

As one of the contributors to codel, pie, fq-codel, cake, and fq-pie
in linux, openwrt, ns2, and ns3, and as the co-founder of the
Bufferbloat Project, I have re-read the L4S drafts, and do not support
forwarding them on to the IESG.

The experiment(s) are over. The technical evidence against the core
L4S approaches is overwhelming. Rather than proceed to any RFC status
these documents should depart into the same memory hole as source
quench did.

Rather than re-iterate technical point after technical point here...
if, in this last call, these drafts are approved to send to the IESG,
I request that the entirety of the SCE related data[1] be attached as
an appendix, much as Richard Feynman affixed his Appendix A to the
Challenger Commission report, lest this be the last triumph of process
and politics over sound science and engineering in the IETF.

In the long run, existing transport responses to RFC3168-style ECN
handling should be improved, and the more permissive aspects of
RFC8311 repudiated, in favor of a consistent, well understood, IP
layer and internet wide, sharp congestion response to a single CE
mark, as is the default, today.

[1] https://github.com/heistp

-- 
Fixing Starlink's Latencies: https://www.youtube.com/watch?v=c9gLo6Xrwgw
Dave Täht CEO, TekLibre, LLC

---

From: Luca Muscariello <[email-masked]>
Date: Tue, 24 Aug 2021 13:16:14 +0200

Hi,

by considering the large amount of data and
experiments that show the significant issues left unsolved,
it is my  opinion that these drafts should not move forward
until these issues are resolved.

I also have little faith that the long list of technical
issues can be resolved and yet, three drafts have been
last called.

Regards
Luca

---

From: David Pullen <[email-masked]>
Date: Tue, 24 Aug 2021 14:14:06 -0400

Hi all,

Broadcom has implemented support for L4S in Cable Modem products and
supports moving the drafts to RFC.

Thanks,
  - Dave

David Pullen
Distinguished Engineer  |  BVG, Cable Modems
Broadcom

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

---

From: "Ranganathan, Ram" <[email-masked]>
Date: Tue, 24 Aug 2021 18:19:56 +0000

We (CommScope) are actively working on L4S support in our labs/test setups (DOCSIS based network) and see the value proposition brought by the protocol.

We support moving these working group drafts to RFC.

Thank you
Ram

Ram Ranganathan
Director, Systems Engineering
CTO Office – Broadband Networks

COMMSCOPE
now meets next

90 Matheson Blvd. West - Suite 410
Mississauga ON
L5R 3R3 Canada
Phone: +1-905-568-7317

Learn how CommScope is shaping the networks of the future<https://www.commscope.com/>

Hi all,

Broadcom has implemented support for L4S in Cable Modem products and supports moving the drafts to RFC.

Thanks,
  - Dave

David Pullen

On Thu, Aug 19, 2021 at 1:30 PM Glenn Deen <[email-masked]<mailto:[email-masked]>> wrote:
I support moving these drafts forward.

-Glenn Deen

On 29 Jul 2021, at 9:18, Wesley Eddy wrote:

> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/<https://secure-web.cisco.com/1GpperBysmvVF1MstLFVLUypBwExAYL1zrdiPtipsLf0U24CdIRmqhKIsMofesS5j0S8syRQ4TqFOAbbmgG44fDTTVN-gVcydKWCS4S8RU4be6OKju-wfHQ35-OjNWP9aY4oz5rj19AiKmlK_hx6CwiuYAVrXBFXdxW-ePPCGLdtWzOzdXzADdhDBzzMvYj1pJ1Kq-B5OAdDPy8CXfyHRsS_W0SsFuG7UGcz3Hqws_-tT3fxCA01qMLCHba2gLkDVlGtggKS6V1c3dl4t-bE2KwNZKgGpwsKocZQEamCZfZ8sdt6IyR56J63LjgbccBQe/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-arch%2F>
>
> - DualQ:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/<https://secure-web.cisco.com/1kdMUsAf9sIPDfIH5oTSzVq0eNug4uzrZ1UZFHWCF5DG4dWDqXPYP_Eabojjx7Ao5HYasDWQ4XOGdN7ZsGdKrFzS4YCXMTwjN6USOL_axtsWfMQeZ2-g1BlOsZoJR-Xmy1SjhSJAz4RF9oINVaxnskQdyYXvpiXmeZECFCfGwEknSb0EtJl4wHS1BtQ3dKyRZVmbhNL7NjhAD-AipVACh0fJT7Lf4wpvAxk1nJYJPpHQBdRfJ_3Tf5OFUSeT3h5M-deK_amUWBCd6zfWWVFW59F97zoEGSRmmJZRFxlPYAe4p_mUoBc5Dh9Ky7G-w6OCr/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-coupled%2F>
>
> - ECN ID:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/<https://secure-web.cisco.com/1y7pXMUUoAoov99jQqx4-4vy38esB-SngBCSYL8MadELskaknSCf5JLCCmKDuQf2UYCqRO-ebQl1DIhIuQ5ehi23m87Y0ky9uku-YSONsFrMncRBXXpGM6XEZDfxeI4VuMSpynhkpskS1OY2RmNfbJB35qibPHbtPodcXAsOJRL-1u3Ub_WrhToZIpJLylN1EwQSxLEzTFvWekJSlKgTP165aQLqu6sOde1pAjeFiDbbcFm9ruraL-lhYFvuKAWaXcNQCuxgFlpIc5v2oG6PS8C3D-MXMIO1B53xgPN2-kd8PVHzYI2gPLyulSk080Yuc/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-id%2F>
>

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

---

From: Adi Masputra <[email-masked]>
Date: Tue, 24 Aug 2021 12:01:19 -0700

I support advancing these L4S drafts to RFC status.

Thanks,
Adi

> On Jul 29, 2021, at 09:18, Wesley Eddy <[email-masked]> wrote:
>

---

From: Pete Heist <[email-masked]>
Date: Tue, 24 Aug 2021 22:27:52 +0200

I have read the latest versions of all three drafts in full, and am
against moving them forward, primarily for these reasons:

1) The incompatible redefinition of CE via ECT(1) and its consequences
2) The parallel, biased network path with increased RTT unfairness
3) Utilization regressions with bursty traffic and link layers

I've expanded on these below with links to test results, and commented
on some quotes from the drafts.

Overall, I'd like to be certain that we're choosing a robust signaling
design without ambiguity, and that we've settled on a system that's
useful for all Internet paths, not just certain link layers, at certain
RTTs, for certain industries. The utilization regressions also mean
that a key goal from the l4s-arch Abstract has not been reached (see
l4s-arch comments below).

Pete

-----

# Detail on reasons for opposition

1) The incompatible redefinition of CE via ECT(1) and its consequences

   I'll split these into consequences that apply *before* all existing
   middleboxes are patched to understand ECT(1) as the L4S ID, and
   those that still apply after.

   Consequences *before* existing middleboxes understand ECT(1) as the
   L4S ID:

   - Competing flows in RFC3168 shared queues may be dominated, causing
     unfairness, increased FCT or video streaming interruptions:
    
https://github.com/heistp/l4s-tests/#unsafety-in-shared-rfc3168-queues
     https://github.com/heistp/ccafct#sample-output
    
https://github.com/heistp/l4s-tests/#harm-to-video-streaming-services

   - FQ can prevent this, but its flow isolation can be defeated by
     tunnels, hash collisions and qdisc hash customization:
    
https://github.com/heistp/l4s-tests/#tunneled-non-l4s-flows-not-protected-by-fq
     https://github.com/heistp/qdisc-custom-classification/

   - L4S transports respond incorrectly to the CE marks sent by current
     AQMs, causing them to inflate their own queues, regardless of
     whether or not FQ is in use:
    
https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes-from-underreaction-to-rfc3168-ce

   - The meaning of CE is ambiguous for monitoring tools. "Making
     congestion visible" is listed as an ECN benefit in RFC8087 Section
     2.5.

   Consequences even *after* existing middleboxes understand ECT(1) as
   L4S ID:

   - Packet classification is lost after CE is marked, so subsequent
     middleboxes must assume a meaning for CE, leading to:

     - Dropped packets for tunnels with anti-replay protection, and the
       potential to use this in a DoS attack:
      
https://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled

     - DualQ reordering CE marks within RFC3168 flows. This is argued
       to be low risk in l4s-id Appendix B, but the tunnel anti-replay
       issue was not thought or for some time, and we may well have
       missed other issues.

     - The potential for poor performance with firewalls that may drop
       reordered packets or terminate flows based on reordering
       criteria.

   - The receiver does not know which CE it has received. The sender
     *may* know the meaning of CE, only if it keeps track of or knows
     which ECT codepoint it sent on the corresponding packet.

   - The meaning of CE is more complicated to ascertain for monitoring
     tools. It would have to keep track flow state at a minimum, and
     still wouldn't necessarily work for tunneled flows.

2) The parallel, biased network path with increased RTT unfairness

   - Our tests suggest that by default, non-L4S flows receive about 50%
     the capacity of L4S flows at the same RTT (characterized as
     "roughly the same flow rate" in l4s-id Section 1):
     https://github.com/heistp/l4s-tests/#network-bias

     50 Mbps / 10ms RTT: Prague 34.09 Mbps, CUBIC 13.04 Mbps
     50 Mbps / 20ms RTT: Prague 31.85 Mbps, CUBIC 15.51 Mbps
     50 Mbps / 80ms RTT: Prague 29.92 Mbps, CUBIC 17.48 Mbps

   - When RTTs differ, the resulting fairness regresses from current
     behavior:
     https://github.com/heistp/l4s-tests/#rtt-unfairness 

     50 Mbps pfifo: CUBIC(10ms) 30.27 Mbps, CUBIC (160ms) 17.22 Mbps
     50 Mbps DualPI2: Prague(10ms) 41.67 Mbps, CUBIC (160ms) 4.87 Mbps
     50 Mbps DualPI2: Prague(10ms) 45.46 Mbps, Prague (160ms) 1.56 Mbps

3) Utilization regressions with bursty traffic and link layers

   - L4S transports in the L queue can underutilize the capacity when
     there is bursty traffic from common applications in the C queue,
     relative to the performance of CUBIC in the same scenario, with
     existing RFC3168 AQMs:
    
https://github.com/heistp/l4s-tests/#underutilization-with-bursty-traffic

   - L4S transports can underutilize the capacity when the packets have
     passed through a bursty link layer such as a wireless device,
     relative to the performance of CUBIC in DualPI2(C) or fq_codel:
    
https://github.com/heistp/l4s-tests/#underutilization-with-bursty-links

# Comments on l4s-arch-10

> l4s-arch Abstract:
>
> These mechanisms aim to ensure that the latency and throughput
> performance using an L4S-compliant congestion controller is usually
> much better (and never worse) than the performance would have been
> using a 'Classic' congestion controller, and that competing flows
> continuing to use 'Classic' controllers are typically not impacted by
> the presence of L4S.

The goal "and never worse" was not reached, as demonstrated by the
throughput performance with bursty traffic and links, relative to the
performance for conventional CCAs and AQMs (see #3 above).

Classic CCAs may be impacted by the presence of L4S (see #1 and #2
above).

> l4s-arch Section 1:
>
> Therefore, the goal is an Internet service with very Low queueing
> Latency, very Low Loss and Scalable throughput (L4S).  Very low
> queuing latency means less than 1 millisecond (ms) on average and
> less than about 2 ms at the 99th percentile.

The latency goal only appears attainable on certain link layers at
present, under certain conditions. Between-flow induced delay from
bursty links, such as competing WiFi traffic, can exceed it:

https://github.com/heistp/l4s-tests/#between-flow-induced-delay

> l4s-arch Section 2(1):
>
> The Dual Queue Coupled AQM was developed as a minimal complexity
> solution to this problem.  It acts like a 'semi-permeable' membrane
> that partitions latency but not bandwidth.  As such, the two queues
> are for transition from Classic to L4S behaviour, not bandwidth
> prioritization.

We have seen a bias in favor of L4S over non-L4S flows (see #2 above).

> l4s-arch Section 4.1(b):
>
> [I-D.ietf-tsvwg-ecn-l4s-id] recommends ECT(1) is used as the
> identifier to classify L4S packets into a separate treatment from
> Classic packets.  This satisfies the requirements for identifying an
> alternative ECN treatment in [RFC4774]. 

It's not clear to me that the requirements are satisfied, given the
text in RFC4774 below, in combination with the test results referred to
in #1 above.

  > RFC4774 Section 4:
  > necessary to ensure that there are no problems caused by old
  > routers along the path that don't understand the alternate ECN
  > semantics.

> l4s-arch Section 5.1:
>
> The consequent smaller amplitude sawteeth fit between an empty queue
> and a very shallow marking threshold (~1 ms in the public Internet),
> so queue delay variation can be very low, without risk of under-
> utilization.

Underutilization can be seen with bursty traffic and bursty links
(see #3 above).

> l4s-arch Section 5.2:
>
> In contrast, because L4S is for all traffic, it needs none of the
> management baggage (traffic policing, traffic contracts) associated
> with favouring some packets over others.  This baggage has probably
> held Diffserv back from widespread end-to-end deployment.

Comments on DiffServ seem like they'd be better made elsewhere.

> l4s-arch Section 5.2:
>
> The L4S approach resolves this tension by ensuring hosts can minimize
> the size of their sawteeth without appearing so aggressive to Classic
> flows that they starve them.

With current middleboxes that don't understand ECT(1), that can happen
(see #1 above).

> l4s-arch Section 5.2:
>
> Per-flow implementations still need to have support for scalable
> congestion control added, which has already been done in FQ-CoDel
> (see Sec.5.2.7 of [RFC8290]).

This is a bit misleading as these are not L4S AQMs but just have a
ce_threshold parameter. L4S support would require treating ECT(1)
traffic differently from ECT(0).

> l4s-arch Section 6.4.2(1):
>
> If no AQM at all was previously deployed for the downstream access,
> an L4S AQM greatly improves the Classic service (as well as adding
> the L4S service).  If an AQM was already deployed, the Classic
> service will be unchanged (and L4S will add an improvement on top).

If a Codel AQM was already deployed, then switching to PIE with the
default targets used for the C queue may increase average latency,
relative to the latency with Codel's defaults.

> l4s-arch Section 6.4.2(2):
>
> In this stage, the name 'TCP Prague'
> [I-D.briscoe-iccrg-prague-congestion-control] is used to represent a
> variant of DCTCP that is safe to use in a production Internet
> environment.

It's only safe to use in production when all existing middleboxes
understand ECT(1) to be the L4S ID, which at present not all do (see #1
above).

# Comments on l4s-id-19

> l4s-id Abstract:
>
> L4S uses 'scalable' congestion control, which induces much more
> frequent control signals from the network and it responds to them
> with much more fine-grained adjustments, so that very low (typically
> sub-millisecond on average) and consistently low queuing delay
> becomes possible for L4S traffic without compromising link
> utilization.  Thus even capacity-seeking (TCP-like) traffic can have
> high bandwidth and very low delay at the same time, even during
> periods of high traffic load.

Testing suggests lower link utilization with bursty links and/or bursty
competing traffic (see #3 above).

> l4s-id Introduction:
>
> The DualQ Coupled AQM, which is defined in a complementary
> experimental specification [I-D.ietf-tsvwg-aqm-dualq-coupled], is an
> AQM framework that enables scalable congestion controls derived from
> DCTCP to co-exist with existing traffic, each getting roughly the
> same flow rate when they compete under similar conditions. 

Testing DualPI2 with the defaults showed about a 2x advantage in
throughput for L4S flows vs non-L4S flows, at the same RTT (see #2
above).

> l4s-id Section 5.1:
>
> In case unforeseen problems arise with the L4S experiment, it MUST be
> they were {ToDo: Not-ECT / ECT(0) ?}.

This ToDo needs completion.

> l4s-id Section 5.4.1.2:
>
> An operator that excludes traffic carrying the L4S identifier from
> L4S treatment MUST NOT treat such traffic as if it carries the ECT(0)
> codepoint, which could confuse the sender.

It might not directly relate to the section, but in order to comply
with this MUST NOT, the operator would also have to make sure that all
existing RFC3168 middleboxes are undeployed (see #1 above).

> l4s-id Section 5.5:
>
> As well as senders needing to limit packet bursts (Section 4.3),
> links need to limit the degree of burstiness they introduce.  ...  as
> suggested in Section 4.3 it would not seem necessary to limit bursts
> lower than roughly 10% of the minimum base RTT expected in the
> typical deployment scenario (e.g. 250 us burst duration for links
> within the public Internet).

250 us is about 1 packet at 50 Mbps, with a 1500 byte MTU. WiFi
aggregates can reach 4 ms. Ubiquiti's fixed framing modes can be either
5ms or 10ms.  Other possible sources of bursts in DSL:
https://mailarchive.ietf.org/arch/msg/tsvwg/s3zYnEhnTnncMW2fIKwux753rok/

At least with today's link technologies, 250 us seems like a difficult
burst duration to stay under. I think we at least want to know more
about different link types that are affected by this and what the
affects are on latency and utilization. See utilization tests in #3
above, and:
https://github.com/heistp/l4s-tests/#between-flow-induced-delay

# Comments on dualq-coupled-16

> dualq-coupled Abstract:
>
> Analytical study and implementation testing of the coupled aqm have
> shown that scalable and classic flows competing under similar
> conditions run at roughly the same rate.

Testing DualPI2 with the defaults showed about a 2x advantage in
throughput for L4S flows vs non-L4S flows, at the same RTT (see #2
above). "Roughly the same rate" isn't quantified, but the results seem
different from that.

> dualq-coupled Abstract:
>
> When tested in a residential broadband setting, DCTCP also achieves
> sub-millisecond average queuing delay and zero congestion loss under
> a wide range of mixes of DCTCP and `Classic' broadband Internet
> traffic, without compromising the performance of the Classic traffic.
> The solution has low complexity and requires no configuration for the
> public Internet.

Bursty traffic or link layers can raise the limits on low queueing
delay:

https://github.com/heistp/l4s-tests/#between-flow-induced-delay

> dualq-coupled Section 1.1:
>
> In contrast, dualq coupled aqms addresses the root cause of the
> latency problem --- they are an enabler for the smooth low latency
> scalable behaviour of scalable congestion controls, so that every
> packet in every flow can enjoy very low latency, then there is no
> need to isolate each flow into a separate queue.

I would like to be sure that this is true, not just with bursty traffic
and link layers, but also cross-flow traffic (tests needed on the
latter).

> dualq-coupled Section 1.4:
>
> Thousands of tests have been conducted in a typical fixed residential
> broadband setting.  experiments used a range of base round trip
> delays up to 100ms and link rates up to 200 mb/s between the data
> centre and home network, with varying amounts of background traffic
> in both queues.

It sounds like these tests were performed on a certain type of link
(residential cable?), in a certain RTT range (below 100ms). Bursty
links and higher RTTs should also be tested if L4S is to be used for
general purpose congestion control (see #3 above). We should end up
with a system that works well under most all conditions, otherwise it's
not clear how users and admins will know to change CCAs from one
activity to the next.

> dualq-coupled Section 4.1.3:
>
> Experiments with the DualPI2 AQM (Appendix A) have shown that
> introducing 'drop on saturation' at 100% L4S marking addresses this
> problem with unresponsive ECN as well as addressing the saturation
> problem.  It leaves only a small range of congestion levels where
> unresponsive traffic gains any advantage from using the ECN
> capability, and the advantage is hardly detectable [DualQ-Test].

I might be misunderstanding the text, but two-flow tests of traffic
that is unresponsive to CE show that, as with RFC3168 ECN, there is an
advantage to setting ECT(0) or ECT(1), then not responding to CE,
that's easy to detect:

https://sce.dnsmgr.net/results/ect1-2020-05-20-s11-ce-unresponsive/l4s-s11-ce-unresponsive/

---

From: Christoph Paasch <[email-masked]>
Date: Tue, 24 Aug 2021 13:41:59 -0700

I support advancing these 3 L4S drafts to RFC status.

Christoph

On 07/29/21 - 12:18, Wesley Eddy wrote:
> The WGLC will last through 4 weeks from today, and then we'll see what to do
> next.  Please submit any comments you have on these to the TSVWG list in
> that timeframe.
> 
> The work on the L4S operational guidance draft is continuing in parallel,
> but that draft is not being last called yet.
> 
>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Tue, 24 Aug 2021 21:35:24 +0000

Hi Wes,

As one of the co-authors I also support progressing the L4S drafts to RFC.

Nokia has implemented L4S in products, even on WiFi. Despite the limitations (latency and jitter) of Physical and MAC protocols like WiFi, it is possible with L4S to get the queuing latency consistently down to the bare minimum needed to support those technologies. It is just a matter of making it work as optimal as possible today and letting L4S drive the design of future Physical and MAC protocols to make their latency and jitter reduction capabilities match those of L4S senders.

Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
Sent: Thursday, July 29, 2021 6:18 PM
To: [email-masked]
Subject: [tsvwg] start of WGLC on L4S drafts

- DualQ:

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 24 Aug 2021 23:57:57 +0100

Mirja, see [BB] inline...

On 18/08/2021 09:41, Mirja Kuehlewind wrote:
>
> Hi Bob,
>
> see three replies below.
>
> *From: *tsvwg <[email-masked]> on behalf of Bob Briscoe 
> <[email-masked]>
> *Date: *Tuesday, 17. August 2021 at 18:01
> *To: *Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>
> *Subject: *[tsvwg] L4S Editorial Reviews (was: start of WGLC on L4S 
> drafts)
>
> Mirja,
>
> [MK] This rebalance is fine but now it feels like there is a figure 
> missing in section 2. Therefore I proposed to _/also/_ add the figure 
> to section 2 but maybe a slightly simplified version without the 
> details of dualQ, e.g. just have some box that says “L4S-aware AQM” or 
> something. The important part would be to show the different 
> components (Scalable CC in the endpoint, 
> Identification/Classification, and AQM) and explain how the “work” 
> together.
>

[BB] OK, I'll try. This is going to be a pretty content-free diagram if 
it's going to cover both FQ & DualQ, 'cos there isn't much in common 
between them.

> [MK] Actually looking at this sentence again, the whole sentence is 
> not super clear. I think rather than saying “it’s similar to classic 
> ECN except it’s not”, it would be better to say what is different. In 
> other words I would actually remove this sentence and rather say (at 
> the end of this paragraph) something like: “This draft specifies the 
> use of the ECN ECT(1) code point as an identifier for L4S traffic.”
>

[BB] The differences from RFC3168 are primarily the behaviour, which is 
what this draft mainly describes, not just the identifier.  But I've had 
previous resistance to saying it's similar to classic ECN, so I am going 
to have to improve it somehow (without making the abstract even longer). 
How about:

    An L4S sender sets the ECT(1) codepoint in the ECN field of the IP header to indicate
    that it is using an L4S congestion control. Then if there is Active Queue Management (AQM) that
    supports L4S on the path, it applies an L4S marking behaviour to such packets.
    The L4S protocol on the wire broadly follows the original (or 'Classic') ECN protocol, but this document
    defines both the transport layer behaviour and the network node behaviour needed to support L4S.

> [BB] I think it's useful to give the names in full in this figure, 
> isn't it? However, given it talks about the L queue and C queue below 
> that, I'll say
>     L4S (L) queue
>     Classic (C) queue
> OK?
>
> [MK] Section 2.3 says:
>
> So my understanding was that L is actually an abbreviation for “low 
> latency” and “L4S”. However looking at the terminology it says:
>

[BB] Yes, this needs to be made consistent.

In ecn-l4s-id it says:

    In this case it would not be appropriate to call the queue an L4S
    queue, because it is shared by L4S and non-L4S traffic.  Instead it
    will be called the low latency or L queue.  The L queue then offers
    two different treatments:

    o  The L4S treatment, which is a combination of the L4S AQM treatment
       and a priority scheduling treatment;

    o  The low latency treatment, which is solely the priority scheduling
       treatment, without ECN-marking by the AQM.

In aqm-dualq-coupled under the definitions of the letters L and C, there 
is a sentence saying "Both Classic and L4S services can cope with a 
proportion of unresponsive or less-responsive traffic as well, ..."

I propose that
a) L always denotes 'low latency' more generally, not just L4S.
b) we use the term "non-congestion controlled low latency treatment" not 
just "low latency treatment" for the non-L4S treatment in the L queue.

I think I can then make everything consistent. I'll see whether this 
works when I actually edit it.

Thanks

Bob

>
> Mirja
>
>
>
>
>
>     On 29.07.21, 18:18, "tsvwg on behalf of Wesley Eddy"<[email-masked] on behalf of [email-masked]>  <mailto:[email-masked]@mti-systems.com>  wrote:
>
>          - Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/  <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/>
>
>          - DualQ:
>
>          https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/  <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/>
>
>          - ECN ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/  <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/>
>
> Bob Briscoehttp://bobbriscoe.net/  <https://protect2.fireeye.com/v1/url?k=83fd01ee-dc66393c-83fd4175-866132fe445e-2ea15d5d62688d61&q=1&e=8f894785-ae8d-45d8-a094-22557c4afbba&u=http%3A%2F%2Fbobbriscoe.net%2F>

---

From: "Flinck, Hannu (Nokia - FI/Espoo)" <[email-masked]>
Date: Wed, 25 Aug 2021 09:32:59 +0000

Hello

I support the publication of these three documents as RFCs. They have gone through we detailed elaboration and updates over the years so they are mature for publication. 

Best regards
Hannu

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
Sent: Thursday, July 29, 2021 7:18 PM
To: [email-masked]
Subject: [tsvwg] start of WGLC on L4S drafts

- DualQ:

---

From: Sowmini Varadhan <[email-masked]>
Date: Wed, 25 Aug 2021 07:17:31 -0400

[Apologies for re-threading, missed the original thread when
I accidentally dropped my tsvwg subscription in between a job 
transition]

I support the publication of the drafts below to RFC.

On (08/25/21 10:34), Bob Briscoe wrote:
>    -------- Forwarded Message --------
> 
>       Date: Thu, 29 Jul 2021 12:18:10 -0400     
>       From: Wesley Eddy <[email-masked]>   
>         To: [email-masked] <[email-masked]>     
> 

--Sowmini

---

From: "Ozer, Sebnem" <[email-masked]>
Date: Wed, 25 Aug 2021 13:54:02 +0000

I support advancing the 3 L4S drafts listed below to RFC status.

Thanks,
Sebnem

On 07/29/21 - 12:18, Wesley Eddy wrote:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-iet
> f-tsvwg-l4s-arch/__;!!CQl3mcHX2A!S3YT3o-fk40wRDlrW2uevEYdSQNZzQZmo3ftt
> cA4hovuYaAUG7Go4UTSytxu8u9JQFg$
>
> - DualQ:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-iet
> f-tsvwg-aqm-dualq-coupled/__;!!CQl3mcHX2A!S3YT3o-fk40wRDlrW2uevEYdSQNZ
> zQZmo3fttcA4hovuYaAUG7Go4UTSytxuyxdN6Fg$
>
> - ECN ID:
> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-iet
> f-tsvwg-ecn-l4s-id/__;!!CQl3mcHX2A!S3YT3o-fk40wRDlrW2uevEYdSQNZzQZmo3f
> ttcA4hovuYaAUG7Go4UTSytxud6gnV2A$
>

---

From: Tommy Pauly <[email-masked]>
Date: Wed, 25 Aug 2021 07:24:53 -0700

Hello,

I’ve reviewed the documents and support publication. I particularly appreciate the background and definitions presented clearly in the architecture document.

Best,
Tommy

---

From: Asad Sajjad Ahmed <[email-masked]>
Date: Wed, 25 Aug 2021 19:13:29 +0200

I support advancing these L4S drafts for RFC publication.

Asad

---

From: Uma Chunduri <[email-masked]>
Date: Wed, 25 Aug 2021 10:58:12 -0700

I support advancing these drafts towards publication.

--
Uma C.

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 25 Aug 2021 18:18:04 +0000

Hi Wes, List,

I support these drafts for publication.
Enabling to operate shallow queues regardless of the BDP is key to support the efforts to reduce latencies of the lower layers in their relevant bodies.

Best,
Olivier
________________________________
From: tsvwg <[email-masked]> on behalf of Uma Chunduri <[email-masked]>
Sent: Wednesday, August 25, 2021 7:58:12 PM
To: Wesley Eddy <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

--
Uma C.

On Thu, Jul 29, 2021 at 9:18 AM Wesley Eddy <[email-masked]<mailto:[email-masked]>> wrote:
This message is starting a combined working group last call on 3 of the
L4S drafts:

- DualQ:

---

From: <[email-masked]>
Date: Thu, 26 Aug 2021 08:08:10 +0000

Some comments on the Architecture draft.

It is a nice document, thanks for all the work.
Some minor suggestions that I think are easy to resolve.

Section 1 Intro
Para 1- maybe some refs could be added, eg "spikes of hundreds of milliseconds are common" and "queuing is typically configured to cause overall network delay to roughly double relative to expected base (unloaded) path delay."

Section 2 L4S Architecture Overview
I think you could expand this section a little to provide a 'protocol model' (RFC4101: "to answer three basic questions:   1. What problem is the protocol trying to achieve?     2. What messages are being transmitted and what do they mean?     3. What are the important, but unobvious, features of the protocol?")
The intro does a good job of answering question 1.
For me, S2 should lay out (at a high-level) the solution. The current Section gives the 'components' without spelling out how they fit together, and also includes justification and background (about state of the art), which I think would be better to discuss separately. Also, it's stated that ECT(1) and CE are used, but not how. I think it would be good to include a figure here (Fig 1, or some version)

Section 4.2
Fig 1
The '1' between the functions Mark and Scheduler could be explained?

I found S4.2 a bit hard to parse. I think my issue is that the intro para implies that bullets are about different per flow options. Bullets b and c are about this, but bullet a is (I think) a solution without per flow, and is actually the preferred approach. Bullet a also includes explanation of the dual queue that would apply to any solution. 
Perhaps: re-write the intro para, especially delete the "Nevertheless" sentence, make bullet (a) main text and explain dual q etc. Then say there are different per flow options, and start bullet (a) by stating it is the option without per-flow.

Section 8.3
Is "catastrophically" the best word?

Best wishes
Phil

> Sent: 29 July 2021 17:18
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf
> .org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-
> arch%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed467
> 1142f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637
> 631723313409962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=fATwA9c
> bixSYOkh3vM2IugJ29oPEpvdgEIxbhAx6lMw%3D&amp;reserved=0
> 
> - DualQ:
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf
> .org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-
> coupled%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed
> 4671142f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C
> 637631723313419949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PR8Rj
> 3%2BmElcld7NBD7TxOgdkkvEyvbjda2qeUANy%2FAY%3D&amp;reserved=0
> 
> - ECN ID:
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf
> .org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-
> id%2F&amp;data=04%7C01%7Cphilip.eardley%40bt.com%7Cb44881804bed46711
> 42f08d952ac8c68%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C63763
> 1723313419949%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj
> oiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=oAg1Pi24Z
> Uc4lZK9zwtXA95tm%2F%2B5Y9V2Fz8pEmBMv9g%3D&amp;reserved=0
> 
> next.  Please submit any comments you have on these to the TSVWG list in that
> timeframe.
>

---

From: Greg White <[email-masked]>
Date: Thu, 26 Aug 2021 18:21:45 +0000

Wes, 

As a co-author of two of the drafts and having reviewed all three, I believe they are mature and I support moving them to RFC.

Support for the L4S architecture and dual-queue-coupled AQM utilizing the L4S identifier has been adopted in the DOCSIS 3.1 and DOCSIS 4.0 specifications, and there are multiple products available that support it. 

Best Regards,
Greg

On 7/29/21, 10:18 AM, "tsvwg on behalf of Wesley Eddy" <[email-masked] on behalf of [email-masked]> wrote:

    - DualQ:

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Thu, 26 Aug 2021 14:08:17 -0700 (PDT)

Wes, tsvwg list,

	I can not support the advancement of these drafts to
the IESG.  A much longer and combined statement shall be posted
by Jonathan Morton representing SOME, but not ALL of my objections
to this work advancing.  Further many technical details are contained
in the objection by Pete Heist.  I support all of these objections
contained in the respective posts of Jonathan and Pete.

	As major hurdles for me personally in the L4S design is
the conflict with an existing Proposed Standards document, RFC3168,
and the lack of a containment mechanism until which time that it
is CERTAIN that safety can be maintained.

	I do support the idea of reduced latency, not just by the
means put forth by L4S.  The aim was too high (well, numberically
low, at 1mS) and too many pointy prickly problems (corner cases)
are left still unresolved.

Regards,
Rod Grimes

> to do next.? Please submit any comments you have on these to the TSVWG 
> list in that timeframe.
> 

-- 
Rod Grimes                                                 [email-masked]

---

From: Karthik Sundaresan <[email-masked]>
Date: Thu, 26 Aug 2021 21:18:47 +0000

Hi Wes, 

Yes ,  I support advancing these L4S drafts to RFC status.  
The DOCSIS 3.1 & 4.0 specifications already refer to these drafts. We have already seen devices which support the L4S architecture and the dual-queue coupled AQMs and have also certified these products , and these will be out in the field with a software update to existing devices.  

Thanks
Karthik
--

    - DualQ:

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 27 Aug 2021 01:20:00 +0300

> On 29 Jul, 2021, at 7:18 pm, Wesley Eddy <[email-masked]> wrote:
> 

I have read through these drafts more times than I really wanted to, and frankly can't shake the feeling of listening to a mistuned piano whilst doing so.  I hereby object to the advancement of these drafts to the next stage of publication, as they are clearly in no state to do so.

My detailed comments, which among other things provide a technical justification for the bullet points read out by David Black during the recent meeting, are available here:

	https://sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf

- Jonathan Morton

---

From: Stuart Cheshire <[email-masked]>
Date: Fri, 27 Aug 2021 03:48:05 -0700

> - ECN ID: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/

These are exceptionally high quality documents. The insights are profound, and the writing is excellent. I have a few minor observations, and one more major one.

=====

<https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/>

I find use of reference-as-noun makes some sentences hard to parse and understand. It assumes every reader has every RFC memorized and unconsciously treats every RFC number as a noun. To see the text as a normal reader would, assume that instead of [RFC1234] all references are written numerically, like [1]. Or, even better, as a little superscript number, as they were back in the days when we still had typesetting.

For example, what does this mean, on page 8: “To enable L4S, the standards track¹ has had to be updated...”

--

Page 7 defines Reno-friendly:

   Reno-friendly:  The subset of Classic traffic that is friendly to the
      standard Reno congestion control defined for TCP in [RFC5681].

The term “friendly” is defined in terms of “friendly”, which seems a bit circular. What does “friendly” mean precisely?

And what is “Classic traffic”? Traffic that uses Classic Congestion Control?

   Classic Congestion Control:  A congestion control behaviour that can
      co-exist with standard Reno [RFC5681] without causing
      significantly negative impact on its flow rate [RFC5033].

If Classic traffic does not negatively impact Reno, and Reno-friendly traffic is friendly to Reno, what’s the difference? Why is Reno-friendly a subset of Classic traffic? What is an example of traffic that is Classic (does not negatively impact Reno) but is not Reno-friendly?

--

Page 17 mentions self-inflicted queuing delay:

      1.  It might seem that self-inflicted queuing delay within a per-
          flow queue should not be counted, because if the delay wasn't
          in the network it would just shift to the sender.  However,
          modern adaptive applications, e.g. HTTP/2 [RFC7540] or some
          interactive media applications (see Section 6.1), can keep low
          latency objects at the front of their local send queue by
          shuffling priorities of other objects dependent on the
          progress of other transfers.  They cannot shuffle objects once
          they have released them into the network.

It might be helpful here to mention the TCP_NOTSENT_LOWAT socket option.

If you want a reference, something like this might do:

<https://blog.cloudflare.com/http-2-prioritization-with-nginx/>

--

Page 22:

   For any one L4S flow to work, it requires 3 parts to have been
   deployed.

I *think* the three parts you mean are: client, bottleneck queue, and server.

But I’m not sure. It would be good for this text to specify what the 3 parts are.

--

Page 27:

   the L4S service relies on self-constraint

I think you meant:

   the L4S service relies on self-restraint

=====

<https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/>

Page 6:

   Even with a perfectly tuned AQM, the additional
   queuing delay will be of the same order as the underlying speed-of-
   light delay across the network.

I know what you’re saying here, but some readers might miss the significance. How about adding some words at the end:

   Even with a perfectly tuned AQM, the additional
   queuing delay will be of the same order as the underlying speed-of-
   light delay across the network, thereby doubling the total round-trip time.

--

Page 16:

   {ToDo: Not-ECT / ECT(0) ?}.

Seems like this bit is not quite ready for IETF Last Call.

--

Page 20, 'Safe' Unresponsive Traffic

   The above section requires unresponsive traffic to be 'safe' to mix
   with L4S traffic.  Ideally this means that the sender never sends any
   sequence of packets at a rate that exceeds the available capacity of
   the bottleneck link.  However, typically an unresponsive transport
   does not even know the bottleneck capacity of the path, let alone its
   available capacity.  Nonetheless, an application can be considered
   safe enough if it paces packets out (not necessarily completely
   regularly) such that its maximum instantaneous rate from packet to
   packet stays well below a typical broadband access rate.

   This is a vague but useful definition, because many low latency
   applications of interest, such as DNS, voice, game sync packets, RPC,
   ACKs, keep-alives, could match this description.

This makes me very nervous.

I agree that it is vague, but I don’t think it’s useful.

A bunch of people doing voice calls over a shared slow Internet connection (think beach café on some remote island) could easily overwhelm the available capacity, and their clients should react by selecting a lower-rate voice codec, not by driving the network into massive packet loss so that nothing works. Conversely, if spare capacity is available, the clients should react by selecting a higher-rate voice codec to make use of that extra capacity and give better voice quality. I don’t see any good reason why voice clients, or video games, should not adapt to give the best user experience possible on a given network capacity. And that adaptation includes adapting down as well as adapting up. A voice call that adapts down (and works) gives a better user experience than a voice call that drives the network into congestion collapse (and completely fails).

--

Page 20:

This seems like a disincentive for client applications to use the L4S identifier, because they might actually get lower latency and lower loss by simply sticking with ECT(0). (More on this below.)

I didn’t find any mention of what these documents assume a Classic (e.g., FQ-CoDel) bottleneck will do with an ECT(1) packet. I think it is assumed that an FQ-CoDel bottleneck will treat ECT(1) exactly the same as ECT(0), but the documents don’t say so.

=====

Final Observation:

With our work slaying the Bufferbloat beast almost complete now, I started thinking about what advice we should be giving to developers at next year’s Apple Worldwide Developer Conference (WWDC). Should their apps use ECT(0) for lower latency and lower loss? FQ-CoDel with ECN is deployed in some places, and increasing. With many home gateways today you just have to turn it on. Or should their apps use ECT(1) for even lower latency? L4S is not widely deployed, so ECT(1) gives zero immediate benefit to the developer using it, but possibly it might be the better long-term direction. As a developer I know what I’d pick: “I’ll use ECT(0) now for the immediate benefit. Once L4S overtakes FQ-CoDel let me know and I may switch.” But if all the app developers are using ECT(0), L4S will never overtake FQ-CoDel, because there will be no reason for anyone to ever deploy L4S with ECT(1) if all the apps are using ECT(0). If all the apps are using ECT(0) to get FQ-CoDel behaviour, then home gateway vendors will implement FQ-CoDel with ECT(0) because that’s what shows a measurable customer benefit immediately.

So, let’s think about the pros and cons of an app developer choosing ECT(0) with Classic Congestion Control vs. ECT(1) with Scalable Congestion Control.

Marking packets ECT(0) instead of “Not ECT” will get the benefit of lower loss and therefore lower retransmission delays, on both Classic FQ-CoDel bottlenecks and newer L4S bottlenecks. These congestion signals however are coarse, so the sender will need to make large sawtooth rate swings in response to a CE mark, so it will never get truly low delay and high throughput consistently. It will either over-fill the queue and get higher delay, or under-fill the queue and get lower throughput.

Marking packets ECT(1) will get the same benefit of lower loss and therefore lower retransmission delays. In addition, if the bottleneck is actually L4S, the sender also gets the benefit of fine congestion signals, allowing more moderate rate adjustments, keeping the queue consistently short while keeping throughput high.

However, if the bottleneck is actually FQ-CoDel, and a single CE mark is supposed to produce a large rate reduction, but the sender makes only a modest rate reduction, then the sender will continue to grow the bottleneck queue. Over time the FQ-CoDel algorithm will slowly get more persistent with its CE marks until eventually the sender makes a sufficient rate correction. In the meantime the bottleneck queue will have grown much worse than it would have done with a classic sender that reduced its congestion window more aggressively at the first CE mark.

So, an L4S ECT(1) sender hitting a Classic FQ-CoDel bottleneck actually gets worse queueing delay than a Classic ECT(0) sender. This is a serious disincentive to client adoption of ECT(1) with Scalable Congestion Control.

I don’t think heuristics to detect the bottleneck queue type can react fast enough to help.

The bottleneck queue for a path can change rapidly.

Consider a customer with 100Mb/s ISP service and a Wi-Fi Access Point in their home.

When the customer is standing close to their Wi-Fi Access Point their Wi-Fi rate may be 300Mb/s, so their downstream bottleneck link is the ISP service coming in to their house. Suppose they have an enlightened ISP that already implements L4S Dual Queue at the downstream link. The customer’s L4S-dependent client software works great.

When the customer moves a little further from their Wi-Fi Access Point their Wi-Fi rate may drop to 50Mb/s, so their downstream bottleneck link is now their Wi-Fi Access Point. Suppose their Wi-Fi Access Point implements Classic FQ-CoDel but not L4S. The customer’s L4S-dependent client software now works even worse than simple loss-based congestion control. Instead of making a drastic reduction in response to a CE mark, it makes a small reduction and the queue (and delay) keep growing.

As this user walks around their house (or as other people walk around them) their downstream bottleneck link is going to flip flop between their ISP’s L4S downstream queue, and their Wi-Fi Access Point’s Classic FQ-CoDel queue.

This suggests that we need a way for client software to use L4S in a way that takes advantage of an L4S bottleneck when available, but does no worse than Classic ECN when the bottleneck happens to be Classic ECN instead of L4S.

Conclusion:

Because of the reasons explained eloquently in the documents, queues are going to be shared. If traffic wants to be in an ultra-low latency shared queue, it needs to signal its agreement to play nicely with others and not be bursty (with associated penalty if it is caught misbehaving). This means we need three input code points:

(a) I am not ECT (please drop my packets after the queue gets fairly big)
(b) I understand Classic ECN (please mark after the queue gets fairly big)
(c) I understand Low-Latency ECN (please mark my packets when the queue starts to grow even slightly, and I promise to reduce my rate promptly)

On the output side what signals do we need?

Input (a) does not have a congestion signal output (a dropped packet has no header fields)
Input (b) can have output (d) below
Input (c) can have output (d) or (e) below

(d) Do drastic congestion window reduction
(e) Do moderate congestion window reduction

This means we need to encode five values. That doesn’t fit in a two-bit field. One of the values has to go somewhere else. Maybe a DiffServ codepoint for input code point (c)?

I know this conclusion won’t make people happy. It doesn’t make me happy, but I don’t see any way around it.

We need a way for Scalable Congestion Control senders to tag their packets in a way that tells the network: “I understand L4S. If my bottleneck queue at this instant in time happens to be a dumb FIFO whose queue is full, then this packet will get dropped and I will do a drastic congestion window reduction in response. If my bottleneck queue at this instant in time happens to support Classic FQ-CoDel and it has determined that it has a standing queue then this packet will be marked (d) and I will do a drastic congestion window reduction in response. If my bottleneck queue at this instant in time happens to support L4S and it has a moderate queue then this packet will be marked (e) and I will do a moderate congestion window reduction in response.”

Stuart Cheshire

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Fri, 27 Aug 2021 23:58:39 +0200

> to do next.  Please submit any comments you have on these to the TSVWG 

I would love to see the stated objective of the L4S architecture
realised. However, given the failure modes that have already been
demonstrated, I am not convinced that the technology can deliver on that
promise. Furthermore, I believe that re-assigning ECT(1) as an
identifier for L4S traffic will end up being a de facto permanent
assignment if this experiment goes ahead; resulting in the worst case in
either an easily exploitable "reverse evil bit", or (more realistically)
bleaching of the bit for non-approved traffic, thus creating a "pay to
play" fast path instead.

Neither of these are desirable outcomes, but the risk of ending up with
them is non-negligible; for this reason, I do not support advancing the
L4S drafts in their current state. Rather, if the experiment is to go
ahead, it should be conducted using an alternative identifier (such as a
diffserv code point).

-Toke

---

From: "Jerome Henry (jerhenry)" <[email-masked]>
Date: Mon, 30 Aug 2021 16:00:49 +0000

Cisco is also closely considering L4S and is supporting moving these documents to RFC.

Thank you

Jerome Henry

--
Jerome Henry
PRINCIPAL ENGINEER
Office of the CTO
[email-masked]<mailto:[email-masked]>
Phone: +1 919 392 2503
Cisco Systems Limited
7100-8 Kit Creek Road PO Box 14987
RESEARCH TRIANGLE PARK
NORTH CAROLINA
27709-4987
US
Cisco.com<http://www.cisco.com/>

From: tsvwg <[email-masked]> on behalf of "Ranganathan, Ram" <[email-masked]>
Date: Tuesday, August 24, 2021 at 2:20 PM
To: "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

Thank you
Ram

Ram Ranganathan

COMMSCOPE
now meets next

Hi all,

Thanks,
  - Dave

David Pullen

-Glenn Deen