Subject: Re: [tsvwg] start of WGLC on L4S drafts

From: "[email-masked]" <[email-masked]>
Date: Thu, 2 Sep 2021 18:26:20 +0000 (UTC)

Hi Gorry, tsvwg;

See inline [AB]

On Thursday, August 12, 2021, 11:23:49 AM GMT+1, Gorry Fairhurst <[email-masked]> wrote: 

---
1b) Issue: I’d also quibble with: “so that all of a user's applications 
can shift to it when their stack is updated.“ - again, is the word “all” 
really necessary or correct here?
---

[AB]
Not  answering your point, but related:

L4s-arch has at the very start the following: “The L4S architecture is intended to enable   _all_ Internet applications to transition away from congestion control algorithms that cause queuing delay, to a new class of  congestion controls that induce very little queuing, aided by  explicit congestion signaling from the network. “

 There are two quite different end states here:

* An internet in which there are, end-to-end, two classes of traffic on an ongoing basis.  

* An internet in which nearly all traffic has transitioned to L4S, remaining non-L4S traffic being supported on a backwards-compatibility basis

As I understand it, if L4S is deployed, either of these could be possible end states. At present it is not clear that all use cases could migrate (eg, long RTT traffic)

It seems like it would be a good idea for the draft to make clear:
* That either of these end states are possible
* What the formal position of the WG would be if the drafts are accepted, IE what decision has been made (or left for later) regarding the desired end state.

regards,

Alex

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 7 Sep 2021 17:46:34 +0100

Ermin, Gorry, Mirja, Phil, Praveen, Neal, Jake, Pete,

Thank you for your review comments, mostly on l4s-arch (all of you), but 
also l4s-id (Jake) and dualq and l4s-id (Gorry, Ermin, Mirja & Pete).

If anyone else not listed above posted comments during the WGLC 
specifically about the text of the drafts, pls shout, as I may have 
overlooked your email.

I hope I've already satisfactorily addressed Mirja's and Praveen's 
comments on the list. As editor of the 3 drafts, I will set to dealing 
with all the others and posting revs to all of them over the coming days 
and perhaps weeks.

I'm aware that many of Pete's comments and some of Jake's wanted more 
than just text changes. Our differences on one or two of those concerns 
are well-rehearsed, but I and others are working on improving the 
congestion controls to address performance in the less common cases, 
which (as with all CCs) will continue to be work in progress for the 
transport community and particularly as the L4S experiment moves to the 
public Internet.

Regards

Bob

On 17/08/2021 16:04, Bob Briscoe wrote:
> Ermin, Gorry, Mirja,
>
> Just to let you know, I'm collecting all your review comments, but I 
> won't respond until the call is over.
>
> Cheers
>
>
> Bob
>
> On 16/08/2021 08:59, Ermin Sakic wrote:
>> Hi Wesley, all,
>>
>> L4S proposals are an important step towards enabling low-latency 
>> transmissions for remote control, AR/VR, and online gaming applications.
>> I support progressing the drafts to RFC status.
>>
>> I have attached a few comments below after re-reading the latest 
>> versions of the three drafts.
>> My apologies if some of the comments came up in discussion earlier.
>>
>> Best wishes,
>> Ermin Sakic, NVIDIA
>>
>> - draft-ietf-tsvwg-l4s-arch-10:
>>          - Figure 1: Is there a specific reason why the “mark/drop” 
>> action is not denoted for L4S queue compared to “mark”-only? Section 
>> 5.1 of draft-ietf-tsvwg-ecn-l4s-id-19 states that “…L4S marking 
>> treatment MUST begin applying drop to L4S traffic until the overload 
>> episode has subsided…”.
>>          - Section 8.2: “…if it is leads…” -> “…if it leads…”
>> - draft-ietf-tsvwg-ecn-l4s-id-19:
>>          - Section 1 Introduction: “…responses to them result..” -> 
>> “…responses to markings result…”
>>          - Section 1.1: “The explanation is summarised without the 
>> maths in the L4S architecture document” -> I suggest referencing to 
>> the specific section number of draft-ietf-tsvwg-l4s-arch here.
>>          - Section 7.1: “TCP. QUIC…” -> “TCP, QUIC…”
>>          - Appendix B: “tranport-layer” -> “transport-layer”
>> - draft-ietf-tsvwg-aqm-dualq-coupled-16:
>>          - Figure 1: As mentioned above, I’d expect to see an 
>> optional drop as the outcome of forwarding to the L traffic queue in 
>> addition to the “ECN marking” action.
>>
>> -----Original Message-----
>> From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
>> Sent: Thursday, 29 July 2021 18:18
>> To: [email-masked]
>> Subject: [tsvwg] start of WGLC on L4S drafts
>>
>> External email: Use caution opening links or attachments
>>
>>
>> This message is starting a combined working group last call on 3 of 
>> the L4S drafts:
>>
>> - Architecture: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-arch%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a7V1lmfJi%2Fbs2H2mZIr3QhveYsPj3QzPguBFDWsy%2BC0%3D&amp;reserved=0
>>
>> - DualQ:
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-coupled%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=PwO0BFTNKv2c4ZbGoQZEvxfLg7NIG9mw8oWtNqdwsjs%3D&amp;reserved=0 
>>
>>
>> - ECN ID: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-id%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=QpBxzuleCeXpxzoBR4oEJWANLk7v1Vvo%2Ft4H8d3kZz0%3D&amp;reserved=0
>>
>> The WGLC will last through 4 weeks from today, and then we'll see 
>> what to do next.  Please submit any comments you have on these to the 
>> TSVWG list in that timeframe.
>>
>> The chairs are considering a possible virtual interim following the 
>> close in order to work through feedback received.
>>
>> The work on the L4S operational guidance draft is continuing in 
>> parallel, but that draft is not being last called yet.
>>
>>
>> ----------------------------------------------------------------------------------- 
>>
>> NVIDIA GmbH
>> Wuerselen
>> Amtsgericht Aachen
>> HRB 8361
>> Managing Director: Karen Theresa Burns
>>
>> ----------------------------------------------------------------------------------- 
>>
>> This email message is for the sole use of the intended recipient(s) 
>> and may contain
>> confidential information.  Any unauthorized review, use, disclosure 
>> or distribution
>> is prohibited.  If you are not the intended recipient, please contact 
>> the sender by
>> reply email and destroy all copies of the original message.
>> ----------------------------------------------------------------------------------- 
>>
>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 7 Sep 2021 18:48:30 +0200

Thanks for your update and work, Bob! Yes, I believe my comments were addressed on list.

> On 7. Sep 2021, at 18:46, Bob Briscoe <[email-masked]> wrote:
> 
> Thank you for your review comments, mostly on l4s-arch (all of you), but also l4s-id (Jake) and dualq and l4s-id (Gorry, Ermin, Mirja & Pete).
> 
> If anyone else not listed above posted comments during the WGLC specifically about the text of the drafts, pls shout, as I may have overlooked your email.
> 
> I hope I've already satisfactorily addressed Mirja's and Praveen's comments on the list. As editor of the 3 drafts, I will set to dealing with all the others and posting revs to all of them over the coming days and perhaps weeks.
> 
> I'm aware that many of Pete's comments and some of Jake's wanted more than just text changes. Our differences on one or two of those concerns are well-rehearsed, but I and others are working on improving the congestion controls to address performance in the less common cases, which (as with all CCs) will continue to be work in progress for the transport community and particularly as the L4S experiment moves to the public Internet.
> 
> 
> Regards
> 
> 
> Bob
> 
>> Just to let you know, I'm collecting all your review comments, but I won't respond until the call is over.
>> 
>> Cheers
>> 
>> 
>> Bob
>> 
>>> L4S proposals are an important step towards enabling low-latency transmissions for remote control, AR/VR, and online gaming applications.
>>> I have attached a few comments below after re-reading the latest versions of the three drafts.
>>>          - Figure 1: Is there a specific reason why the “mark/drop” action is not denoted for L4S queue compared to “mark”-only? Section 5.1 of draft-ietf-tsvwg-ecn-l4s-id-19 states that “…L4S marking treatment MUST begin applying drop to L4S traffic until the overload episode has subsided…”.
>>>          - Section 1 Introduction: “…responses to them result..” -> “…responses to markings result…”
>>>          - Section 1.1: “The explanation is summarised without the maths in the L4S architecture document” -> I suggest referencing to the specific section number of draft-ietf-tsvwg-l4s-arch here.
>>>          - Figure 1: As mentioned above, I’d expect to see an optional drop as the outcome of forwarding to the L traffic queue in addition to the “ECN marking” action.
>>> 
>>> This message is starting a combined working group last call on 3 of the L4S drafts:
>>> 
>>> - Architecture: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-arch%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a7V1lmfJi%2Fbs2H2mZIr3QhveYsPj3QzPguBFDWsy%2BC0%3D&amp;reserved=0
>>> 
>>> - DualQ:
>>> - ECN ID: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-id%2F&amp;data=04%7C01%7Cesakic%40nvidia.com%7C911e34a8568a4c43d67c08d952ac82f5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637631724620979753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=QpBxzuleCeXpxzoBR4oEJWANLk7v1Vvo%2Ft4H8d3kZz0%3D&amp;reserved=0
>>> 
>>> The WGLC will last through 4 weeks from today, and then we'll see what to do next.  Please submit any comments you have on these to the TSVWG list in that timeframe.
>>> 
>>> The chairs are considering a possible virtual interim following the close in order to work through feedback received.
>>> 
>>> The work on the L4S operational guidance draft is continuing in parallel, but that draft is not being last called yet.
>>> 
>>> 
>>> This email message is for the sole use of the intended recipient(s) and may contain
>>> confidential information.  Any unauthorized review, use, disclosure or distribution
>>> is prohibited.  If you are not the intended recipient, please contact the sender by

---

From: "Jerome Henry (jerhenry)" <[email-masked]>
Date: Tue, 14 Sep 2021 13:11:52 +0000

(clarification: when I write ‘Cisco’, I mean my group, the Enterprise Network Wireless CTO group.)
Best
Jerome  Henry

From: "Jerome Henry (jerhenry)" <[email-masked]>
Date: Monday, August 30, 2021 at 12:00 PM
To: "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

Cisco is also closely considering L4S and is supporting moving these documents to RFC.

Thank you

Jerome Henry

--
Jerome Henry
PRINCIPAL ENGINEER
Office of the Wireless CTO
[email-masked]<mailto:[email-masked]>
Phone: +1 919 392 2503
Cisco Systems Limited
7100-8 Kit Creek Road PO Box 14987
RESEARCH TRIANGLE PARK
NORTH CAROLINA
27709-4987
US
Cisco.com<http://www.cisco.com/>

From: tsvwg <[email-masked]> on behalf of "Ranganathan, Ram" <[email-masked]>
Date: Tuesday, August 24, 2021 at 2:20 PM
To: "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

We (CommScope) are actively working on L4S support in our labs/test setups (DOCSIS based network) and see the value proposition brought by the protocol.

We support moving these working group drafts to RFC.

Thank you
Ram

Ram Ranganathan
Director, Systems Engineering
CTO Office – Broadband Networks

COMMSCOPE
now meets next

90 Matheson Blvd. West - Suite 410
Mississauga ON
L5R 3R3 Canada
Phone: +1-905-568-7317

Learn how CommScope is shaping the networks of the future<https://www.commscope.com/>

Hi all,

Broadcom has implemented support for L4S in Cable Modem products and supports moving the drafts to RFC.

Thanks,
  - Dave

David Pullen
Distinguished Engineer  |  BVG, Cable Modems
Broadcom

On Thu, Aug 19, 2021 at 1:30 PM Glenn Deen <[email-masked]<mailto:[email-masked]>> wrote:
I support moving these drafts forward.

-Glenn Deen

On 29 Jul 2021, at 9:18, Wesley Eddy wrote:

> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/<https://secure-web.cisco.com/1GpperBysmvVF1MstLFVLUypBwExAYL1zrdiPtipsLf0U24CdIRmqhKIsMofesS5j0S8syRQ4TqFOAbbmgG44fDTTVN-gVcydKWCS4S8RU4be6OKju-wfHQ35-OjNWP9aY4oz5rj19AiKmlK_hx6CwiuYAVrXBFXdxW-ePPCGLdtWzOzdXzADdhDBzzMvYj1pJ1Kq-B5OAdDPy8CXfyHRsS_W0SsFuG7UGcz3Hqws_-tT3fxCA01qMLCHba2gLkDVlGtggKS6V1c3dl4t-bE2KwNZKgGpwsKocZQEamCZfZ8sdt6IyR56J63LjgbccBQe/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-l4s-arch%2F>
>
> - DualQ:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/<https://secure-web.cisco.com/1kdMUsAf9sIPDfIH5oTSzVq0eNug4uzrZ1UZFHWCF5DG4dWDqXPYP_Eabojjx7Ao5HYasDWQ4XOGdN7ZsGdKrFzS4YCXMTwjN6USOL_axtsWfMQeZ2-g1BlOsZoJR-Xmy1SjhSJAz4RF9oINVaxnskQdyYXvpiXmeZECFCfGwEknSb0EtJl4wHS1BtQ3dKyRZVmbhNL7NjhAD-AipVACh0fJT7Lf4wpvAxk1nJYJPpHQBdRfJ_3Tf5OFUSeT3h5M-deK_amUWBCd6zfWWVFW59F97zoEGSRmmJZRFxlPYAe4p_mUoBc5Dh9Ky7G-w6OCr/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-aqm-dualq-coupled%2F>
>
> - ECN ID:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/<https://secure-web.cisco.com/1y7pXMUUoAoov99jQqx4-4vy38esB-SngBCSYL8MadELskaknSCf5JLCCmKDuQf2UYCqRO-ebQl1DIhIuQ5ehi23m87Y0ky9uku-YSONsFrMncRBXXpGM6XEZDfxeI4VuMSpynhkpskS1OY2RmNfbJB35qibPHbtPodcXAsOJRL-1u3Ub_WrhToZIpJLylN1EwQSxLEzTFvWekJSlKgTP165aQLqu6sOde1pAjeFiDbbcFm9ruraL-lhYFvuKAWaXcNQCuxgFlpIc5v2oG6PS8C3D-MXMIO1B53xgPN2-kd8PVHzYI2gPLyulSk080Yuc/https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-ietf-tsvwg-ecn-l4s-id%2F>
>
> The WGLC will last through 4 weeks from today, and then we'll see what
> to do next.  Please submit any comments you have on these to the

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 20 Sep 2021 17:40:15 +0100

Alex, Gorry, [Sry, just discovered I didn't click end on this]

In "all of a user's applications can shift" the word "can" is important.
Similarly, in "The L4S architecture is intended to enable all Internet 
applications to transition" the word "enable" is important.

Neither statement was meant to imply that all applications would move, 
or should move, just that they would all be _able_ to.
These statements are primarily intended as counterpoints to the commonly 
held view that low latency comes from priority scheduling (e.g. via 
differentiated service) so low latency can only be provided for a subset 
of traffic, because if all packets requested higher priority, there 
would be no difference to any packet's latency.

I've made a note to myself to make it clear that this is not an 
intention that all applications will transition, rather it is an 
intention not to prevent any application from transitioning.

Bob

On 02/09/2021 19:26, [email-masked] wrote:
> Hi Gorry, tsvwg;
>
>
> See inline [AB]
>
>
>
> On Thursday, August 12, 2021, 11:23:49 AM GMT+1, Gorry Fairhurst<[email-masked]>  wrote:
>
> ---

---

From: "Gorry (erg)" <[email-masked]>
Date: Mon, 20 Sep 2021 18:56:07 +0100

Bob,

This makes sense to me he way you explain it - a little tweak here and there can perhaps avoid potential mis-reading 

Gorry

> On 20 Sep 2021, at 17:40, Bob Briscoe <[email-masked]> wrote:
> 
> Similarly, in "The L4S architecture is intended to enable all Internet applications to transition" the word "enable" is important.
> 
> Neither statement was meant to imply that all applications would move, or should move, just that they would all be _able_ to.
> These statements are primarily intended as counterpoints to the commonly held view that low latency comes from priority scheduling (e.g. via differentiated service) so low latency can only be provided for a subset of traffic, because if all packets requested higher priority, there would be no difference to any packet's latency.
> 
> I've made a note to myself to make it clear that this is not an intention that all applications will transition, rather it is an intention not to prevent any application from transitioning.
> 
> 
> Bob
>

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 21 Sep 2021 13:04:47 -0400

With permission from those involved, I'm relaying a few messages to the 
list from an offlist discussion that Bob had started in regard to Jake's 
review comments.

All involved agreed that their conversation should continue on the 
list.  I will be sending this initial message from Bob, below, and then 
subsequently Jake's reply, and then a short chain of exchanges between 
Mike Heard, Bob, and Jake.

On 8/28/2021 6:45 AM, Bob Briscoe wrote:
> Jake, Wes,
>
> I'm top-posting because I'd like to focus solely on the suggestion to 
> switch to the stds track in this email (but thank you, Jake, for all 
> the other review comments, which I'll respond to next week). Also, 
> Jake in particular, thank you for your continued willingness to think 
> deeply about this.
>
> I'm happy to discuss this on-list, but thought it better to start 
> off-list, which makes it easier to clear-up any initial misconceptions 
> without noise. I've included the co-chairs, co-authors (whom I have 
> checked for agreement with first), and Mike and Steve because they 
> agreed with Jake. Here goes...
>
> I think the outcome Jake is trying to achieve would be more likely by 
> continuing on the exp track while initiating work on a stds track RFC 
> fairly quickly - as soon as Internet deployment of the L4S experiment 
> gets going. I think Jake's arguments for stds track are no stronger 
> whether L4S moves to stds track before the drafts progress or soon after.
>
>   * Moving L4S to stds track is pretty orthogonal to addressing
>     coexistence with /existing/ RFC3168 equipment.
>       o Jake's argument for stds track centres around being able to
>         recommend that any existing marking queues should be changed.
>         That's not what a stds track update does; it signals
>         implementers to change future implementations; it doesn't
>         magically change the past.
>       o If there are interworking problems between a stds track update
>         and the previous installed base, they still have to be
>         addressed just as diligently  - no less, no more - as an
>         interworking problem between an experimental RFC and the stds
>         track.
>       o Certainly, if developers updated their existing RFC3168 code
>         for future deployments, that would lead some existing deployed
>         kit to be auto-updated.
>       o But sending a stronger signal to encourage developers to move
>         to L4S is not a /prerequisite/ for L4S experiments to start.
>         L4S can start on the experimental track, while work on a stds
>         track update can start at any time in parallel, in order to
>         send out that stronger signal.
>   * The question of moving to stds track is likely to divide the WG
>     further.
>       o In discussions with just a few colleagues, I'm already hearing
>         some oppose moving to stds track quite strongly (of course,
>         this could be established in an interim better than by
>         anecdote). In contrast, the WGLC has shown that a large number
>         of people are not against the current exp track process, and
>         impatient to get on with it (setting aside those who don't
>         want L4S on any track at all).
>       o I agree with Jake that assignment of ECT(1) would not be
>         reversible for some decades to come. But, that is not why the
>         EXP track is being used here (ecn-l4s-id says that); in this
>         case it signals the possibility of changes during deployment,
>         because testing cannot be fully trusted without an
>         Internet-wide experiment.
>       o Finally a nitty point. Use of the EXP track was devised
>         (primarily by David in the lead-up to RFC8311) after careful
>         consideration of the other process alternatives. It's possible
>         that robust detection of RFC3168 was in everyone's minds while
>         writing RFC8311. But I've just re-read it and I cannot see any
>         evidence of such assumptions. I doubt this was behind choosing
>         the EXP track. I suspect you (Jake) are seeing the past
>         through the eyes of the present.
>
>
> Cheers
>
>
> Bob

On 8/21/2021 4:11 PM, Holland, Jake wrote:
> Overall:
> The documents are in much better shape than the last time I reviewed
> them, thanks for all the improvements.
>
> I'm reviewing l4s-id and l4s-arch but didn't have time to get to
> dualq-aqm yet.  But I wanted to make sure these got posted, and
> might not have more time before the deadline.
>
> ----
>
> draft-ietf-tsvwg-ecn-l4s-id-19:
> There's a few major issues.  These really should be fixed before
> sending it to the IESG, IMO, but with these fixed I'd be happy to
> see it shipped.
>
> Major:
>
> 1. This should be proposed standard, not experimental.
>
> This assigns a new and incompatible meaning to an IP header codepoint
> in a way that can't be rolled back or kept isolated, with declared
> intent to roll it out widely.  This should be considered a proposed
> standard action and make an update to RFCs 3168 and 8311 that clearly
> states that any existing marking queues should be changed where
> possible to treat ECT(1) as either L4S or Not-ECT, even if the device
> is not participating in an L4S-related experiment.
>
> Such an update would align the specs with the stated expectations
> that monitoring requirements should be in place mainly as a legacy
> workaround, and would provide appropriate and potentially actionable
> standards track mitigations that are available to on-path devices
> implicated in any problems that are observed, whereas the specs as
> currently written have none.
>
> I understand that RFC 8311 was originally intended to cover this, but
> that was published back when the L4S proposal still intended to make
> the ECT(1)-marking compatible with classic marking via a robust
> detection scheme, and RFC 8311 embeds that assumption throughout.  Now
> that the proposal has changed to rolling this out without solving the
> coexistence problem, RFC 8311's provisions are no longer adequate to
> cover this IMO, and must be updated if we intend to have a self-
> consistent set of specs published.
>
> (Alternately, a separate standards-track doc could recommend changing
> ECT(1) handling to Not-ECT, but I really think there's no point
> separating them, and such a doc would still have to be a prerequisite
> for moving this forward IMO.  I suggest that the most expedient
> path as well as the better choice here is to make l4s-id standards
> track and to add the Not-ECT option for ECT(1) here as a reasonable
> problem mitigation strategy available to marking bottlenecks.)
>
> 2. This text from Section 4.3 should be strengthened to recommend
> detection of paths that might be classic-marking L4S traffic (shared
> queue or not):
>
>     o  In uncontrolled environments, monitoring MUST be implemented to
>        support detection of problems with an ECN-capable AQM at the path
>        bottleneck that appears not to support L4S and might be in a
>        shared queue.
>
> Even in fq, the standing queue that will be built by a scalable cc with
> classic marking is contrary to "1-2ms latency at 99th percentile" goal
> of L4S, even leaving aside the harm to colliding flows (though of course
> the harm to colliding flows, including from hash collisions in fq, is
> also another reason to make this change).
>
> 3. This text from Section 4.3 should be strengthened to require that
> sending L4S traffic in uncontrolled environments does not happen when
> classic marking of L4S traffic is detected for a shared queue, and at
> least recommend that it not happen even for fq:
>
>     o  In uncontrolled environments, monitoring MUST be implemented to
>        support detection of problems with an ECN-capable AQM at the path
>        bottleneck that appears not to support L4S and might be in a
>        shared queue.  Such monitoring SHOULD be applied to live traffic
>        that is using Scalable congestion control.  Alternatively,
>        monitoring need not be applied to live traffic, if monitoring has
>        been arranged to cover the paths that live traffic takes through
>        uncontrolled environments.
>
> The current text requires monitoring, but only gives a single SHOULD for
> live traffic, plus non-normatively permits one alternative.  This allows
> operators to monitor but not cut off (so this requirement as currently
> written would be satisfied by write-only logging for instance, with the
> SHOULD easily dismissable with an "implementation complexity" hand-wave
> while still following the spec).
>
> Suggested alternative, feel free to edit:
>
>     o  In uncontrolled environments, L4S traffic MUST NOT be sent without
>        monitoring to detect marking of L4S traffic by non-L4S bottlenecks.
>        This monitoring can for example be performed on live traffic, or
>        can rely on monitoring that covers the paths live traffic takes
>        through uncontrolled environments.  Where non-L4S bottlenecks are
>        observed marking L4S traffic, L4S sending MUST be disabled if the
>        bottleneck is a shared queue, and SHOULD be disabled if it is FQ.
>
> 4. Although l4s-arch claims that l4s-id satisfies the RFC 4774
> requirements, it's hard to tell whether it does so.  Specifically:
>
> 4.a. From section 7 of RFC 4774:
>     Specifications of alternate ECN semantics must clearly state how they
>     address the issues raised in this document, particularly the issues
>     discussed in Section 2.
>
> I don't see how issues 2 and 3 from section 2 are covered in l4s-id.
>  From section 4 of RFC 4774:
>     (2) How does the possible presence of old routers affect the
>         performance of the alternate-ECN connections?
>
>     (3) How does the possible presence of old routers affect the
>         coexistence of the alternate-ECN traffic with competing traffic
>         on the path?
>
>     When alternate semantics are defined for the ECN field, it is
>     necessary to ensure that there are no problems caused by old routers
>     along the path that don't understand the alternate ECN semantics.
>
> 4.b. Section 4 goes on to describe 3 options for how alternate ECN
> semantics should be treated.  I don't see a claim in the L4S docs
> specifying which of the 3 options the L4S spec for alternate ECN semantics
> matches, but it implicitly appears to be trying to say it's option 3
> (unsafe) and asserting that the detection and adaptive response satisfies
> the requirement for isolation on option 3, I think?
>
> Maybe there are sections in l4s-id that intend to cover these points,
> and there just needs to be text listing what they are, but I don't think
> the link is obvious enough to satisfy the "clearly state" requirement
> from RFC 4774.  So it would be very helpful to add a list of references
> to sections that are intended to address these RFC 4774 requirements.
>
>
> Nits:
> - the list in section 7.1 has a weird formatting problem for the sub-
>    list:
>
>     o  Did use of L4S over the Internet result in improvements to the
>        following metrics:
>
>     o
>
>        *  queue delay (mean and 99th percentile) under various loads;
>
> ----
>
> draft-ietf-tsvwg-l4s-arch-10:
> Summary: Almost ready
>
> +1 to Gorry's comments here, especially regarding the use of "all traffic":
> https://mailarchive.ietf.org/arch/msg/tsvwg/vMMsQpXs65lk1E7NpV5RlmpyqdI/
>
> Minor:
> 1. l4s-arch section 1:
>     It has been demonstrated that, once access network bit rates reach
>     levels now common in the developed world, increasing capacity offers
>     diminishing returns if latency (delay) is not addressed.
> - This needs a reference.
>
> Editorial:
> 1. l4s-arch 4.2 section a:
> - This is a confusing wall of text.  I think it would be better to
>    give a much briefer summary here with a reference.  Exposition
>    like "the obvious part" and "the less obvious part" are a minus
>    here--I don't think the obviousness claims made here generalize
>    well.
>
> 2. All the uses of underlining for emphasis are a minus.  The
>    places where it seems necessary or useful are a good hint that
>    the text on its own is not adequately capturing the intended
>    meaning.
>    Leaning on this kind of toned emphasis makes assumptions about
>    connotations that don't hold very well even for native English
>    speakers and break down entirely for non-native speakers, so they
>    are generally out of place in a technical document that will need
>    to be correctly interpreted by an international audience with many
>    non-native speakers, IMO.
>
> ----
>
> I'm going to be too pressed for time to do a more detailed review,
> but I wanted to get the above comments in.
>
> As a final aside, I'd like to see this happen.  The only goal I'm
> pursuing at this point wrt this work is avoiding preventable harm
> from pushing this out in a way that's likely to cause confusion
> when and if problems are encountered in production.  (In particular,
> I have given up on efforts to improve the signaling design, since
> the authors have rejected all such suggestions and this work needs
> to get moved out of the wg one way or another.)
>
> Best,
> Jake
>
>
> On 07-29, 9:18 AM, "Wesley Eddy" <[email-masked]> wrote:
>
> - Architecture: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>
> - DualQ:
> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
>
> - ECN ID: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>
> to do next.  Please submit any comments you have on these to the TSVWG
> list in that timeframe.
>

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 21 Sep 2021 13:07:46 -0400

Below are Jake's thoughts on Bob's response that I previously sent.

On 8/28/2021 2:08 PM, Holland, Jake wrote:
> Hi Bob,
>
> From: Bob Briscoe <[email-masked]>
>> • Moving L4S to stds track is pretty orthogonal to addressing coexistence with /existing/ RFC3168 equipment.
>> o Jake's argument for stds track centres around being able to recommend that any existing marking queues should be changed. That's not what a stds track update does; it signals implementers to change future implementations; it doesn't magically change the past.
> Obviously we can't magically change the past, and changing a spec doesn't
> change what's out there today.
>
> However, a change to a future implementation is a prerequisite for
> updating existing deployments.  Providing a clear signal to implementers
> is a critical factor in reducing the time gap between the onset of harm
> to bystanders and the deployment of updates to fix it.
>
>> o If there are interworking problems between a stds track update and the previous installed base, they still have to be addressed just as diligently  - no less, no more - as an interworking problem between an experimental RFC and the stds track.
> The interoperability problem is of course between congestion controllers
> using L4S and the classic queues.
>
> In order to enable L4S senders as a safe experiment, we should ensure
> that all standards-compliant networks will be safe for that traffic.
>
> Due to the change in semantics and the uncertainty that all devices
> have been upgraded, of course the monitoring for unsafe classic queues
> would still be necessary, however the experiment should not move
> forward if the current standards do not permit its safe operation, and
> operators of deployed devices should have a standards-track document
> to point to as an argument for updating to something safe for this
> traffic if we hope to avoid hacks like bleaching as a workaround when
> they see problems.
>
>> o Certainly, if developers updated their existing RFC3168 code for future deployments, that would lead some existing deployed kit to be auto-updated.
> Yes, this would of course reduce the harm as L4S experiments start.
>
>> o But sending a stronger signal to encourage developers to move to L4S is not a /prerequisite/ for L4S experiments to start. L4S can start on the experimental track, while work on a stds track update can start at any time in parallel, in order to send out that stronger signal.
> I disagree.  The current proposal is to publish documents that
> encourage violating advice from the suite of standards-track documents
> and expect operators to do the right thing anyway.  We should ensure
> the documents state the right thing to do before publishing, which at
> least marginally improves the chances of a good outcome.
>
> Even if only the auto-updating devices get an update, that would
> still be a big improvement.
>
> But the safety story for L4S (such as it is) stands on the claim
> that there's very few marking shared queues deployed, so the guidance
> should be very clear that deploying them *now* would be a bad idea,
> else there's no reason that operators shouldn't respond to the
> increasing focus on latency by turning on the widely available shared
> queue marking features in their existing routers that have previously
> not been enabled (as e.g. section 4 of RFC 7567 advises).
>
> Preventing this development from happening is another key factor in
> the suggestion to make this std.
>
>> • The question of moving to stds track is likely to divide the WG further.
> This seems relatively unlikely, as it seems we're already almost
> maximally divided.
>
>> o In discussions with just a few colleagues, I'm already hearing some oppose moving to stds track quite strongly (of course, this could be established in an interim better than by anecdote). In contrast, the WGLC has shown that a large number of people are not against the current exp track process, and impatient to get on with it (setting aside those who don't want L4S on any track at all).
> Did they say why they'd oppose it strongly?  This seems
> (hopefully) better than de-adoption and obviously better than
> inviting the harm and increased chance of failure that would
> happen if operators were to follow the existing standards-track
> docs.
>
> Perhaps there are good reasons, and someone can convince me.  Or
> perhaps I'll be in the rough.  If so, of course the wg can and
> should override my objections after due consideration, but
> having come this far, I feel it would be irresponsible not to
> raise the suggestion, as I do believe there is a possibility of
> end user harm with an unknown high-end scope from just shipping
> this as-is that has been systematically and aggressively
> discounted on very thin evidence.  It makes me wonder how many
> of the supporters have given it due consideration and how many
> are just impatient.
>
> I'm not sure how to interpret the large number of people supporting
> the docs with no indication that they've studied the debate or
> considered the complex technical issues that were raised, but the
> moxie to cite their numbers as strong support in the face of some
> 15-20% of the commenters expressing strong and deep technical
> opposition is breathtaking.
>
>> o I agree with Jake that assignment of ECT(1) would not be reversible for some decades to come. But, that is not why the EXP track is being used here (ecn-l4s-id says that); in this case it signals the possibility of changes during deployment, because testing cannot be fully trusted without an Internet-wide experiment.
> I understand and agree that changes during deployment might be a
> good idea.
>
> But I thought the current docs provide enough flexibility on the
> semantics of the markings (in terms of the exact algorithms and
> tuning parameters that will be used) to accommodate the expected
> changes without spec updates?
>
> Anyway, the irreversibility coupled with the incompatibility in
> semantics is why I'm arguing this should be STD, regardless of the
> prior reasoning for EXP.
>
>> o Finally a nitty point. Use of the EXP track was devised (primarily by David in the lead-up to RFC8311) after careful consideration of the other process alternatives. It's possible that robust detection of RFC3168 was in everyone's minds while writing RFC8311. But I've just re-read it and I cannot see any evidence of such assumptions. I doubt this was behind choosing the EXP track. I suspect you (Jake) are seeing the past through the eyes of the present.
> And I suspect you're seeing the past through a glass of rosy
> expectations that there aren't actually any real problems here and
> that fairness isn't an issue we should be especially troubled with
> anyway, as you've argued frequently.
>
> Here’s some of the evidence I saw from RFC 8311 that led me to make
> the claim it was assumed that the L4S as experimental would solve the
> coexistence problem as a prerequisite for advancing as experimental:
>
> 1.
> Section 2.1 (“Effective Congestion Control is Required”):
> Just cites RFC 2914, which of course lists Fairness for best effort
> traffic as a key issue in sections 3.2 and 6.2.
>
> 2.
>  From section 2.2:
>     ECN experiments are expected to coexist with deployed ECN
>     functionality, with the responsibility for that coexistence falling
>     primarily upon designers of experimental changes to ECN.
>
> The current L4S specs place the responsibility primarily on the
> L4S senders via the vague monitoring requirements (rather than
> the protocol designers as indicated in the above quote).
>
> This seems to be unenforceable in practice primarily because the
> protocol design choice to use incompatible signaling has already
> been rolled out in hardware, so the notion that it's now subject
> to IETF change control (outside of maybe "yes or no") is a fiction.
>
> 3.
>  From section 2.2:
>     1.  Forwarding behavior as described in RFC 3168 remains the
>         preferred approach for routers that are not involved in ECN
>         experiments, in particular continuing to treat the ECT(0) and
>         ECT(1) codepoints as equivalent, as specified in Section 4.2
>         below.
>
> That directly advises standards-compliant deployments to mark ECT(1)
> in the classic way, which of course coupled with the “Deployed AQM
> algorithms SHOULD support ECN” from section 4 of RFC 7567 seems to
> rely on an expectation that they'll interoperate successfully.  (The
> alternate interpretation is that it's ok to increase harm to end users
> for anyone endeavoring to follow current standards and BCPs while L4S
> experiments are ongoing, which I assume was not the intent.)
>
> HTH.
>
> -Jake
>
>

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 21 Sep 2021 13:10:27 -0400

Finally, here is the remaining part of the offlist exchange that we 
wanted to bring to the list.

On 8/31/2021 12:39 PM, Holland, Jake wrote:
>
> Hi Bob,
>
> I agree there was a discrepancy in our phrasing about updating vs. 
> obsoleting RFC 3168, but I don’t really see how these differ.
>
> In either case, what I’m looking for is for the active latest 
> standards-track specs to say that ECT(0)->CE handling stays the same, 
> but ECT(1) is to be treated as NECT or L4S’s ECT(1), and all due 
> effort should be made to ensure deployed queues will never henceforth 
> treat ECT(1) as ECT(0), and that this is a necessary prerequisite to 
> approving a doc allowing for new incompatible handling as described in 
> L4S. (And as a suggestion, it seems likely the most expedient to 
> achieve this prerequisite in the same document at standards track, 
> though if the proponents would prefer to park the L4S documents to 
> wait for a new standards track spec that changes ECT(1) to NECT 
> instead of ECT(0) and permits experimental use, that would also be 
> acceptable to me.)
>
> If you’re saying that obsoleting RFC 3168 would do that job but 
> updating it would not, then I’ll adjust my position to say we should 
> obsolete RFC 3168, but as I said, I don’t see how these differ 
> substantively.  In either case, my position is that we should not 
> publish an experimental doc that is not compatible with the latest 
> standards track docs, in the way that L4S is not presently compatible 
> with RFC 3168 as updated by RFC 8311, and that this can be solved by 
> updating the current standards track docs, since we evidently can’t 
> achieve it more cleanly by change to the signaling to make it 
> compatible in any timely manner.
>
> HTH.
>
> Best,
>
> Jake
>
> *From: *Bob Briscoe <[email-masked]>
> *Date: *Tue,2021-08-31 at 8:37 AM
> *To: *"C. M. Heard" <[email-masked]>
> *Cc: *"Holland, Jake" <[email-masked]>, Wesley Eddy 
> <[email-masked]>, Greg White <[email-masked]>, "De 
> Schepper, Koen (Koen)" <[email-masked]>, Steven BLAKE 
> <[email-masked]>, "[email-masked]" <[email-masked]>
> *Subject: *Re: [OFFLIST for now] ECT(1) on stds track? (was: start of 
> WGLC on L4S drafts)
>
> Mike,
>
> The only misunderstanding I've seen is between whether we're talking 
> here about updating RFC3168 (and RFC8311), or obsoleting / deprecating 
> RFC3168 (and probably 8311). Jake used the former words, whereas I 
> notice you use the latter.
>
> If the IETF were to obsolete RFC3168, then I think it would have the 
> effect that you and Jake are hoping to achieve. But I don't see that 
> the IETF would have any reason to obsolete RFC3168. It is in use, 
> continuing to be deployed (we believe in FQ-CoDel but possibly 
> otherwise too) and providing some benefit.
>
> In contrast, updating RFC3168 to interwork more cleanly with RFC8311 
> experiments (specifically L4S) is something that I can imaging the 
> IETF doing. But then my point was that (IMO) it hardly resolves the 
> coexistence problem any more than an EXP RFC would.
>
>
> Bob
>
> On 28/08/2021 19:34, C. M. Heard wrote:
>
>     On Sat, Aug 28, 2021 at 3:45 AM Bob Briscoe wrote:
>
>         I think the outcome Jake is trying to achieve would be more
>         likely by continuing on the exp track while initiating work on
>         a stds track RFC fairly quickly - as soon as Internet
>         deployment of the L4S experiment gets going. I think Jake's
>         arguments for stds track are no stronger whether L4S moves to
>         stds track before the drafts progress or soon after.
>
>     **IF** RFC 3168 were declared obsolete first, **THEN** I would
>     agree that starting L4S as experimental and then moving to
>     standard later would be a reasonable course of action. And that
>     could be done, of course. But it seems to me that it would be
>     simpler to make the L4S document set standards track and mark it
>     as obsoleting RFC 3168 -- and probably RFC 8311 as well. I think
>     that Jake is correct when he argues that RFC 8311 was never
>     intended to permit incompatible and uncontained experimentation.
>     It would actually be good to hear from the editor of that document
>     on what his take is (wearing his editor hat, not his WG co-chair
>     hat, of course).
>
>     Thanks
>
>     Mike Heard
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!B4mHJRqHlL17qfu6aBRa-hojLOZ290aq_E7aTeqn-zlp25C3FDhIhd0u1pCYRo8$>

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 21 Sep 2021 19:47:17 +0000 (UTC)

Hi Bob, Gorry,

This would also clarify things from my point of view.

Thanks,
Alex

On Monday, September 20, 2021, 07:32:19 PM GMT+1, Gorry (erg) <[email-masked]> wrote: 

Bob,

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 21 Sep 2021 22:54:04 +0200

Hi Bob,

> On Sep 20, 2021, at 18:40, Bob Briscoe <[email-masked]> wrote:
> 

	[SM] Given that at the core of the L4S reference AQM there sits a (conditional) priority scheduler*, I fail to see how L4S' low latency does not come from priority scheduling? Also in L4S low latency can only be provided to a subset of traffic, namely flows that are well behaved and ECT(1)-marked (given the shared queue design, one bad apple in the L-queue can affect latency/throughput for all flows). I am not sure whether L4S really is contrary to the "commonly held view" here, sure the details differ a bit, but at its core it seems rather similar to traditional techniques for low latency.

Regards
	Sebastian

*) https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-19:
"In this case it would not be appropriate to call the queue an L4S
   queue, because it is shared by L4S and non-L4S traffic.  Instead it
   will be called the low latency or L queue.  The L queue then offers
   two different treatments:

   o  The L4S treatment, which is a combination of the L4S AQM treatment
      and a priority scheduling treatment;

   o  The low latency treatment, which is solely the priority scheduling
      treatment, without ECN-marking by the AQM."

https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-10:
"The tension between prioritizing L4S and coupling the
       marking from the Classic AQM results in approximate per-flow
       fairness.  To protect against unresponsive traffic in the L4S
       queue taking advantage of the prioritization and starving the
       Classic queue, it is advisable not to use strict priority, but
       instead to use a weighted scheduler (see Appendix A of [I-D.ietf-tsvwg-aqm-dualq-coupled])."

https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled
"Classic traffic needs to build a large queue to prevent under-
   utilization.  Therefore a separate queue is provided for L4S traffic,
   and it is scheduled with priority over the Classic queue.  Priority
   is conditional to prevent starvation of Classic traffic."

I ignore the yarn spun in those drafts, why this prioritization would only result in latency but not longer-time-scale throughput prioritization, because Pete's data show that for quite a number of realistic traffic patterns DualQ fails to arbitrate capacity equitably between the two queues.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 21 Sep 2021 22:58:34 +0100

Jake, (and thank you, Wes, for forwarding this to the list),

On 21/09/2021 18:10, Wesley Eddy wrote:
>> approving a doc allowing for new incompatible handling as described 
>> in L4S.
>>

[BB] Before I address the main procedural aspect of your email, can I 
ask that everyone please stops suggesting it is necessary for RFC3168 
equipment to treat ECT(1) as Not-ECT (or NECT as you call it). There are 
much better ways to proceed.

draft-ietf-tsvwg-l4sops proposes a number of potential changes to 
RFC3168 AQMs to make them support L4S, either fully or at least 
partially. Treating ECT(1) as Not-ECT comes way down the list as a 
non-preferred option. Indeed the editor (Greg White) has said that, in 
the next revision, he wants to draw a clearer line to explain that 
solutions that treat ECT(1) as Not-ECT are definitely not recommended or 
desirable.

The text about modifying FQ-CoDel (and CAKE) implementations of RFC3168 
is a particular case in point. The L4S architecture describes a really 
simple modification to Linux FQ-CoDel that fully supports both RFC3168 
ECN and L4S ECN. This is what l4sops mean when it says "update RFC3168 
FQ bottlenecks to be L4S-aware", but it is not clear what it means.

The l4sops draft then describes other less preferred options starting 
with treating ECT(1) as Not-ECT.  But it needs to be made much clearer 
(in l4sops and on this list now) what the preferred option is, and how 
easy it is to implement. It means the following, quoting from l4s-arch:

        ... For instance within each queue of an FQ-CoDel
        system, as well as a CoDel AQM, there is typically also ECN
        marking at an immediate (unsmoothed) shallow threshold to support
        use in data centres (see Sec.5.2.7 of [RFC8290  <https://datatracker.ietf.org/doc/html/rfc8290>]).  This can be
        modified so that the shallow threshold is solely applied to
        ECT(1) packets.  Then if there is a flow of non-ECN or ECT(0)
        packets in the per-flow-queue, the Classic AQM (e.g.  CoDel) is
        applied; while if there is a flow of ECT(1) packets in the queue,
        the shallower (typically sub-millisecond) threshold is applied.

That threshold is already in both the FQ-CoDel RFC and the Linux 
implementation. We have implemented and tested this minor change and can 
confirm that it fully supports both RFC3168 and L4S, so that hosts can 
switch from one to the other at any time without any dilemma.

>> (And as a suggestion, it seems likely the most expedient to achieve 
>> this prerequisite in the same document at standards track, though if 
>> the proponents would prefer to park the L4S documents to wait for a 
>> new standards track spec that changes ECT(1) to NECT instead of 
>> ECT(0) and permits experimental use, that would also be acceptable to 
>> me.)
>>

[BB] You are really highlighting that we need another status where an 
exp track document can put a stds track document "on hold" - to 
recommend new implementations and deployments are discouraged while 
experiments on how best to update the stds track proceed. IOW, to 
discourage or even stop future implementations, even though it cannot 
undo the past.

Actually, thinking about it, that is effectively what RFC8311 did. 
Someone approaching RFC3168 without having heard anything about L4S 
would discover it is updated by RFC8311. And, by reading RFC8311, they 
would find that it enables experiments with ECN and refers to L4S. And 
by reading L4S they would find how it is hoped to improve on RFC3168. 
That's pretty-close to putting RFC3168 "on hold" - probably as close as 
the IETF can get. Because to go any further (actively updating or 
obsoleting present and future RFC3168 implementations) would need 
evidence from deployment over the real public Internet that the 
contender is indeed a viable improvement.

You want the RFC series to be self-consistent, but what you are hoping 
for would just lead to a different sort of inconsistency within the stds 
track - an experiment with a fake "standards track" label on it. There 
is loads of content in the L4S docs about what needs to be tested in L4S 
experiments, none of which can be tested properly except over the real 
Internet, which in turn cannot be done without assigning a codepoint. 
The WG cannot pretend that there is no experiment to do by just writing 
"standards track" in the header block. There is no tidy consistent way 
through this.

As I said, the closest to what you want that would be feasible (IMO) 
would be to draft a stds track update to RFC3168 as soon as L4S 
deployment starts, which might possibly even be while the L4S drafts are 
in the RFC editor queue.

Cheers

Bob

>> here about updating RFC3168 (and RFC8311), or obsoleting / 
>> deprecating RFC3168 (and probably 8311). Jake used the former words, 
>> whereas I notice you use the latter.
>>
>>         likely by continuing on the exp track while initiating work
>>         on a stds track RFC fairly quickly - as soon as Internet
>>     It would actually be good to hear from the editor of that
>>     document on what his take is (wearing his editor hat, not his WG
>>     co-chair hat, of course).
>>
>>     Thanks
>>
>>     Mike Heard
>>
>>
>>
>> -- 

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: <[email-masked]>
Date: Wed, 22 Sep 2021 08:51:39 +0000

Wes, Bob, Jake,

thanks for moving this to the list.

As someone interested in having faster progress for a standard, but not participating in a detailed discussion, my motivation is:

a) I'm out for transport function realizing reliably low jitter and low loss to support applications expecting a short RTT, say 50 ms in the case of congestion today, maybe 20 ms in the future. Cost is to be regarded, I don't expect the data center being mounted on the BNG or EnodeB anytime soon (so keeping the buffers small is important).
b) I don't expect any kind of higher layer admission control - the simpler and the more universally deployable the architecture, the better.
c) I'm out for a universal transport, not for a QoS, SLA and bleaching solution.
d) Is seeing standardization and deployment to allow large scale experimentation in suitable environments.

Please correct me if I'm wrong: L4S is the best approach offered by IETF as of today (I'd appreciate a pointer to other work which has advanced in standards and implementation to the same state as L4S did up to now).

I don't regard L4S as perfect. L4S introduces several new ways of signaling impeding congestion. And it strongly reduces buffering. The latter results in a fairness issue, if RTT is a dominating factor determining the way transport streams share a resource. As an example, if the bottleneck operates a 100 ms(@linespeed) buffer and max RTT is 200 ms, and assuming RTT to dominate resource sharing, the remote flow will see more than 1/3 of the available resources. This changes significantly, if the bottleneck buffer is 5 ms or less.  This issue requires more research, but I think it shouldn't block experimentation, if these experiments can be limited to suitable environments.

I'd appreciate a mechanism which allows existing ECN implementations to deliver acceptable service, once L4S is deployed. But I'm not sure, whether the L4S queue and a queue used for buffer/drop based congestion indication need to be coupled for an experimental RFC. If de-coupling L4S and buffer/drop based transport allows or significantly weakens passible way forward with relaxed ECN backward compatibility requirements and a deployment option without entering a QoS, SLA and bleaching, I'd prefer that. Part of the latter to me is default forwarding, apart from the network edge aka bottleneck, which is to support L4S. It's clear to me, that DSCPs are one option I introduce by that.

I understand and welcome the motivation to couple both queues, and that there's a solution which shares resources roughly fair for a limited range of RTTs. The latter deserves respect. I don't object to this approach.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von Wesley Eddy
Gesendet: Dienstag, 21. September 2021 19:08
An: [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

On 8/28/2021 2:08 PM, Holland, Jake wrote:
> Obviously we can't magically change the past, and changing a spec 
> doesn't change what's out there today.
>
> updating existing deployments.  Providing a clear signal to 
> implementers is a critical factor in reducing the time gap between the 
> onset of harm to bystanders and the deployment of updates to fix it.
>
> The interoperability problem is of course between congestion 
> controllers using L4S and the classic queues.
>
> Even if only the auto-updating devices get an update, that would still 
> be a big improvement.
>
> But the safety story for L4S (such as it is) stands on the claim that 
> there's very few marking shared queues deployed, so the guidance 
> (hopefully) better than de-adoption and obviously better than inviting 
> the harm and increased chance of failure that would happen if 
> operators were to follow the existing standards-track docs.
>
> perhaps I'll be in the rough.  If so, of course the wg can and should 
> override my objections after due consideration, but having come this 
> far, I feel it would be irresponsible not to raise the suggestion, as 
> I do believe there is a possibility of end user harm with an unknown 
> high-end scope from just shipping this as-is that has been 
> systematically and aggressively discounted on very thin evidence.  It 
> makes me wonder how many of the supporters have given it due 
> consideration and how many are just impatient.
>
> I understand and agree that changes during deployment might be a good 
> idea.
>
> semantics of the markings (in terms of the exact algorithms and tuning 
> parameters that will be used) to accommodate the expected changes 
> without spec updates?
>
> The current L4S specs place the responsibility primarily on the L4S 
> senders via the vague monitoring requirements (rather than the 
> protocol designers as indicated in the above quote).
>
> protocol design choice to use incompatible signaling has already been 
> rolled out in hardware, so the notion that it's now subject to IETF 
> change control (outside of maybe "yes or no") is a fiction.
>
> 3.
>  From section 2.2:

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 22 Sep 2021 13:16:40 +0300

> On 22 Sep, 2021, at 11:51 am, <[email-masked]> <[email-masked]> wrote:
> 

I'll bite.

Here is a collection of published RFCs which, in combination, meet all the goals you set out above.  You can deploy them *in production* right now, rather than having to wait for even a limited, experimental deployment.

RFC-3168 (ECN)
RFC-8289 (CoDel)
RFC-8290 (FQ-CoDel)
RFC-8312 (CUBIC)

How do they meet your goals?  Let's go through them point by point.

a) Low jitter & loss:  FQ-CoDel keeps queuing delay under sufficient control under the path conditions mentioned, and prevents flows from inducing delay in each other.  The only significant shortcoming is in CoDel's response to the slow-start transient; endpoints can improve this characteristic by also implementing HyStart (or HyStart++).  ECN already eliminates losses due to congestion signalling.  Other causes of loss are not addressed by L4S anyway.

b) There is no admission control required for the above technologies, as they implement a best-effort service.  FQ-CoDel robustly implements max-min fairness between flows.

c) & d) It's hard to get more universal than what's already implemented and widely deployed in Linux et al.  CUBIC is the default CC algorithm for most Internet servers today.  All you need to do is actively turn on ECN negotiation in the client endpoint, and ensure CoDel or FQ-CoDel is deployed at network bottlenecks.

If the above is in fact insufficient for your needs, then please state the remaining shortcomings *with specificity* so that we can have a productive discussion.

 - Jonathan Morton

---

From: <[email-masked]>
Date: Wed, 22 Sep 2021 10:31:47 +0000

Hi Jonathan,

Thanks, I note that I forgot one point, no IP flow based resource sharing:

e) no IP flow based mechanism for the resource to be shared (doesn't scale well with available network provider HW to the extent I'm aware) - that means bulk traffic treatment only

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <[email-masked]> 
Gesendet: Mittwoch, 22. September 2021 12:17
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

I'll bite.

RFC-3168 (ECN)
RFC-8289 (CoDel)
RFC-8290 (FQ-CoDel)
RFC-8312 (CUBIC)

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 22 Sep 2021 13:39:40 +0300

> On 22 Sep, 2021, at 1:31 pm, <[email-masked]> <[email-masked]> wrote:
> 

Okay, then just use CoDel instead of FQ-CoDel.  All you lose is the isolation of flows' delay influences from each other, and the max-min fairness.  I would however suggest that you at least treat traffic to a subscriber as "a flow", and thus perform max-min fairness and delay isolation between subscribers.

With that said, I'm aware of production hardware that implements Approximate Fairness AQM at 100Gbps, and an implementation of FQ-CoDel in P4.  The latter, as far as I'm aware, should scale up to the point where you start using optical switching anyway, at which point the right strategy is probably over-provisioning.  Even software implementations of FQ-CoDel have been tested successfully at 40Gbps.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 22 Sep 2021 11:36:02 +0000

Jonathan

I think that we have discussed this particular topic (isn't CoDel good
enough?) inside and out over the last two years. I posted some comparisons
of CoDel-ECN vs L4S on this list (can be found with some search in the mail
archive). 
That said.. It is apparent that there will be no agreement around this but
that is not necessary for the L4S experiment, and as I see it, not the scope
of the present discussion either.

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
> Sent: den 22 september 2021 12:40
> To: [email-masked]
> Cc: [email-masked]
> > e) no IP flow based mechanism for the resource to be shared (doesn't
scale
> well with available network provider HW to the extent I'm aware) - that
> means bulk traffic treatment only
> 
> Okay, then just use CoDel instead of FQ-CoDel.  All you lose is the
isolation of
> flows' delay influences from each other, and the max-min fairness.  I
would
> however suggest that you at least treat traffic to a subscriber as "a
flow", and
> thus perform max-min fairness and delay isolation between subscribers.
> 
> With that said, I'm aware of production hardware that implements
> Approximate Fairness AQM at 100Gbps, and an implementation of FQ-CoDel
> in P4.  The latter, as far as I'm aware, should scale up to the point
where you
> start using optical switching anyway, at which point the right strategy is
> probably over-provisioning.  Even software implementations of FQ-CoDel
> have been tested successfully at 40Gbps.
> 
>  - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 22 Sep 2021 14:51:58 +0300

> On 22 Sep, 2021, at 2:36 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

There is one key advantage that CoDel-ECN has over L4S: it is deployable on the existing Internet.

> of the present discussion either.

He asked, I answered.  That's all there is to it.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 22 Sep 2021 13:30:59 +0000

Hi

For the sake of completeness. 
Attached are three graphs that depict simulated results with SCReAM
congestion control, the scenario is 5G UL real time video streaming (e.g AR
or remote control) where the channel is degraded due to shadow fading. Note
that results can vary dependent on system configuration and overall radio
conditions so see this as only examples.

SCReAM-5G-noECNorL4S.png : Classic ECN and L4S disabled, SCReAM operates
only based on estimated queue delay 

SCReAM-5G-CoDel-ECN.png : SCReAM is ECN capable and 5G bottleneck implements
CoDel ECN marking

SCReAM-5G-L4S.png : SCReaM is L4S capable and 5G bottleneck implements L4S
marking

These graphs show that CoDel does not achieve the same performance as L4S.
One can of course argue that this almost 30ms less delay jitter with L4S is
not important, but then again nobody is forced to use L4S. 
I anyway see substantial results that L4S can improve performance in 5G for
services like XR, cloud gaming or remote control and therefore look forward
to have L4S ID published as experimental RFC.

/Ingemar

> From: Jonathan Morton <[email-masked]>
> Sent: den 22 september 2021 13:52
> To: Ingemar Johansson S <[email-masked]>
> Cc: [email-masked]; [email-masked]
> > On 22 Sep, 2021, at 2:36 pm, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > enough?) inside and out over the last two years. I posted some
> > comparisons of CoDel-ECN vs L4S on this list (can be found with some
> > search in the mail archive).
> 
> There is one key advantage that CoDel-ECN has over L4S: it is deployable
on
> the existing Internet.
> 
> > That said.. It is apparent that there will be no agreement around this
> > but that is not necessary for the L4S experiment, and as I see it, not
> > the scope of the present discussion either.
>

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 22 Sep 2021 16:36:55 +0300

> On 22 Sep, 2021, at 4:30 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

These charts also show that Codel-ECN gives a significant improvement over having a dumb FIFO at the bottleneck, even with a congestion control that acts on delay information rather than only losses or explicit marking.  That was the point I was making for Herr Geib.

Codel-ECN also shows significantly higher and more consistent goodput than the L4S solution, which may be important in many applications.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 22 Sep 2021 13:42:03 +0000

Jonathan

Thing is that you can still be using CoDel if you prefer to do so and prefer
the higher throughput (peak throughput is not always top priority) . 

/Ingemar

> Sent: den 22 september 2021 15:37
> > On 22 Sep, 2021, at 4:30 pm, Ingemar Johansson S
> > SCReAM-5G-noECNorL4S.png : Classic ECN and L4S disabled, SCReAM
> > operates only based on estimated queue delay
> >
> > SCReAM-5G-CoDel-ECN.png : SCReAM is ECN capable and 5G bottleneck
> > implements CoDel ECN marking
> >
> > SCReAM-5G-L4S.png : SCReaM is L4S capable and 5G bottleneck
> implements
> > L4S marking
> >
> > One can of course argue that this almost 30ms less delay jitter with
> > L4S is not important, but then again nobody is forced to use L4S.
> 
> These charts also show that Codel-ECN gives a significant improvement over
> having a dumb FIFO at the bottleneck, even with a congestion control that
> acts on delay information rather than only losses or explicit marking.
That
> was the point I was making for Herr Geib.
> 
> Codel-ECN also shows significantly higher and more consistent goodput than
> the L4S solution, which may be important in many applications.
> 
>  - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 22 Sep 2021 15:51:29 +0200

Hi Ingemar,

the issue with L4S-SCReAM I see is that it clearly trades in utilization for more robustness against unexpected rate drops... it it not clear how much of its lower packet delay it actually gains from responding quicker to capacity changes (so L4S signaling) and to what degree it is giving itself more time by essentially aiming at only filling the link to 80% or so. Because if all senders would/could do that (aim well below their estimate of true capacity), we would not considerable less queuing delay... The real teats is how will SCReAM behave if the same link also carries a TCP Prague File up- or download that makes sure the ink is running at capacity... 
I mean no disrespect to SCReAM here, it might be well suited for its intended purpose, but as a generalizable model for "internet traffic" it seems to fall short a bit.

Best Regards
	Sebastian

> On Sep 22, 2021, at 15:42, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Jonathan
>

---

From: <[email-masked]>
Date: Wed, 22 Sep 2021 13:55:26 +0000

Jonathan,

several IP flows pass a single access shaper at fixed network access and some may briefly fill a queue there. Per flow treatment doesn't scale well - up to now at least and from those dealing with the issue inside Deutsche Telekom, I didn't yet hear the opposite. There should be Best Effort flows which indicate that they respond to ECN and see low jitter then. The other Best Effort flows may see a queue, as they'd expect. I don't yet get the point what's the benefit balancing traffic of different accesses against each other - the link to DSL gear will likely not be congested. What I expect is, that the operator access router is configured for ECN AQM (and maybe for one new classifier and one new Queue), then things need to work in the way the customers expect (for all flows). I'm not sure I understood conditions and requirements for this to work with your proposal, if scale is regarded as an important criterium.

I've discussed L4S with colleagues in fixed and mobile R&D branch and both look at it as feasible from technical and scalability point of view. I wouldn't communicate here otherwise. If I was aware of alternatives pursued within Deutsche Telekom and requiring support in IETF WG, I'd do so (which is not to say, there are none - there may be some, that I'm not aware of). 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <[email-masked]> 
Gesendet: Mittwoch, 22. September 2021 12:40
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 22 Sep 2021 14:17:38 +0000

Hi

I believe that we can discuss around this for a long while. But I'd suggest
that this is something up to evaluation during the L4S experiment.

/Ingemar

> From: Sebastian Moeller <[email-masked]>
> Sent: den 22 september 2021 15:51
> Cc: Jonathan Morton <[email-masked]>; [email-masked]
> the issue with L4S-SCReAM I see is that it clearly trades in utilization
for more
> robustness against unexpected rate drops... it it not clear how much of
its
> lower packet delay it actually gains from responding quicker to capacity
> changes (so L4S signaling) and to what degree it is giving itself more
time by
> essentially aiming at only filling the link to 80% or so. Because if all
senders
> would/could do that (aim well below their estimate of true capacity), we
> would not considerable less queuing delay... The real teats is how will
> SCReAM behave if the same link also carries a TCP Prague File up- or
> download that makes sure the ink is running at capacity...
> I mean no disrespect to SCReAM here, it might be well suited for its
intended
> purpose, but as a generalizable model for "internet traffic" it seems to
fall
> short a bit.
> 
> 
> Best Regards
> 	Sebastian
> 
> 
> > On Sep 22, 2021, at 15:42, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> > Jonathan
> >
> > Thing is that you can still be using CoDel if you prefer to do so and
> > prefer the higher throughput (peak throughput is not always top
priority) .
> >>> These graphs show that CoDel does not achieve the same performance
> >>> as
> >> L4S.
> >> These charts also show that Codel-ECN gives a significant improvement
> >> over having a dumb FIFO at the bottleneck, even with a congestion
> >> control that acts on delay information rather than only losses or
explicit
> marking.
> >> Codel-ECN also shows significantly higher and more consistent goodput
> >> than the L4S solution, which may be important in many applications.
> >>
> >> - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 22 Sep 2021 17:37:15 +0300

> On 22 Sep, 2021, at 4:55 pm, <[email-masked]> <[email-masked]> wrote:
> 
> several IP flows pass a single access shaper at fixed network access and some may briefly fill a queue there. Per flow treatment doesn't scale well - up to now at least and from those dealing with the issue inside Deutsche Telekom, I didn't yet hear the opposite.

I would also consider the case where an abnormal sustained load is placed upon such an access node.  This could occur, for example, due to a fault elsewhere in the network that has been routed around; you want to maximise the (inevitably reduced) QoE for subscribers while the fault is being worked on.  The infrastructure to handle this task must be in place in advance of need.

> There should be Best Effort flows which indicate that they respond to ECN and see low jitter then. The other Best Effort flows may see a queue, as they'd expect.

An AQM applies to both ECT and Not-ECT traffic equally.  It signals to ECT traffic using ECN marks, and to Not-ECT traffic using packet drops.  So both see the reduced queuing due to the induced congestion responses.

> I don't yet get the point what's the benefit balancing traffic of different accesses against each other - the link to DSL gear will likely not be congested.

It is exactly that point - when there is insufficient access capacity to service all subscribers' demands - where it's important to avoid starving any one subscriber of traffic.  Relying on RTT-fair sharing behaviour between individual flows, which occurs in a dumb FIFO or a simple AQM, is often but not always sufficient.  Applying explicit per-subscriber max-min fairness, which requires examining only the Layer 3 headers (not Layer 4), offers a stronger guarantee that one aggressive traffic flow will not deny service to other subscribers.

> What I expect is, that the operator access router is configured for ECN AQM (and maybe for one new classifier and one new Queue), then things need to work in the way the customers expect (for all flows). I'm not sure I understood conditions and requirements for this to work with your proposal, if scale is regarded as an important criterium.

It is not necessary to add an extra queue to support conventional RFC-3168 ECN.  It is only necessary to add one to support L4S, purely because L4S and conventional traffic are incompatible in the same queue.

> I've discussed L4S with colleagues in fixed and mobile R&D branch and both look at it as feasible from technical and scalability point of view. I wouldn't communicate here otherwise. If I was aware…

Hence why I am trying to raise your awareness of these alternatives.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Wed, 22 Sep 2021 17:25:12 +0200

Hi Ingemar,

The stated goal of L4S as I understand it from the second paragraph of
the l4s-arch Abstract, is that _both_ latency and throughput should be
better than a "classic" congestion controller, but these results would
seem to show that only one of the two has been improved. This is
similar to what we've seen with TCP Prague and BBR tests over wireless
links, or other sources of bursty arrivals in L. Both of our test
results have shown a mismatch between this goal and what we see over
WiFi, airMAX and 5G networks, which seems like it should be addressed
somehow. I've not done any testing on 5G so far, but I would think that
if any wireless technology had a chance of achieving this goal, 5G
would be it.

IMO, we should do better than just mentioning somewhere deeper in the
draft that this goal may not be achieved. One possibility is that the
implementation could be changed to achieve these goals with current
technologies. Another is that it's made clear up front how existing
link/wireless technologies must change before the experiment will run
successfully on them. In that case though, I think we're right to be
cautious before starting an experiment that hasn't been shown to
achieve its stated goal on any currently known wireless technologies.

Regards,
Pete

On Wed, 2021-09-22 at 13:30 +0000, Ingemar Johansson S wrote:
> implements L4S
> L4S is

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 22 Sep 2021 19:28:50 +0000

Perhaps we delve off topic here but I'll try with the answer with the hope 
that I don't mess up this thread

The reason why SCReAM with L4S gives a lower throughput is two-fold

1) Fast fading : Even though the average link throughput is 10Mbps, the 
instantaneous throughput may vary up and down, as L4S strives for low queue 
delay it will mark packets until the queue delay target is reached also when 
the link throughput is below the average link throughput.
2) Varying video frame sizes. Video coders don't produce equal sized frames. 
It is not only occasional I-frames that deviate in size, also P-frames vary in 
size. Larger frames will hit the queue more and therefore cause more packet 
marking, with the result that the target bitrate goes down. This the larger 
the variations in frame size, the more the target bitrate is reduced.

Of the two above , I'd say that #1 is a cellular access specific feature (not 
only 5G), it merely says that you cannot get both full link utilization and 
low queue delay. I don't however really see that as a flaw in L4S, rather that 
L4S is a good tool to find a good working spot that gives low queue delay but 
still high link utilization (albeit not full ditto).

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Pete Heist
> Sent: den 22 september 2021 17:25
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Jonathan Morton
> <[email-masked]>
> Cc: [email-masked]
> The stated goal of L4S as I understand it from the second paragraph of the
> l4s-arch Abstract, is that _both_ latency and throughput should be better
> than a "classic" congestion controller, but these results would seem to show
> that only one of the two has been improved. This is similar to what we've
> seen with TCP Prague and BBR tests over wireless links, or other sources of
> bursty arrivals in L. Both of our test results have shown a mismatch between
> this goal and what we see over WiFi, airMAX and 5G networks, which seems
> like it should be addressed somehow. I've not done any testing on 5G so far,
> but I would think that if any wireless technology had a chance of achieving
> this goal, 5G would be it.
>
> cautious
> before starting an experiment that hasn't been shown to achieve its stated
> goal on any currently known wireless technologies.
>
> Regards,
> Pete
>
> > congestion control, the scenario is 5G UL real time video streaming
> > (e.g AR or remote control) where the channel is degraded due to shadow
> > fading. Note that results can vary dependent on system configuration
> > and overall radio conditions so see this as only examples.
> >
> > I anyway see substantial results that L4S can improve performance in
> > 5G for services like XR, cloud gaming or remote control and therefore
> > look forward to have L4S ID published as experimental RFC.
> >
> > /Ingemar
> >
> >
> > > > comparisons of CoDel-ECN vs L4S on this list (can be found with
> > > > some search in the mail archive).
> > >
> > > > That said.. It is apparent that there will be no agreement around
> > > > this but that is not necessary for the L4S experiment, and as I
> > > > see it, not the scope of the present discussion either.
> > >

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 22 Sep 2021 19:37:33 +0000

From: Bob Briscoe <[email-masked]>
Date: Tue,2021-09-21 at 2:58 PM
> Actually, thinking about it, that is effectively what RFC8311 did. Someone approaching RFC3168 without having heard anything about L4S would discover it is updated by RFC8311. And, by reading RFC8311, they would find that it enables experiments with ECN and refers to L4S. And by reading L4S they would find how it is hoped to improve on RFC3168. That's pretty-close to putting RFC3168 "on hold" - probably as close as the IETF can get. Because to go any further (actively updating or obsoleting present and future RFC3168 implementations) would need evidence from deployment over the real public Internet that the contender is indeed a viable improvement.

RFC 8311 would have sufficed if the original requirement for
backward compatibility had stayed in place and L4S had worked
within its constraints.  But unfortunately the signaling changes
that could have accomplished that were all rejected, so RFC 8311
doesn't end up covering the current proposal.

> You want the RFC series to be self-consistent, but what you are hoping for would just lead to a different sort of inconsistency within the stds track - an experiment with a fake "standards track" label on it. There is loads of content in the L4S docs about what needs to be tested in L4S experiments, none of which can be tested properly except over the real Internet, which in turn cannot be done without assigning a codepoint. The WG cannot pretend that there is no experiment to do by just writing "standards track" in the header block. There is no tidy consistent way through this. 

I did not ask for standards-track on all the drafts, just on
l4s-id.  That's the one with the incompatible change to an IP
header codepoint, and there needs to be a standards track
recommendation for handling ECT(1) that's compatible with the
L4S proposal in order to address my objection.

The algorithm definitions in aqm-dualq and TCP Prague can and
probably should remain experimental for the reasons you outlined.
(And on examination maybe there would be a few passages from
l4s-id that should move into one of the other docs for being too
experimental, though I do expect the impact will be limited.)

I don't see how this would create a new kind of incompatibility.
The parts that can change during the experiment would still be in
the experimental docs, it would essentially just be the codepoint
assignment and the explanation of what its new meaning is that
would be standards track.

We've heard from a few people that compliance with standards-track
RFCs representing IETF consensus is necessary in order to check in
this kind of change to the handling of the codepoints in IP packets
for contexts like major linux distributions, which are important to
the distribution of the marking AQMs that will need updating.

> As I said, the closest to what you want that would be feasible (IMO) would be to draft a stds track update to RFC3168 as soon as L4S deployment starts, which might possibly even be while the L4S drafts are in the RFC editor queue.

As I've said, I believe this is a prerequisite, and can't just be
dropped in someone else's lap (with a TBD volunteer) as you seem to
be suggesting.  As it stands, L4S proposes to roll out an experiment
that is demonstrated unsafe under the recommendations of current
standards-track RFCs, and that mismatch will be a barrier to solving
the problems in production if it's not addressed.

I thought Mike framed it well.  This is incompatible with the current
standards, so either the proposal has to change or the standards have
to change.

-Jake

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 23 Sep 2021 03:18:07 +0000

Hi Ruediger,

> e) no IP flow based mechanism for the resource to be shared (doesn't scale well with available network provider HW to the extent I'm aware) - that means bulk traffic treatment only

If that's the target it seems like PIE or CoDel (without FQ) should
suffice, and L4S's under-1ms target is just not necessary, as I think
Jonathan was saying.  (Not that smaller wouldn't be better, I'm just
puzzled with these targets why there's any blockers to deploying with
the existing implementations.)

I'm not clear on whether you've tested these and rejected them?  I think
Comcast just reported results in this range from a PIE deployment without
marking:
https://datatracker.ietf.org/meeting/111/materials/slides-111-maprg-comparative-latency-under-load-performance-of-broadband-cpe-00.pdf#page=4

I think there's AQMs like these that are already widely implemented and
available in a lot of devices (but apparently rarely enabled), and that
MUST NOT enable marking without some kind of upgrade to handle ECT(1)
differently if L4S ships as-is.

I'm curious whether you've evaluated any of the existing implementations
and if so whether you saw any trouble?  Or are they not present on the
devices relevant to the bottlenecks in your networks?

Best,
Jake

On 09-22, 6:55 AM, "[email-masked]" <[email-masked]> wrote:

Jonathan,

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <[email-masked]> 
Gesendet: Mittwoch, 22. September 2021 12:40
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

 - Jonathan Morton

---

From: <[email-masked]>
Date: Thu, 23 Sep 2021 07:41:38 +0000

Hi Jake,

My domain is the backbone, but QoS only works, if designs are end-to-end, so I'm in exchange with colleagues from other network sections (but don't claim to know them all). I further participate in IETF. That's my "framework of transfers". 

The main requirement I hear from colleagues working on specific applications is, that jitter is to be avoided. Voice conversations suffer more from jitter than they suffer from packet loss. To the extend I understand that, online gaming also shows a better QoE with constant delay. Of course a somewhat reasonable AQM is smoothing traffic, and the queue isn't filled to the max. The backbone AQMs I'm aware of still result in jitter - and I doubt this is any different at the access. I copy in a section from the complete version of the IRTF paper you've referenced (note "*" insertions at the end are from me):

"At a high level, when a device had AQM it consistently experienced between 15-30 milliseconds 
of latency under load. In contrast, without AQM but using the older buffer control configuration, 
a device would experience significantly higher and more variable of working latency. Those non-
AQM devices experienced in many cases 250 milliseconds or higher latency under load. This 
demonstrates two key points: not only does AQM enable significantly better latency under load 
performance, but that latency performance is much more uniform and consistently (e.g., 
*relatively lower jitter*)."

https://arxiv.org/ftp/arxiv/papers/2107/2107.13968.pdf (they do a bit of name dropping related to DOCSIS AQM there too, interesting)

Allright then, the best way to have no jitter is to overprovision. Overprovisioning at a bottleneck means to keep traffic below resources. Throughput optimization may benefit from overbooking and burst buffering (noting that buffer requirements decay with smaller RTT, a higher number of flows and constant MTUs combined with increasing access bandwidth). I'm happy serving both kinds of applications, scalable, reliable and with least effort (including engineering, operation an HW requirements). I translated the requirement "no jitter" to "no jitter". If I'm out for 20-50 ms RTT (RTT, not jitter), I'm not in for 15-30 ms jitter. You are correct however, I didn't come up with a jitter figure. Guts feeling: max jitter 10% of the RTT or less. That's why I for the time being end up with preferring two queues, sharing same resources. 

To complete: PIE and Codel are great innovations and I'd like to have them in the backbone routers I work with, as both produce a queue of constant depth independent of the dequeue speed. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Holland, Jake <[email-masked]> 
Gesendet: Donnerstag, 23. September 2021 05:18
An: Geib, Rüdiger <[email-masked]>; [email-masked]
Cc: [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

Hi Ruediger,

I think your criteria only asked for 20-50ms:

> scale well with available network provider HW to the extent I'm aware) 
> - that means bulk traffic treatment only

If that's the target it seems like PIE or CoDel (without FQ) should suffice, and L4S's under-1ms target is just not necessary, as I think Jonathan was saying.  (Not that smaller wouldn't be better, I'm just puzzled with these targets why there's any blockers to deploying with the existing implementations.)

I'm not clear on whether you've tested these and rejected them?  I think Comcast just reported results in this range from a PIE deployment without
marking:
https://datatracker.ietf.org/meeting/111/materials/slides-111-maprg-comparative-latency-under-load-performance-of-broadband-cpe-00.pdf#page=4

I think there's AQMs like these that are already widely implemented and available in a lot of devices (but apparently rarely enabled), and that MUST NOT enable marking without some kind of upgrade to handle ECT(1) differently if L4S ships as-is.

I'm curious whether you've evaluated any of the existing implementations and if so whether you saw any trouble?  Or are they not present on the devices relevant to the bottlenecks in your networks?

Best,
Jake

Jonathan,

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <[email-masked]>
Gesendet: Mittwoch, 22. September 2021 12:40
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

> scale well with available network provider HW to the extent I'm aware) 
> - that means bulk traffic treatment only

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Fri, 24 Sep 2021 09:20:02 +0200

In my Prague/BBR tests with airMAX, the underutilization was caused by
link layer burstiness rather than rate changes or application bursts,
so we seem to be looking at different causes.

The first issue I have is with the goal as stated in the drafts and the
results we see. There are various possible remedies to that which may
lead to different discussions, but IMO that should be fixed in some
way, and we can also revisit this after any draft changes.

Thanks for the explanations...

Pete

On Wed, 2021-09-22 at 19:28 +0000, Ingemar Johansson S wrote:
> > > These graphs show that CoDel does not achieve the same
> > > performance as
> > > L4S.
> > > > > I think that we have discussed this particular topic (isn't
> > > > > CoDel
> > > > > good

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 24 Sep 2021 09:38:37 +0000

Hi Pete

You are right, what I describe below is not really the same cellular access 
related feature that you described.
Perhaps one can add some wording around this (unless there already is, I need 
to take an extra look).

I don't however think that we should try to quantify the impact on peak 
throughput at this stage as there are so many variables to tune here. Perhaps 
one will have some decent figures to present in the course of the L4S 
experiment.

/Ingemar

> Sent: den 24 september 2021 09:20
> To: [email-masked]
> In my Prague/BBR tests with airMAX, the underutilization was caused by link
> layer burstiness rather than rate changes or application bursts, so we seem
> to be looking at different causes.
>
> results we
> see. There are various possible remedies to that which may lead to different
> discussions, but IMO that should be fixed in some way, and we can also
> revisit this after any draft changes.
>
> > Perhaps we delve off topic here but I'll try with the answer with the
> > hope that I don't mess up this thread
> >
> > 1) Fast fading : Even though the average link throughput is 10Mbps,
> > the instantaneous throughput may vary up and down, as L4S strives for
> > low queue delay it will mark packets until the queue delay target is
> > reached also when the link throughput is below the average link
> > throughput.
> > It is not only occasional I-frames that deviate in size, also P-
> > frames vary in size. Larger frames will hit the queue more and
> > therefore cause more packet marking, with the result that the target
> > bitrate goes down. This the larger the variations in frame size, the
> > more the target bitrate is reduced.
> >
> > Of the two above , I'd say that #1 is a cellular access specific
> > feature (not only 5G), it merely says that you cannot get both full
> > link utilization and low queue delay. I don't however really see that
> > as a flaw in L4S, rather that L4S is a good tool to find a good
> > working spot that gives low queue delay but still high link
> > utilization (albeit not full ditto).
> >
> > /Ingemar
> >
> >
> > > The stated goal of L4S as I understand it from the second paragraph
> > > of the l4s-arch Abstract, is that _both_ latency and throughput
> > > should be better than a "classic" congestion controller, but these
> > > results would seem to show that only one of the two has been
> > > improved. This is similar to what we've seen with TCP Prague and BBR
> > > tests over wireless links, or other sources of bursty arrivals in L.
> > > Both of our test results have shown a mismatch between this goal and
> > > what we see over WiFi, airMAX and 5G networks, which seems like it
> > > should be addressed somehow. I've not done any testing on 5G so far,
> > > but I would think that if any wireless technology had a chance of
> > > achieving this goal, 5G would be it.
> > >
> > > IMO, we should do better than just mentioning somewhere deeper in
> > > the draft that this goal may not be achieved. One possibility is
> > > that the implementation could be changed to achieve these goals with
> > > current technologies. Another is that it's made clear up front how
> > > existing link/wireless technologies must change before the
> > > experiment will run successfully on them. In that case though, I
> > > think we're right to be cautious before starting an experiment that
> > > hasn't been shown to achieve its stated goal on any currently known
> > > wireless technologies.
> > >
> > > Regards,
> > > Pete
> > >

---

From: Greg White <[email-masked]>
Date: Mon, 27 Sep 2021 23:51:38 +0000

This is a key point. 

L4S shifts the discussion from one of "average" latency to one of reliable low latency, where the metric is (e.g.) 99th percentile packet latency under load. The Comcast data shows that traditional AQMs, while they are great at reducing average latency under load, they aren't great at eliminating the latency spikes. This can be seen in the plots that Jake linked to (https://datatracker.ietf.org/meeting/111/materials/slides-111-maprg-comparative-latency-under-load-performance-of-broadband-cpe-00.pdf#page=4).  In this slide, the vertical axis is cumulative frequency *among the tests*, and the horizontal axis is either mean latency measured during the test or max latency measured during the test.  

You can see that the maximum latencies experienced during the tests (using AQM) are typically around 100ms, with 10% of tests showing max latencies above ~160ms, and 5% showing above 200ms.  This compared to the mean latency under load of 30-45ms.  

From the paper (arxiv link below), it is further explained that the latency test uses the "UDP_RR" feature in Netperf and runs for 7 seconds.  The UDP_RR feature uses a ping-pong mechanism where only one measurement packet is in-flight at any time, which causes it to significantly under-sample the latency peaks.  So, the median test result on the "Max" chart of 100ms means that it is sampling the latency of the path once every 100ms during that peak. For those who have looked at the slow-start overshoot typical of TCP flow startups, it isn't hard to imagine that if you are only sampling every 100ms, you may not be capturing the true peak.

The paper also states that the latency measurement starts "as the TCP ramp-up of bandwidth measurement reaches its peak steady-state" and "This period is after the TCP ramp-up".  It is unclear to me whether that means that it misses all of the slow-start overshoot, or whether it catches some part of it. 

There are various definitions of the term "jitter" but it wouldn't be completely absurd to estimate it as the difference between the Max CDF and the Mean CDF from the Comcast data.  So by that definition, the data shows a median "jitter" of 70ms and a 95th percentile "jitter" of 155ms.

-Greg

On 9/23/21, 1:42 AM, "tsvwg on behalf of [email-masked]" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Jake,

    Regards,

    Ruediger

    -----Ursprüngliche Nachricht-----
    Von: Holland, Jake <[email-masked]> 
    Gesendet: Donnerstag, 23. September 2021 05:18
    An: Geib, Rüdiger <[email-masked]>; [email-masked]
    Cc: [email-masked]
    Betreff: Re: [tsvwg] start of WGLC on L4S drafts

    Hi Ruediger,

    Best,
    Jake

    Jonathan,

    Regards,

    Ruediger

    -----Ursprüngliche Nachricht-----
    Von: Jonathan Morton <[email-masked]>
    Gesendet: Mittwoch, 22. September 2021 12:40
    An: Geib, Rüdiger <[email-masked]>
    Cc: [email-masked]; [email-masked]
    Betreff: Re: [tsvwg] start of WGLC on L4S drafts

     - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 28 Sep 2021 11:36:47 +0200

Hi Greg,

> On Sep 28, 2021, at 01:51, Greg White <[email-masked]> wrote:
> 
> This is a key point. 
> 
> L4S shifts the discussion from one of "average" latency to one of reliable low latency, where the metric is (e.g.) 99th percentile packet latency under load.

	[SM] The problem is, L4S might shift the discussion, but in reality it only delivers on this "promise" for carefully selected flows with the expected/correct behavior. Since it does not police and enforce this behavior one must expect in the real internet, that L4S will not deliver reliable low latency at acceptable utilization. And that in a nutshell is why the L4S drafts should be retracted, they do not work robustly and reliably in a mildly adversarial environment, let alone the wild wild west that is the internet, at least not without robust and reliable enforcement of "proper" behavior per flow. And "thanks" to the coupling a flow in either queue can affect both queues, a consequence of using coupling instead of isolation...

> The Comcast data shows that traditional AQMs, while they are great at reducing average latency under load, they aren't great at eliminating the latency spikes.

	[SM] The linked slides compare single-queue DOCSIS PIE versus no AQM (DOCSIS buffer control), I am not convinced that either the slides, nor the paper https://arxiv.org/pdf/2107.13968.pdf actually allow to generalize to all "traditional" AQMs, what data am I missing?

> This can be seen in the plots that Jake linked to (https://datatracker.ietf.org/meeting/111/materials/slides-111-maprg-comparative-latency-under-load-performance-of-broadband-cpe-00.pdf#page=4).  In this slide, the vertical axis is cumulative frequency *among the tests*, and the horizontal axis is either mean latency measured during the test or max latency measured during the test.  
> 

	[SM] Again, this seems all true for the DOCSIS-PIE versus DOCSIS buffer-control comparison made in that paper, but I am not convinced that this actually generalizes to all AQMs, let alone FQ_AQMs (that maintain separate queues and AQM state per stochastic flow) or AFQ_AQMs (that maintain AQM state per stochastic flow, but that use a single queue). Sure, Ruediger does not seem interested in FQ solutions (for lack of FQ options in the gear he uses), so the paper nicely demonstrates how much can already be achieved with the relative modest means of a single PIE instance (but gain, IIUC the paper only tests the upstream direction, where typical DOCSIS plans top out at ~50Mbps, hardly a speed at whit FQ becomes computationally infeasible and hardly a demonstration that even single queue PIE will work at 100 or 400 Gbps backbone rates*).

Regards
	Sebastian

---

From: <[email-masked]>
Date: Tue, 28 Sep 2021 10:00:18 +0000

Sebastian,

I'm interested in an edge/access router forwarding (and transport) mechanism fulfilling the requirements I've posted. Please don't quote me otherwise. 

To repeat from one of my posts:
"To complete: PIE and Codel are great innovations and I'd like to have them in the backbone routers I work with, as both produce a queue of constant depth independent of the dequeue speed." Constant depth in terms of [ms] (to express it correctly). 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Dienstag, 28. September 2021 11:37
An: Greg White <[email-masked]>
Cc: Geib, Rüdiger <[email-masked]>; [email-masked]; [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

Hi Greg,

Regards
	Sebastian

>    https://arxiv.org/ftp/arxiv/papers/2107/2107.13968.pdf (they do a 
> bit of name dropping related to DOCSIS AQM there too, interesting)
> 
> https://datatracker.ietf.org/meeting/111/materials/slides-111-maprg-co
> mparative-latency-under-load-performance-of-broadband-cpe-00.pdf#page=
> 4
>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 28 Sep 2021 13:41:21 +0300

> On 28 Sep, 2021, at 1:00 pm, <[email-masked]> <[email-masked]> wrote:
> 

May I ask if your hardware supports P4, or at least if hardware in the appropriate class is available supporting P4? That may be an avenue for introducing Codel in the near term, if so.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 28 Sep 2021 12:52:18 +0200

Ruediger,

I am sorry if I mis-paraphrazed your point:

"e) no IP flow based mechanism for the resource to be shared (doesn't scale well with available network provider HW to the extent I'm aware) - that means bulk traffic treatment only"

as "Ruediger does not seem interested in FQ solutions (for lack of FQ options in the gear he uses)". 

So is your position that you would not use FQ AQMs even if your HW would offer it at the rates you need? 

Best Regards
	Sebastian

P.S.: IMHO your points a) and e) seem mutually exclusive:

The reliable low-jitter from a) only seems robustly and reliable achievable if you isolate well-behaving packets from mis-behaving packets and that violates at least  e) and maybe b) and c). I agree it would be desirable if a simple best-effort "cooperative" system like L4S proposes would work, but as Pete's tests have demonstrated, it fails to do so robustly and reliably. 

And as before I consider d) to be backwards, you start out with experimentation in suitable environments, before you standardize something; how would you know whether your proposed design and implementation actually achieves its promises? 

> On Sep 28, 2021, at 12:00, <[email-masked]> <[email-masked]> wrote:
> 
> Sebastian,
>

---

From: <[email-masked]>
Date: Tue, 28 Sep 2021 14:19:06 +0000

Jonathan,

We are working with vendors shipping their own HW (or partially do so), and they also integrate HW on their own. QoS isn't the key driver, when these systems are specified. VoQ, "per flow treatment" and fair scheduling are part of the discussions we have, but on backbone scale. PIE has been created by Cisco. That doesn't mean, it has or will be deployed in all their gear. Linecard throughput is in Tbit/s range on backplane, dequeue-speed is on Gbit/s range. Buffers have to cope with that, when they operate fair per flow per queue features.

Deutsche Telekom is also engineering a disaggregated access router / BNG, see https://www.telekom.com/en/media/media-information/archive/deutsche-telekom-s-access-4-0-platform-goes-live-615974 . I'm not working on it, but I'm in contact with colleagues who are involved. But even if an operator starts to act as a system integrator: if you run several hundred routers serving millions of customers, any single feature must be very important to justify a replacement of HW (and all the effort linked to this action). QoS is one aspect of many determining choices and solutions. My colleagues are aware of recent AQMs like PIE/(FQ) codel and L4S.

I've had a glance on https://arxiv.org/pdf/2010.04528.pdf (July 2021). P4 may be a useful approach, however it doesn't read as if that is ready for deployment in a commercial network, or being close to that.  

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <[email-masked]> 
Gesendet: Dienstag, 28. September 2021 12:41
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

 - Jonathan Morton

---

From: <[email-masked]>
Date: Tue, 28 Sep 2021 14:42:04 +0000

Sebastian,

what is relevant is what suit's requirements at scale (and in time, cost and the other factors relevant for a long term stable deployment serving a mass market). I'm not religious for any particular solution. If an offered solution is to allow for flows creating congestion / queue build up to optimize throughput and "no jitter" for flows which do not want to experience jitter all in a single queue, I doubt that this can be realised with today's queuing mechanisms. I've had done so in the backbone, if tools were at hand. I don't claim to know it all, and the backbone isn't access. No jitter means no jitter, apart from random collisions, router-process differences and some single digit ms buffer indicating queue-build up. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Dienstag, 28. September 2021 12:52
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

Ruediger,

Best Regards
	Sebastian

---

From: Luca Muscariello <[email-masked]>
Date: Tue, 28 Sep 2021 16:49:42 +0200

Hi Ruediger,

off topic, just FYI.

On Tue, Sep 28, 2021 at 4:19 PM <[email-masked]> wrote:

> Jonathan,
>
> We are working with vendors shipping their own HW (or partially do so),
> and they also integrate HW on their own. QoS isn't the key driver, when
> these systems are specified. VoQ, "per flow treatment" and fair scheduling
> are part of the discussions we have, but on backbone scale. PIE has been
> created by Cisco. That doesn't mean, it has or will be deployed in all
> their gear. Linecard throughput is in Tbit/s range on backplane,
> dequeue-speed is on Gbit/s range. Buffers have to cope with that, when they
> operate fair per flow per queue features.
>

Even if you're talking about a different routing set of products, the ASIC
that is used in the Cisco Nexus 9000 series

https://www.ciscolive.com/c/dam/r/ciscolive/us/docs/2019/pdf/CTHDCN-2301.pdf

is using flow aware technologies, like AFD and DPP that scale very well in
hardware
as they need no egress scheduling.

Best
Luca

---

From: <[email-masked]>
Date: Wed, 29 Sep 2021 07:25:05 +0000

Hi Luca,

I think to have dealt with VoQ features like “Approximate Fair Queueing”, but don’t favor them within the backbone. I think it is important to consider the definition of “flow” at which these systems reach the scale expected by the company operating them, and the presentation you’ve linked doesn’t define “flow” (it mentions a configurable buffer threshold to classify a “flow” as elephant). To offer a different sight: the operator wants to serve n egress interfaces and needs an appropriate number of queues and related counters to reliably operate a router with the desired load range over a long period. That impacts the number of flows, I think and that will impact how to define “flow”.
Dynamic Packet Prioritisation of low volume flows isn’t offered with the routers I’m working with, at least I’m not aware of ways to configure it. In general, if that’s a “two queue solution” and if a reasonable classification and a separate AQM can be easily configured, it might indeed be useful.

Regards,

Ruediger

Von: Luca Muscariello <[email-masked]>
Gesendet: Dienstag, 28. September 2021 16:50
An: Geib, Rüdiger <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; tsvwg IETF list <[email-masked]>
Betreff: Re: [tsvwg] start of WGLC on L4S drafts

Hi Ruediger,

off topic, just FYI.

On Tue, Sep 28, 2021 at 4:19 PM <[email-masked]<mailto:[email-masked]>> wrote:
Jonathan,

Even if you're talking about a different routing set of products, the ASIC that is used in the Cisco Nexus 9000 series

is using flow aware technologies, like AFD and DPP that scale very well in hardware
as they need no egress scheduling.

Best
Luca

---

From: Luca Muscariello <[email-masked]>
Date: Wed, 29 Sep 2021 10:13:47 +0200

Hi Ruediger

On Wed, Sep 29, 2021 at 9:25 AM <[email-masked]> wrote:

> Hi Luca,
>
>
>
> I think to have dealt with VoQ features like “Approximate Fair Queueing”,
> but don’t favor them within the backbone. I think it is important to
> consider the definition of “flow” at which these systems reach the scale
> expected by the company operating them, and the presentation you’ve linked
> doesn’t define “flow” (it mentions a configurable buffer threshold to
> classify a “flow” as elephant). To offer a different sight: the operator
> wants to serve n egress interfaces and needs an appropriate number of
> queues and related counters to reliably operate a router with the desired
> load range over a long period. That impacts the number of flows, I think
> and that will impact how to define “flow”.
>

I agree with you. I was not proposing to use that ASIC in a backbone router
and indeed
Cisco does not use it there. I was just bringing that up to say that Cisco
has proved
for quite a long time that we all have reached the technology level to have
flow aware processing in silicon today. For any line rate.

If a backbone link is creating queuing, that's an issue that should be
addressed with
traffic engineering, proactively in principle, or reactively in the worst
case.
If that issue happens, for as little time as possible, relevant classes of
applications should be protected. But I also think diffserv serves well for
that need with
very predictable performance.

Best
Luca

> Dynamic Packet Prioritisation of low volume flows isn’t offered with the
> routers I’m working with, at least I’m not aware of ways to configure it.
> In general, if that’s a “two queue solution” and if a reasonable
> classification and a separate AQM can be easily configured, it might indeed
> be useful.
>
>
>
> Regards,
>
>
>
> Ruediger
>
>
>
> *Von:* Luca Muscariello <[email-masked]>
> *Gesendet:* Dienstag, 28. September 2021 16:50
> *An:* Geib, Rüdiger <[email-masked]>
> *Cc:* Jonathan Morton <[email-masked]>; tsvwg IETF list <
> [email-masked]>
> *Betreff:* Re: [tsvwg] start of WGLC on L4S drafts
>
>
>
> Hi Ruediger,
>
>
>
> off topic, just FYI.
>
>
>
>
>