Subject: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude 24th March 2021

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 9 Mar 2021 16:05:43 +0000

TSVWG,

The WG is running an adoption call for:

     Operational Guidance for Deployment of L4S in the Internet
     https://tools.ietf.org/html/draft-white-tsvwg-l4sops-01

This document is currently marked as INFORMATIONAL. The adoption call 
will end at midnight UTC on 24th March 2021.

Please do read the draft, and send any comments/concerns to either the 
tsvwg or to the chairs <[email-masked]>. Notes of support to 
progress this work, and volunteers to review the drafts will also be 
very welcome at this stage.

Best wishes,
Gorry, Wes and David
(tsvwg co-chairs)

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 10 Mar 2021 16:43:43 +0000

Gorry, chairs,

I support the adoption of this draft.
I've read it
I'm prepared to contribute text

Note, contrary to the call below, the draft is not at -01. It has been 
at rev -02 since 22 Feb 21:
https://datatracker.ietf.org/doc/html/draft-white-tsvwg-l4sops-02

(Use the datatracker, not tools.ietf.org at the moment - seems to have 
been bugged for last few weeks. See PS. below)

Bob

PS. I have raised a ticket about the tools facility, which seems to be 
intermittently and partially switching the drafts database to some date 
a few weeks ago .
It's possible Gorry got caught out by that bug.
Others have reported it on [email-masked] over the past few weeks.

Submitted draft is missing from tools: 
https://mailarchive.ietf.org/arch/msg/tools-discuss/RA5Pw2624MwgmWmF_hFUOd9ttGc/
tools.ietf.org oddness: 
https://mailarchive.ietf.org/arch/msg/tools-discuss/4-_ipvdBMxwP0BuDw9UYFZfZVi8/
Glitch? 
https://mailarchive.ietf.org/arch/msg/tools-discuss/yhVVnOjqEitdiownV0QUfz3v7lI/
Glitch? [Fwd: I-D Action: draft-ietf-opsec-v6-24.txt]: 
https://mailarchive.ietf.org/arch/msg/tools-discuss/ZID1X53FJCLTAz3V8SYx79reAVo/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 10 Mar 2021 12:43:00 -0500

I've read the -01 version, and volunteer to review future versions of the
draft.

best regards,
neal

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 10 Mar 2021 21:07:37 +0000

I support adoption of this draft.  I will make a reasonable attempt
to send at least one review at a useful time.

Best regards,
Jake

On 3/9/21, 8:05 AM, "Gorry Fairhurst" <[email-masked]> wrote:

TSVWG,

     Operational Guidance for Deployment of L4S in the Internet
     https://urldefense.com/v3/__https://tools.ietf.org/html/draft-white-tsvwg-l4sops-01__;!!GjvTz_vk!G52Z49Is5LRm5Cedp3ChsLIrs42fE2xvh_azOMppphImhzDznDEVS8ra-GPlWW0$ 

Best wishes,
Gorry, Wes and David
(tsvwg co-chairs)

---

From: "Smith, Kevin, Vodafone Group" <[email-masked]>
Date: Thu, 11 Mar 2021 15:54:07 +0000

I support adoption of the draft, have read -01 and will provide a review.

All best,
Kevin

C2 General
From: tsvwg <[email-masked]> On Behalf Of Neal Cardwell
Sent: 10 March 2021 17:43
To: Bob Briscoe <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude 24th March 2021

CYBER SECURITY WARNING: This email is from an external source - be careful of attachments and links. Please follow the Cyber Code and report suspicious emails.
I support the adoption of this draft.
I've read the -01 version, and volunteer to review future versions of the draft.

best regards,
neal

On 09/03/2021 16:05, Gorry Fairhurst wrote:
> tsvwg or to the chairs <[email-masked]<mailto:[email-masked]>>. Notes of support to

---

From: "Livingood, Jason" <[email-masked]>
Date: Thu, 11 Mar 2021 19:08:50 +0000

I support adoption of the latest draft & can provide review/comment in the future.

Jason

On 3/9/21, 11:08 AM, "tsvwg on behalf of Gorry Fairhurst" <[email-masked] on behalf of [email-masked]> wrote:

    TSVWG,

         Operational Guidance for Deployment of L4S in the Internet
         https://urldefense.com/v3/__https://tools.ietf.org/html/draft-white-tsvwg-l4sops-01__;!!CQl3mcHX2A!StPJzTgJPtIYZUEgSvqrb8RKy53CXJif_Kt30cCPF7rXhC0CpaeQTkRgt3GMRQyHIpBvTw$

    Best wishes,
    Gorry, Wes and David
    (tsvwg co-chairs)

---

From: Pete Heist <[email-masked]>
Date: Fri, 12 Mar 2021 13:31:14 +0100

I'll explain my position of not taking a stance on this document.

The available data suggest that L4S and conventional traffic cannot
coexist safely in the same RFC3168 signaling queue, and that those
queues have been deployed to some extent. I don't think this is
something that operational guidance by itself can fix, and would prefer
to see that addressed first.

However, I'm also not sure that impeding progress on this document is
helpful. In case the WG decides that there is rough consensus to
proceed with known safety issues, then I'd likely be among those
insisting on WGLC for this draft together with the L4S drafts. This
will provide published information on what to expect, and document what
issues were known by the IETF prior to approval for experiment.

If the document is adopted, then I should be able to provide at least
one review down the line.

Pete

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 15 Mar 2021 12:35:17 +0200

I do not think this document is ready for adoption in its current form.  Let me explain why, and suggest some ways it could be improved.

L4S has a fundamental incompatibility with conventional AIMD traffic in the presence of RFC-3168 ECN AQMs, just like DCTCP upon which it was based.  L4S therefore requires mitigations to ensure that the harm caused by this incompatibility is minimised to an acceptable level.  Since the harm is primarily caused to "innocent bystanders" rather than "involved participants" or "interested observers", the acceptable level of harm and risk is especially low, and the mitigations need to be correspondingly robust.

However, robust mitigations are not what l4s-ops currently describes.  Most of the measures described fall into three categories:

1: Reliance on detecting an RFC-3168 AQM and disabling the L4S behaviour, using heuristics that have not yet been shown in a reliably working state, even under lab conditions.  It is impossible to state that such a heuristic can be relied upon until such a showing has been made.  A previous attempt at implementing such a heuristic was unsuccessful and is now disabled by default in the reference implementation.  Hence, the reliability of such a heuristic would necessarily be a subject of the experiment, not the primary safeguard.

2: Requirements placed upon "innocent bystanders" to avoid the harm, mostly by reconfiguring, replacing, or disabling their RFC-3168 AQMs (sometimes in an RFC-ignorant manner).  This is obviously unworkable, since by definition "innocent bystanders" are unaware of the experiment, and even if made aware, are disinterested in doing work to accommodate it.

3: Recommendation to deploy L4S hosts on networks that have been prepared to receive it.  Which is a step in the right direction.  But this is not accompanied by a corresponding requirement to *contain* L4S traffic to each prepared network.  Without such a requirement, it would be very easy for L4S hosts on different networks, which may individually have been prepared, to communicate over the path between those networks that has *not* been prepared, and upon which the risk of disrupting bystander traffic therefore exists.

It is perhaps noteworthy that gaps in the second and third classes of mitigation are proposed to be covered by the first class of mitigation.  I also note that there is still an assertion in the text that RFC-3168 AQMs are "rare", which is refuted by recent data.  Finally, in the context of a CDN-ISP pairing for an experimental deployment, the ISP subscribers' LANs and WLANs are technically separate networks that would be difficult to "prepare" for L4S in advance; it would be wise to consider the ramifications of that.

I also note in passing that a modification of tunnel encapsulation semantics is also proposed.  Given that tunnel implementations are more diverse than RFC-3168 AQM implementations, I also consider this unlikely to be practical, though I haven't studied in detail whether it would be effective if achieved.

I am currently aware of four theoretical methods of robustly mitigating the risk posed by L4S.  I think that l4s-ops would be considerably improved by proposing that at least one of them be employed as a prerequisite to the L4S experiment actually taking place:

1: Develop, implement, demonstrate, and open for scrutiny an RFC-3168 detection heuristic that is reliable and prompt enough to serve as a primary safeguard for the experiment.  In my opinion this will be difficult and will take significant time, but is not impossible to achieve.

2: Deprecate RFC-3168, or amend it to remove drop-equivalent marking of ECT(1) packets, and require the removal of all unmodified ECN AQMs from the Internet.  This is unlikely to get much support given the increasing deployment rates of RFC-3168 AQMs at the present time.  In any case it would take a very long time to eliminate existing RFC-3168 AQM deployments at Internet scale, so I consider this impractical.

3: Explicitly contain L4S traffic to networks that have been prepared or designated for the experiment.  That could be done by marking all L4S traffic with a designated DSCP at origin, and blocking traffic carrying that DSCP from traversing border gateways into unprepared networks.  This has the effect of making users and administrators of these networks at least "interested observers" and isolating L4S traffic from "innocent bystanders".  Within the designated networks, observing the practical interactions between L4S and conventional traffic would be part of the experiment.

4: Redesign L4S to shift the risk burden away from "innocent bystanders".  The most obvious way to do so is to implement unambiguous signalling by the network, so that the receiver knows for certain whether it is receiving congestion signals from an RFC-3168 AQM requesting an immediate MD response, or from an AQM of the new type requesting a new type of response.  The risk of performance trouble is then restricted to network nodes that produce the new signals and transport endpoints that understand them - in other words, to the relatively small number of "involved participants" who have the knowledge and incentive to study the problem and find solutions.  The incentives are thus aligned correctly and risks are not "externalised".

The SCE proposal does exactly that, in a manner that is totally transparent to existing RFC-3168 endpoints and middleboxes.  It becomes practical, for example, to use a Differentiated Services Code Point to differentiate a low-latency service onto a second bearer and provide a single-queue SCE AQM there, while providing a single-queue RFC-3168 AQM (without SCE) on the primary bearer.  Because of the unambiguous signalling, SCE traffic missing the DSCP would still compete on equal terms with conventional traffic, instead of dominating it or being dominated.

I realise that this last method is not strictly in scope for the l4s-ops draft (and that mentions of SCE tend to raise hackles among L4S proponents), but I include it because it appears to be the most robust mitigation method available.  It also has the advantage of running code being available to try it out immediately.

I am not hugely optimistic that the l4s-ops draft will incorporate the above advice before the adoption call ends.  But unless and until it does, my position is that it SHOULD NOT be adopted.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 15 Mar 2021 20:36:55 +0100

Hi Gory, hi List,

as noted elsewhere, this draft needs some more work before being done; and also I am not sure that I consider any of L4S ready to proceed, so I do not see much urgency, but unless the other drafts are dropped (as I hope they would have been long ago) I see no point in not also adopting this draft.

I will try to send comments the the last version this months.

Best Regards
	Sebastian

> On Mar 9, 2021, at 17:05, Gorry Fairhurst <[email-masked]> wrote:
> 
> TSVWG,
> 
> This document is currently marked as INFORMATIONAL. The adoption call will end at midnight UTC on 24th March 2021.
> 
> Please do read the draft, and send any comments/concerns to either the tsvwg or to the chairs <[email-masked]>. Notes of support to progress this work, and volunteers to review the drafts will also be very welcome at this stage.
> 
> Best wishes,
> Gorry, Wes and David
> (tsvwg co-chairs)
>

---

From: Greg White <[email-masked]>
Date: Mon, 15 Mar 2021 21:17:38 +0000

Hi Jonathan,

This is an adoption call, not a call for approval of the text.  In my view, the question is whether the working group members are interested in providing guidance to L4S experimenters to address the issue of compatibility with RFC3168 bottlenecks.  

I'll respond to some of your points below in a separate thread so as not to disrupt the adoption call.

-Greg 

On 3/15/21, 4:35 AM, "tsvwg on behalf of Jonathan Morton" <[email-masked] on behalf of [email-masked]> wrote:

     - Jonathan Morton

---

From: Martin Duke <[email-masked]>
Date: Fri, 19 Mar 2021 07:20:31 -0700

As an individual, I support adoption.

On Mon, Mar 15, 2021 at 2:17 PM Greg White <[email-masked]> wrote:

> Hi Jonathan,
>
> This is an adoption call, not a call for approval of the text.  In my
> view, the question is whether the working group members are interested in
> providing guidance to L4S experimenters to address the issue of
> compatibility with RFC3168 bottlenecks.
>
> On 3/15/21, 4:35 AM, "tsvwg on behalf of Jonathan Morton" <
> [email-masked] on behalf of [email-masked]> wrote:
>
>     I do not think this document is ready for adoption in its current
> form.  Let me explain why, and suggest some ways it could be improved.
>
>     L4S has a fundamental incompatibility with conventional AIMD traffic
> in the presence of RFC-3168 ECN AQMs, just like DCTCP upon which it was
> based.  L4S therefore requires mitigations to ensure that the harm caused
> by this incompatibility is minimised to an acceptable level.  Since the
> harm is primarily caused to "innocent bystanders" rather than "involved
> participants" or "interested observers", the acceptable level of harm and
> risk is especially low, and the mitigations need to be correspondingly
> robust.
>
>     However, robust mitigations are not what l4s-ops currently describes.
> Most of the measures described fall into three categories:
>
>     1: Reliance on detecting an RFC-3168 AQM and disabling the L4S
> behaviour, using heuristics that have not yet been shown in a reliably
> working state, even under lab conditions.  It is impossible to state that
> such a heuristic can be relied upon until such a showing has been made.  A
> previous attempt at implementing such a heuristic was unsuccessful and is
> now disabled by default in the reference implementation.  Hence, the
> reliability of such a heuristic would necessarily be a subject of the
> experiment, not the primary safeguard.
>
>     2: Requirements placed upon "innocent bystanders" to avoid the harm,
> mostly by reconfiguring, replacing, or disabling their RFC-3168 AQMs
> (sometimes in an RFC-ignorant manner).  This is obviously unworkable, since
> by definition "innocent bystanders" are unaware of the experiment, and even
> if made aware, are disinterested in doing work to accommodate it.
>
>     3: Recommendation to deploy L4S hosts on networks that have been
> prepared to receive it.  Which is a step in the right direction.  But this
> is not accompanied by a corresponding requirement to *contain* L4S traffic
> to each prepared network.  Without such a requirement, it would be very
> easy for L4S hosts on different networks, which may individually have been
> prepared, to communicate over the path between those networks that has
> *not* been prepared, and upon which the risk of disrupting bystander
> traffic therefore exists.
>
>     It is perhaps noteworthy that gaps in the second and third classes of
> mitigation are proposed to be covered by the first class of mitigation.  I
> also note that there is still an assertion in the text that RFC-3168 AQMs
> are "rare", which is refuted by recent data.  Finally, in the context of a
> CDN-ISP pairing for an experimental deployment, the ISP subscribers' LANs
> and WLANs are technically separate networks that would be difficult to
> "prepare" for L4S in advance; it would be wise to consider the
> ramifications of that.
>
>     I also note in passing that a modification of tunnel encapsulation
> semantics is also proposed.  Given that tunnel implementations are more
> diverse than RFC-3168 AQM implementations, I also consider this unlikely to
> be practical, though I haven't studied in detail whether it would be
> effective if achieved.
>
>
>     I am currently aware of four theoretical methods of robustly
> mitigating the risk posed by L4S.  I think that l4s-ops would be
> considerably improved by proposing that at least one of them be employed as
> a prerequisite to the L4S experiment actually taking place:
>
>     1: Develop, implement, demonstrate, and open for scrutiny an RFC-3168
> detection heuristic that is reliable and prompt enough to serve as a
> primary safeguard for the experiment.  In my opinion this will be difficult
> and will take significant time, but is not impossible to achieve.
>
>     2: Deprecate RFC-3168, or amend it to remove drop-equivalent marking
> of ECT(1) packets, and require the removal of all unmodified ECN AQMs from
> the Internet.  This is unlikely to get much support given the increasing
> deployment rates of RFC-3168 AQMs at the present time.  In any case it
> would take a very long time to eliminate existing RFC-3168 AQM deployments
> at Internet scale, so I consider this impractical.
>
>     3: Explicitly contain L4S traffic to networks that have been prepared
> or designated for the experiment.  That could be done by marking all L4S
> traffic with a designated DSCP at origin, and blocking traffic carrying
> that DSCP from traversing border gateways into unprepared networks.  This
> has the effect of making users and administrators of these networks at
> least "interested observers" and isolating L4S traffic from "innocent
> bystanders".  Within the designated networks, observing the practical
> interactions between L4S and conventional traffic would be part of the
> experiment.
>
>     4: Redesign L4S to shift the risk burden away from "innocent
> bystanders".  The most obvious way to do so is to implement unambiguous
> signalling by the network, so that the receiver knows for certain whether
> it is receiving congestion signals from an RFC-3168 AQM requesting an
> immediate MD response, or from an AQM of the new type requesting a new type
> of response.  The risk of performance trouble is then restricted to network
> nodes that produce the new signals and transport endpoints that understand
> them - in other words, to the relatively small number of "involved
> participants" who have the knowledge and incentive to study the problem and
> find solutions.  The incentives are thus aligned correctly and risks are
> not "externalised".
>
>     The SCE proposal does exactly that, in a manner that is totally
> transparent to existing RFC-3168 endpoints and middleboxes.  It becomes
> practical, for example, to use a Differentiated Services Code Point to
> differentiate a low-latency service onto a second bearer and provide a
> single-queue SCE AQM there, while providing a single-queue RFC-3168 AQM
> (without SCE) on the primary bearer.  Because of the unambiguous
> signalling, SCE traffic missing the DSCP would still compete on equal terms
> with conventional traffic, instead of dominating it or being dominated.
>
>     I realise that this last method is not strictly in scope for the
> l4s-ops draft (and that mentions of SCE tend to raise hackles among L4S
> proponents), but I include it because it appears to be the most robust
> mitigation method available.  It also has the advantage of running code
> being available to try it out immediately.
>
>
>     I am not hugely optimistic that the l4s-ops draft will incorporate the
> above advice before the adoption call ends.  But unless and until it does,
> my position is that it SHOULD NOT be adopted.
>
>      - Jonathan Morton
>
>
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 19 Mar 2021 14:26:45 +0000

Hi

I support adoption of draft-white-tsvwg-l4sops and will help with the review of the document

/Ingemar

From: tsvwg <[email-masked]> On Behalf Of Martin Duke
Sent: den 19 mars 2021 15:21
To: Greg White <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude 24th March 2021

On Mon, Mar 15, 2021 at 2:17 PM Greg White <[email-masked] <mailto:[email-masked]> > wrote:

Hi Jonathan,

-Greg 

On 3/15/21, 4:35 AM, "tsvwg on behalf of Jonathan Morton" <[email-masked] <mailto:[email-masked]>  on behalf of [email-masked] <mailto:[email-masked]> > wrote:

     - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Fri, 19 Mar 2021 07:39:12 -0700

I think this working group has far too much on its plate, and too many
unresolved issues elsewhere to adopt further work on this subject.

I do not support adoption at this time.

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 20 Mar 2021 08:13:35 +0000

> unresolved issues elsewhere to adopt further work on this subject.
<snip>

If people on the list plan to read and review draft-white-tsvwg-l4sops, 
they should say so on this list before the end of the adoption call period.

Completing the adoption call is considered by the Chairs as an important 
step towards completing the following work items:

     Oct 2021     Submit "Identifying Modified Explicit Congestion 
Notification (ECN)

     Semantics for Ultra-Low Queuing Delay" as an Experimental RFC

     draft-ietf-tsvwg-ecn-l4s-id

     Oct 2021     Submit "DualQ Coupled AQM for Low Latency, Low Loss and

     Scalable Throughput" as an Experimental RFC

     draft-ietf-tsvwg-aqm-dualq-coupled

     Oct 2021     Submit "Low Latency, Low Loss, Scalable Throughput (L4S)

     Internet Service: Architecture" as an Informational RFC

     draft-ietf-tsvwg-l4s-arch

Please respond by 24th March 2021,

Best wishes,

Gorry, David and Wes

tsvwg co-chairs.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 21 Mar 2021 00:54:16 +0100

Hi Gorry,

I am puzzled, what has changed in regards to the documented short comings of L4S core technologies the dual queue coupled AQM and TCP Prague in respect to what is claimed in the internet drafts that would justify for the drafts to proceed in the process to gain experimental RFC status?

A quick look at https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667
should help refresh the memory how dualQ and TCP Prague fare in comparison with the status quo in the internet, a dumb FIFO in regards to equitable sharing*.
Giving dualQ's main reason to exist is to equitably share between the new L4S LL-queue and what is called the classic-queue, this failure alone should IMHO disqualify L4S from any further discussion/progress in the WG to give team L4S the time and incentive to fix this. But alas, not my call to make so all I can do is remind everybody that these short-coming/failures are no secret and nobody should be surprised if L4S in the real internet under-deliver in its (over-)promises. 	Unless somebody can explain why this failure easily testable in the lab will not materialize in the real internet?

Best Regards
	Sebastian

> On Mar 20, 2021, at 09:13, Gorry Fairhurst <[email-masked]> wrote:
> 
> If people on the list plan to read and review draft-white-tsvwg-l4sops, they should say so on this list before the end of the adoption call period.
> 
> Completing the adoption call is considered by the Chairs as an important step towards completing the following work items:
> 
>     Oct 2021     Submit "Identifying Modified Explicit Congestion Notification (ECN)
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 21 Mar 2021 02:55:39 +0000

> Giving dualQ's main reason to exist is to equitably share between the new L4S LL-queue and what is called the classic-queue, this failure alone should IMHO disqualify L4S from any further discussio…
Sebastian,

I think people are well aware of your own thoughts. This email is part 
of a thread on the planned working group adoption of 
draft-white-tsvwg-l4sops.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 21 Mar 2021 09:54:15 +0100

Hi Gory,

I just hoist your response to the top to make things easier to read:

"I think people are well aware of your own thoughts. This email is part of a thread on the planned working group adoption of draft-white-tsvwg-l4sops."

Which puzzles me for two reasons:

a) The current state of L4S design and implementation: data I cited shows catastrophic failures of L4S's two core components under realistic conditions. Why this is not considered a show-stopper and/or all-hand-on-deck situation for the WG and team L4S, is hard to understand. 
	Also, why people in this WG are willing to just ignore that fact instead of engaging in a discussion of how to meaningfully remedy that (so changing the actual L4S core technologies and not just watering down the drafts text, to "legally" permit tose failure modes). So what is the hurry of adding yet another L4S draft, which in an ideal world would probably need major changes once the other L4S drafts are actually fixed?
	I understand that everybody is entitled to their own opinion and also to ignore other's opinions, but we all have to operate on the same set of data, and it is not my subjective judgement (which would be L4S: " too little, too late") but hard cold data that shows L4S failing. Ignoring that now, will not reflect well on the IETF and this WG once L4S hits the internet and starts to disintegrate, sorry. I have heard excuses that this is part of the IETF process, but I do not see chapter and verse in the RFCs that a WG needs to entertain incomplete glacially advancing internet drafts for all eternity, sending team L4S away with the home work to fix their stuff seems well in the purview of this WG. Heck the WG could even increase the motivation for team L4S to act fast for a change, by ending the ban on oher proposals for using ECT(1) (under the rationale that the current state of L4S demonstrates that ECT(1) as network input simply does not work, at which point the WG decision to use ECT(1) as input becomes moot)).

b) The ops draft itself is also not ready. I gave detailed feed-back (at multiple stages in its development and ) and have (unfortunately) have read it multiple times. How about we first let this last feed-back round play out before making adoption calls?

	All that said, my prediction is that we as a WG are going to accept the L4S drafts in autumn as is as experimental RFCs (as far as this WG can, I understand that we  will not have the last word, but I expect no headwind in the later process either, as detailed evaluation and judgement seems the prerogative of the WGs, so higher ups will assume due diligence from the lower stages by default). With the fig leaf that the 4S-ops draft would fix up the safety story for the L4S design and implementation. And that is something I do not think to be a correct (and ops draft can not retroactively defang dualQ's mis-design (unless the recommendation were :never use dialQ")).
	The sad part is, all of this is not rocket science. For example requiring a guarding DSCP for the duration of the experiment to ascertain that only knowing and willing parties participate in the experiment would have beed relative easy to specify and would have been something where an OPs draft could shine, by giving the right points how to implement passing of said DSCP from participating to domain (e.g. SLAs, TCAs, ...) Alas, the draft did not, it did add however a few gratuitous paragraphs of what I call L4S-fan-fiction. I guess a possibility squandered.

Best Regards
	Sebastian

> On Mar 21, 2021, at 03:55, Gorry Fairhurst <[email-masked]> wrote:
> 
> I think people are well aware of your own thoughts. This email is part of a thread on the planned working group adoption of draft-white-tsvwg-l4sops.
> 
> Gorry

---

From: <[email-masked]>
Date: Mon, 22 Mar 2021 08:48:06 +0000

Hi Gorry,

I've read the draft and support adoption.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von Gorry Fairhurst
Gesendet: Dienstag, 9. März 2021 17:06
An: [email-masked]
Betreff: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude 24th March 2021

TSVWG,

Best wishes,
Gorry, Wes and David
(tsvwg co-chairs)

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 22 Mar 2021 08:54:25 +0000

Thanks for your thoughts. It's up to TSVWG to work on the 
specifications, this isn't about one team versus another. The "L4S Team" 
you mention, can develop implementations and encourage their use.

Gorry

P.S. I'll send an email on my own thoughts in a short while.

On 21/03/2021 08:54, Sebastian Moeller wrote:
> 	The sad part is, all of this is not rocket science. For example requiring a guarding DSCP for the duration of the experiment to ascertain that only knowing and willing parties participate in the experiment would have been relative easy to specify and would have been something where an OPs draft could shine, by giving the right points how to implement passing of said DSCP from participating to domain (e.g. SLAs, TCAs, ...) Alas, the draft did not, it did add however a few gratuitous paragraphs of what I call L4S-fan-fiction. I guess a possibility squandered.
>
>
> Best Regards
> 	Sebastian
>
>
>

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 22 Mar 2021 09:10:54 +0000

Hi Gorry,

I support adoption of that draft.
I have read -02 and will review future versions.

Best,
Olivier

 > -----Original Message-----
 > From: tsvwg <[email-masked]> On Behalf Of Gorry Fairhurst
 > Sent: Tuesday, March 9, 2021 5:06 PM
 > To: [email-masked]
 > Subject: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude
 > 24th March 2021
 > 
 > TSVWG,
 >

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 22 Mar 2021 10:28:47 +0100

Hi Gory,

> On Mar 22, 2021, at 09:54, Gorry Fairhurst <[email-masked]> wrote:
> 
> 
> Thanks for your thoughts. It's up to TSVWG to work on the specifications, 

	[SM] My initial thought also, but it turned out that the WG left the text and hence L4S' specification effectively to its authors. For example, the idea of putting the L4S experiment behind a mandatory DSCP guard was basically dropped as the authors of the drafts did not want to go there. IMHO that does not look like it is the WG that is in control of the specifications, sorry.

> this isn't about one team versus another.

	[SM] Again I agree that is how it should be, but for example the ECT(1) input or output meeting, was clearly a team sport, where instead of dscussion the relative merits of the two alternative uses of the ECT(1) codepoint, we voted for L4S versus SCE. 

> The "L4S Team" you mention, can develop implementations and encourage their use.

	[SM] Yes, but ATM the only known existing implementation of the specifications FAIL to meet those specifications. It is unclear to day whether that failure is caused by the specifications, the implementation or both. 
	"Ratifying" the specification any time soon, before the source of the failure is at least diagnosed and ideally fixed, seems premature to me. 
And I am puzzled that again we are having a procedural discussion instead of actually addressing the issues in design and/or implementation of the L4S proposals.

Regards
	Sebastian

P.S.: I note that nobody so far addressed the data demonstrating the hard failures of both the DUalQ AQM and TCP Prague, so I guess the good thing is that all of us seem to agree about their existence.

---

From: "Black, David" <[email-masked]>
Date: Mon, 22 Mar 2021 19:38:20 +0000

As an individual, I likewise support adoption.  Thanks, --David

From: tsvwg <[email-masked]> On Behalf Of Martin Duke
Sent: Friday, March 19, 2021 10:21 AM
To: Greg White
Cc: [email-masked]
Subject: Re: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude 24th March 2021

[EXTERNAL EMAIL]

On Mon, Mar 15, 2021 at 2:17 PM Greg White <[email-masked]<mailto:[email-masked]>> wrote:
Hi Jonathan,

-Greg

On 3/15/21, 4:35 AM, "tsvwg on behalf of Jonathan Morton" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

     - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 22 Mar 2021 21:40:30 +0000

Hi Sebastian,

I can take a crack at answering some of your questions, speaking only
for myself.

> 	Also, why people in this WG are willing to just ignore that fact instead of engaging in a discussion of how to meaningfully remedy that (so changing the actual L4S core technologies and not just w…

This has been under discussion for a long time, of course.  Just to
recap the way I remember it (slightly simplified to elide some side
points): the initial idea was floated during the "hallway discussion"
mentioned in the minutes at the tail end of the first tsvwg interim
last year [1] to make an "operational considerations" draft to explain
ways that operators could constrain the deployment of L4S to address
the safety issues, particularly from the seemingly intractable issue
#16 [2] about TCP Prague behavior in a shared classic queue.

The point was raised that there are many networks where it's known that
there are not in fact classic marking AQM queues (e.g. 5G networks when
sending traffic from embedded edge compute nodes or other path-aware
senders to in-network receivers, as well as some e.g. cable or fiber
networks that manage their users' wifi boxes), and that in such networks
for a constrained set of senders, issue #16 is not a hazard, and with the
associated knowledge about the bottlenecks, the latency benefits shown in
the early L4S lab experiments might be realizable.  (Likewise, they can
have pretty good expectations about whether there's dumb fifos at the
bottlenecks, if that's a concern.)

This was a fair point, and done properly could result in L4S+TCP Prague
deployments that do not cause damage to innocent bystanders.

Meanwhile, the L4S proponents raised a few well-founded objections to
SCE with ECT(1) as output, for instance that in a shared queue the
high fidelity congestion-responsive flows will necessarily be subjected
to latency spikes driven by classic flow competition.  This was borne
out in the SCE testing, e.g. in [3]--although it represents a substantial
overall improved latency profile relative to the classic flow without a
high-fidelity signal response, you can see the green spikes for the SCE
latency in the shared queue tests, and how this limits the potential to
stay within in a tight latency budget, even under favorable conditions.

As I understand it, this was the key driver behind the well-founded
points raised by several participants in the 3rd interim last year [4],
when they mentioned issues like the need for a classifier and traffic
segregation in order to achieve better latency than classic AQMs via
use of a high fidelity congestion signal from the network.

There remain some fair questions to ask here, most notably IMO the
same one you've raised, which I'll phrase as "is it appropriate to
assign ECT(1) to this when the only known safe deployments would be
known-path scenarios that could almost equally well use a local DSCP
assignment for classification?"  I think several others have tried to
ask roughly the same question, but it has so far been dismissed by
the authors for the reasons given in Appendix B.4 of l4s-id [5].

I'd expect this question and perhaps some others to see some more
discussion during tsvwg last call if not before then, and also
likely during IETF last call if it achieves rough consensus in
tsvwg, but I'm hoping the discussion will be sharpened somewhat if
we have a mature version of the operational considerations doc to
refer to, which is a big part of why I think it should be adopted.

To me the key conclusion at this stage is that the operational
considerations draft needs to share fate with the rest of the L4S
drafts.  This is true regardless of whether L4S ultimately gets
dropped, gets substantially changed, or achieves consensus and moves
forward as-is, since as it stands today, the operational considerations
is a key part of the safety story for avoiding damage to bystanders.
Given that the other L4S drafts are currently adopted, this means to
me that the operational considerations also needs to be in the same
state, IMO.

In the meantime, absent new data or new unconsidered proposals, there is
nothing new to say, so I don't see much point in going around in circles. 
(I want to note here however an exception for the comments raised by
Jonathan and Pete from re-analyzing the prior results in terms of harm
to bystanders and the consequences of our choices [6], which I think
does meaningfully contribute without going in circles, in spite of not
coming as a result of new observations or new proposed changes to the
protocols.  New methods of analysis with different or clearer insights
also can be helpful IMO, even without a new proposal or data.)

> b) The ops draft itself is also not ready. I gave detailed feed-back (at multiple stages in its development and ) and have (unfortunately) have read it multiple times. How about we first let this l…

I'm glad you're contributing feedback, thanks for doing that.  As I
understand it, adopting the draft in the wg just formalizes the work
as in-scope and establishes consensus that it's worth working on as a
group. There's a decent explanation of the procedural considerations in
RFC 7221.

But IMO, given that there are proponents looking to move forward with some
form of deployment of the technology in hopes of getting the benefits
shown in lab, it's important to document the precautions necessary to do
so safely in deployment, and as far as I can tell, this is currently the
intended role of the operational considerations draft.  And in that context,
as RFC 7221 puts it, the pragmatics to consider here are "Initial, not final",
and "Adoption, not approval", so while it's great that feedback is being
generated and hopefully considered carefully, it's not (IMO) helpful to put
off adopting a doc when we have a strong belief that other currently-adopted
docs can't be deployed safely at scale without considering the advice that's
in-scope for that doc.

(Note also that adoption is not a permanent commitment--see section 5.2 of
RFC 7221, which says that the WG can later decide that adopted docs are no
longer useful to keep as working group documents.  Presumably this will
eventually be the result if rough consensus cannot be reached and the
authors don't think it's worth addressing the objections raised to the
proposed final form of the docs, if enough people still hold substantive
objections at that time.)

>	All that said, my prediction is that we as a WG are going to accept the L4S drafts in autumn as is as experimental RFCs (as far as this WG can, I understand that we  will not have the last word, bu…

This has not been my experience with the IESG.  It's reasonably common
that they'll send a draft back to the working group or ask for further
revisions to a document.  There's many examples in which they've raised
detailed and significant objections to wg documents, e.g.:
https://datatracker.ietf.org/doc/draft-ietf-babel-source-specific/ballot/
https://datatracker.ietf.org/doc/draft-ietf-cdni-uri-signing/ballot/

>	The sad part is, all of this is not rocket science. For example requiring a guarding DSCP for the duration of the experiment to ascertain that only knowing and willing parties participate in the ex…

Since it's been stated several times by several people, the authors are
presumably aware that some number of people hold the opinion that more
discussion on using DSCP as a part of the classifier would be useful,
and if those people remain unconvinced they'll presumably state their
reservations during last call, and it will be considered during the
evaluation of rough consensus.

Again, here I'm hopeful that good progress on the operational
considerations draft will contribute to making that discussion more
productive than it would be otherwise, which is an important part of
why I support adopting it.

Not re-raising the same objections (even unaddressed objections)
when progress on another front happens is different from dropping the
objections.  I think there's reason to believe that there are still
several people with serious concerns about the safety (as they've
previously stated), and who have some reservations about the proposed
classifier solution.

Since there's a relatively confusing and painful email series across
many threads that got a lot of discussion in a lot of different
directions, I'm basically waiting for a set of docs that the authors
present as ready for another round of review (or for last call) to see
if the concerns have been addressed by then, since it's been so hard
to tell exactly what's on the table short of that.

I hope that's a helpful walkthrough of my reasons for supporting
adoption, and that it gives some insight toward answering your
questions.

Best regards,
Jake

[1] Interim 1 of last year:
    https://datatracker.ietf.org/doc/minutes-interim-2020-tsvwg-01-202002200900/
[2] Issue #16: https://trac.ietf.org/trac/tsvwg/ticket/16
[3] There are many great SCE lab measurements, but this one:
    http://sce.dnsmgr.net/results/ect1-2020-04-23-final/sce-s2-twoflow/sce-s2-twoflow-ns-cubic-vs-cubic-sce-twin_codel_af-20ms_tcp_delivery_with_rtt.svg
    is a good example, shown in the 20ms twin-codel-af from Scenario 2 of
    https://github.com/heistp/sce-l4s-ect1, demonstrating ~5+ ms regular
    additional latency spikes driven by the competing classic traffic.
[4] Interim 3 of last year:
    https://datatracker.ietf.org/doc/minutes-interim-2020-tsvwg-03-202004271000/
    Specifically in the section 4 "Discussion of slides" and the several
    people who said things about classifiers and the need for multiple queues.
[5] The documented objections to DSCP, from Appendix B.4 in l4s-id:
    https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-B.4
[6] Jonathan and Pete's slides from the end of tsvwg 110 day 1:
    https://datatracker.ietf.org/meeting/110/materials/slides-110-tsvwg-sessa-621-thoughts-on-the-ecn-deployment-observations-00#page=11
    This seems like a follow-up on Ware et.al's paper on harm as a metric,
    presented at the IRTF open meeting during IETF 109
    https://www.cs.cmu.edu/~rware/assets/pdf/ware-hotnets19.pdf

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 23 Mar 2021 01:07:53 +0200

> On 22 Mar, 2021, at 11:40 pm, Holland, Jake <jholland=[email-masked]> wrote:
> 

This is not an insurmountable problem, but one to discuss in another thread at another time.  It refers to a particular prototype qdisc which was deliberately designed to show reasonable coexistence between SCE and conventional traffic in a single queue - which it actually succeeds at.  More sophisticated approaches are possible, in particular FQ, or using Diffserv classification for that rare class of traffic that can't even tolerate 5ms of jitter.

Ultimately, though, SCE is intended not as an "ultra-low-latency at any cost" mechanism, but as a congestion control signal that strikes a better balance between throughput and latency, by eliminating the sawtooth oscillation inherent to AIMD and settling at a cwnd close to the actual BDP.  Any performance achieved beyond that basic standard is gravy - nice to have, but not considered essential.

It's actually strange to me that some people consider this nit a show-stopper while ignoring the horde of angry Japanese Hornets on their own turf.

> the authors for the reasons given in Appendix B.4 of l4s-id [5].

I believe those arguments do not apply to the scenario, so are invalid - they are predicated specifically on allowing end-to-end L4S over the global Internet.  But without some significant technical changes to L4S, that is simply not going to happen, and all such technical changes have been rejected outright by L4S draft authors.  Editorial changes, some of which have gone through, do not improve the technical content.

> me that the operational considerations also needs to be in the same

A reasonable argument, but…

> docs can't be deployed safely at scale without considering the advice that's

This presupposes that the draft is expected to be refined after adoption, which in turn requires that the author is receptive to constructive feedback.  If I had some assurance that my detailed feedback so far was being taken seriously by this draft's author, I might change my stance.  Since it has simply been dismissed without adequate explanation, objection to adoption is the only voice I have, and I won't apologise for using it.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Mon, 22 Mar 2021 20:56:05 -0400

Sec. 4 (Operator of a Network) of the draft presumes that deployed
equipment is capable to classifying packets specifically on ECT(1).
Have the authors confirmed that this feature is available on commonly
deployed operator gear (e.g., IOS-XR, JUNOS)?

Regards,

// Steve

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 24 Mar 2021 22:50:27 +0000

Steven,

> deployed operator gear (e.g., IOS-XR, JUNOS)?

[BB]
(Aside: I think you're reading an old (-01) draft. That section has been 
Sec. 5. since draft-02 on 22 Feb 2021.
See my response to the initial adoption call about the probable cause of 
this confusion - suspected problems with the IETF tools servers.
)

To your point, I checked the manuals of one or two OSs of common makes 
of router before I proposed the WRED technique for addition to the 
draft. And I discussed the hardware capabilities with people within one 
or two router vendors. In the cases I checked, the CLI limits the 
flexibility that the admin has to define classifiers as general bit 
patterns. However the hardware underneath does have that flexibility. So 
this would require a CLI update for the routers I checked. The Linux 
classifier architecture does provide sufficient flexibility for such a 
classifier.

I also suggested the ECT(1) tunnel bypass technique, but I didn't 
exhaustively check the manuals of all the different types of tunnel 
(there are dozens).

I think this list of techniques is most useful for router developers, 
who can then find the easiest and most efficient one for their 
particular kit; whether they have to update the CLI, or whether they can 
find a way for their users to configure their unmodified systems in the 
field.

Bob

---

From: Steven Blake <[email-masked]>
Date: Wed, 24 Mar 2021 19:12:56 -0400

> this confusion - suspected problems with the IETF tools servers.

Oops! You're right. s/Sec. 4/Sec. 5.

> find a way for their users to configure their unmodified systems in

So operators that *don't wish to participate in L4S experiments* may
need to update *their* deployed software? Ask your favorite router
vendor how many customer-specific releases they are maintaining because
customers don't want to move forward once they get a working validated
release.

Regards,

// Steve

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Mar 2021 09:26:48 +0000

Steven,

> customers don't want to move forward once they get a working validated

[BB] There is a common belief that, if any RFC3168 FIFO AQMs exist, they 
will be rare. But Jake and Jonathan raised the concern that it still 
needs to be possible to deploy RFC3168 routers from now onwards. In that 
case, operators that *don't wish to participate* would be updating their 
config, and l4sops then gives router developers ideas for how they might 
be able to prevent an existing implementation of RFC3168 from acting on 
ECT(1), given an ECN implementation is likely to be hard-coded against 
the ECN codepoints.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 25 Mar 2021 10:40:13 +0100

Hi Bob,

> On Mar 25, 2021, at 10:26, Bob Briscoe <[email-masked]> wrote:
> 
> Steven,
> 
> [BB] There is a common belief that, if any RFC3168 FIFO AQMs exist, they will be rare. But Jake and Jonathan raised the concern that it still needs to be possible to deploy RFC3168 routers from now onwards. In that case, operators that *don't wish to participate* would be updating their config, and l4sops then gives router developers ideas for how they might be able to prevent an existing implementation of RFC3168 from acting on ECT(1), given an ECN implementation is likely to be hard-coded against the ECN codepoints.

	[SM] This asks the question, how would an operator that is about to enable an rfc3168 AQM know that he better read and follow the L4S-ops ID/RFC? Are we expecting all operators to read and follow all RFCs meticulously all the time? 
	IMHO an operator intending on employing an rfc3168 AQM might read RFC3168 and RFCs referenced from there (which is IMHO already less likely), while an operator interested in L4S might read all of the L4S IDs/RFCs. But here we would need the rfc3168 deploying operators to read and follow an L4S ID/RFC...
	I guess adding an updated by to rfc3168 pointing to the L4S-ops RFC might offer a solution, but can/should a informational RFC update a PS document (honest question, I am just not sure about whether our process permits that)?

Best Regards
	Sebastian

P.S.: This is basically the same issue I have with the only mildly related NQB ID: in both contexts, we seem to expect parties genuinely not interested in the topic of the ID to act in a specific way to accommodate either the NQB or the L4S IDs/RFCs. And in both cases arguably bad things happen if those parties do not follow the recommendations.

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 25 Mar 2021 11:35:20 +0000

Sebastian.. 

Isn't the updated by RFC8311 sufficient in RFC3168 ?. It refers to L4S work,
namely the L4S ID which targets experimental standards status.
Also, to me it sounds odd to add an 
Updated by : [L4S Ops - Informational RFC] 
to a proposed standard ? , In any case it is the first time in RFC3168's
history that it happens, unless I missed something. 

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
> Sent: den 25 mars 2021 10:40
> Subject: Re: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to
conclude
> >>>> Have the authors confirmed that this feature is available on
> >>>> commonly deployed operator gear (e.g., IOS-XR, JUNOS)?
> >>> [BB]
> >>> (Aside: I think you're reading an old (-01) draft. That section has
> >>> been Sec. 5. since draft-02 on 22 Feb 2021.
> >>> See my response to the initial adoption call about the probable
> >>> cause of this confusion - suspected problems with the IETF tools
> >>> servers.
> >>> )
> >>> To your point, I checked the manuals of one or two OSs of common
> >>> makes of router before I proposed the WRED technique for addition to
> >>> the draft. And I discussed the hardware capabilities with people
> >>> within one or two router vendors. In the cases I checked, the CLI
> >>> limits the flexibility that the admin has to define classifiers as
> >>> general bit patterns. However the hardware underneath does have that
> >>> flexibility.
> >>> So
> >>> classifier architecture does provide sufficient flexibility for such
> >>> a classifier.
> >>>
> >>> I think this list of techniques is most useful for router
> >>> developers, who can then find the easiest and most efficient one for
> >>> their particular kit; whether they have to update the CLI, or
> >>> whether they can find a way for their users to configure their
> >>> unmodified systems in the field.
> >>
> >> vendor how many customer-specific releases they are maintaining
> >> because customers don't want to move forward once they get a working
> >> validated release.
> >
> will be rare. But Jake and Jonathan raised the concern that it still needs
to be
> possible to deploy RFC3168 routers from now onwards. In that case,
> operators that *don't wish to participate* would be updating their config,
> and l4sops then gives router developers ideas for how they might be able
to
> prevent an existing implementation of RFC3168 from acting on ECT(1), given
> an ECN implementation is likely to be hard-coded against the ECN
> codepoints.
> 
> 
> 	[SM] This asks the question, how would an operator that is about to
> enable an rfc3168 AQM know that he better read and follow the L4S-ops
> ID/RFC? Are we expecting all operators to read and follow all RFCs
> meticulously all the time?
> 	IMHO an operator intending on employing an rfc3168 AQM might
> read RFC3168 and RFCs referenced from there (which is IMHO already less
> likely), while an operator interested in L4S might read all of the L4S
IDs/RFCs.
> But here we would need the rfc3168 deploying operators to read and follow
> an L4S ID/RFC...
> 	I guess adding an updated by to rfc3168 pointing to the L4S-ops RFC
> might offer a solution, but can/should a informational RFC update a PS
> document (honest question, I am just not sure about whether our process
> permits that)?
> 
> Best Regards
> 	Sebastian
> 
> P.S.: This is basically the same issue I have with the only mildly related
NQB
> ID: in both contexts, we seem to expect parties genuinely not interested
in
> the topic of the ID to act in a specific way to accommodate either the NQB
or
> the L4S IDs/RFCs. And in both cases arguably bad things happen if those
> parties do not follow the recommendations.
> 
> 
> >
> >
> >
> > Bob
> >
> >>
> >>
> >> Regards,
> >>
> >> // Steve
> >>
> >>
> >>
> >>
> >
> > --
> >
> > Bob Briscoe
> https://protect2.fireeye.com/v1/url?k=532c5e52-0cb76757-532c1ec9-
> 86d2114eab2f-53274140f9ce9692&q=1&e=8a9723d1-2c84-4bf6-b7d7-
> b62af3457d9a&u=http%3A%2F%2Fbobbriscoe.net%2F

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 25 Mar 2021 12:45:04 +0100

Hi Ingemar,

thanks for responding.

> On Mar 25, 2021, at 12:35, Ingemar Johansson S <[email-masked]> wrote:
> 
> Sebastian.. 
> 

	[SM] IMHO not really, rfc8311 has no big warning signs, that those experimental standards are allowed/expected to carry negative side-effects for rfc3168-compliant AQMs and operators of rfc3168 AQM need to employ specific steps to ensure safety/functionality of their AQM to accommodate such experimental standards traffic.
	I am not trying to start a discussion about whether doing that at all is a good idea, but how to make sure that information reaches the parties that would need to follow those instructions. IMHO the link to rfc8311 does not convey enough urgency for rfc3168 deployers to go digging deeper, but I might be just naive here, not being/working for an operator.

> to a proposed standard ? ,

	[SM] Same sentiment, that why I asked. The information though seems important. If rfc3168 deployers need to do something extra to guarantee their safety and functionality, because of changes somewhere else, I believe the party responsible for those changes (aka this WG) should make sure that even casual reads of rfc3168 know about the additional steps we expect them to take.

Best Regards
	Sebastian

> In any case it is the first time in RFC3168's

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Mar 2021 12:19:17 +0000

Sebastian,

On 25/03/2021 09:40, Sebastian Moeller wrote:
>>>>> commonly
>>>> makes
>>>> a
>>>> developers,

[BB] An EXP can only update a PS in exceptional circumstances - called a 
downref. The only clue will be that RFC8311 updates RFC3168 to say that 
ECT(1) is available for experiments, and then the IANA assignment for 
ECT(1) should eventually point to ecn-l4s-id (and RFC8311 already 
references ecn-l4s-id). There has been talk of a further update to 
RFC3168 (or RFC8311) to formally deprecate RFC3168 AQMs marking ECT(1) 
packets.

I think you're imagining that the world should be easy to follow for 
companies that don't put effort into following the world. In practice, 
companies pay staff or consultants to brief them on what's going on in 
standards. And there are plenty of people in any networking company who 
will follow what's going on sufficiently to have an idea where to look. 
Nonetheless, you are right to be concerned, because many companies do a 
very poor job of tracking RFC updates, not helped by all the outdated 
tutorials on the web. That's why s**t happens.

For instance, try an image search for "IPv4 header" and nearly every 
highly ranked image still show the ToS field. Even the Wikipedia page 
about the IPv6 header still says it has a traffic class field, despite 
RFC3260 being 19 years old.

Nonetheless, your own efforts to create controversy around L4S have 
greatly helped to raise its profile in this respect :)

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 25 Mar 2021 14:26:13 +0200

> On 25 Mar, 2021, at 2:19 pm, Bob Briscoe <[email-masked]> wrote:
> 
> There has been talk of a further update to RFC3168 (or RFC8311) to formally deprecate RFC3168 AQMs marking ECT(1) packets.

I assume that would have to be by a PS RFC updating 3168, which would indeed provide a breadcrumb trail that could be followed.  However, I don't see very much support for that idea, so it's simply not going to happen in the immediate future.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 25 Mar 2021 13:08:04 +0000

Hi

One reflection around all this is that it really does not matter how many
warning signs you put, information may still not reach the intended
audience. 
I guess there must be an abundance of cases with other IETF work that does
not reach the audience, sometimes with suboptimal performance as a result.
And I don't find it too productive to dump a lot of informational text in
RFC8311 around what can possibly go wrong, especially as it is not fully
clear how serious these problems really are. Only way forward I see is to
move on with the L4S experiment and document possible issues as they come.
The L4S ops draft provide good initial input here and it will likely be
complemented with more best current practice.

/Ingemar

> From: Sebastian Moeller <[email-masked]>
> Sent: den 25 mars 2021 12:45
> To: Ingemar Johansson S <[email-masked]>
> Cc: Bob Briscoe <[email-masked]>; [email-masked]
conclude
> 24th March 2021
> 
> Hi Ingemar,
> 
> thanks for responding.
> 
> > On Mar 25, 2021, at 12:35, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > Sebastian..
> >
> > Isn't the updated by RFC8311 sufficient in RFC3168 ?. It refers to L4S
> > work, namely the L4S ID which targets experimental standards status.
> 
> 	[SM] IMHO not really, rfc8311 has no big warning signs, that those
> experimental standards are allowed/expected to carry negative side-effects
> for rfc3168-compliant AQMs and operators of rfc3168 AQM need to employ
> specific steps to ensure safety/functionality of their AQM to accommodate
> such experimental standards traffic.
> 	I am not trying to start a discussion about whether doing that at
all is
> a good idea, but how to make sure that information reaches the parties
that
> would need to follow those instructions. IMHO the link to rfc8311 does not
> convey enough urgency for rfc3168 deployers to go digging deeper, but I
> might be just naive here, not being/working for an operator.
> 
> > Updated by : [L4S Ops - Informational RFC] to a proposed standard ? ,
> 
> 	[SM] Same sentiment, that why I asked. The information though
> seems important. If rfc3168 deployers need to do something extra to
> guarantee their safety and functionality, because of changes somewhere
> else, I believe the party responsible for those changes (aka this WG)
should
> make sure that even casual reads of rfc3168 know about the additional
steps
> we expect them to take.
> 
> Best Regards
> 	Sebastian
> 
> 
> > In any case it is the first time in RFC3168's history that it happens,
> > unless I missed something.
> >
> > /Ingemar
> >
> >>>>>> Sec. 4 (Operator of a Network) of the draft presumes that
> >>>>>> deployed equipment is capable to classifying packets specifically
on
> ECT(1).
> >>>>> (Aside: I think you're reading an old (-01) draft. That section
> >>>>> has been Sec. 5. since draft-02 on 22 Feb 2021.
> >>>>> makes of router before I proposed the WRED technique for addition
> >>>>> to the draft. And I discussed the hardware capabilities with
> >>>>> people within one or two router vendors. In the cases I checked,
> >>>>> the CLI limits the flexibility that the admin has to define
> >>>>> classifiers as general bit patterns. However the hardware
> >>>>> underneath does have that flexibility.
> >>>>> So
> >>>>> this would require a CLI update for the routers I checked. The
> >>>>> Linux classifier architecture does provide sufficient flexibility
> >>>>> for such a classifier.
> >>>>>
> >>>>> exhaustively check the manuals of all the different types of
> >>>>> tunnel (there are dozens).
> >>>>>
> >>>>> developers, who can then find the easiest and most efficient one
> >>>>> for their particular kit; whether they have to update the CLI, or
> >>>> So operators that *don't wish to participate in L4S experiments*
> >>>> may need to update *their* deployed software? Ask your favorite
> >>>> router vendor how many customer-specific releases they are
> >>>> maintaining because customers don't want to move forward once they
> >>>> get a working validated release.
> >>>
> >> needs
> >> operators that *don't wish to participate* would be updating their
> >> config, and l4sops then gives router developers ideas for how they
> >> might be able
> > to
> >> prevent an existing implementation of RFC3168 from acting on ECT(1),
> >> given an ECN implementation is likely to be hard-coded against the
> >> ECN codepoints.
> >>
> >>
> read
> >> RFC3168 and RFCs referenced from there (which is IMHO already less
> >> But here we would need the rfc3168 deploying operators to read and
> >> follow an L4S ID/RFC...
> >> might offer a solution, but can/should a informational RFC update a
> >> PS document (honest question, I am just not sure about whether our
> >> process permits that)?
> >>
> >> Best Regards
> >> 	Sebastian
> >>
> >> the L4S IDs/RFCs. And in both cases arguably bad things happen if
> >> those parties do not follow the recommendations.
> >>
> >>
> >>>
> >>>
> >>>
> >>> Bob
> >>>
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> // Steve
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>>
> >>

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 25 Mar 2021 14:20:09 +0100

Hi Ingemar,

> On Mar 25, 2021, at 14:08, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	[SM] While potentially true, not really an actionable position to take... We can expect that parties interested in participating n the L4S experiment actually read the relevant L4S RFCs, but if we require others to also change their behavior/configurations/expectations we should take active steps to reach these parties, even if we can not guarantee to reach them all. In short we need to make a good faith effort. At least that is what common sense tells me, is there any relevant RFC instructing us to do otherwise?

> not reach the audience, sometimes with suboptimal performance as a result.

	[SM] We are actively contemplating of releasing protocols into the wide internet that run roughshod over rfc3168 and the expected sharing-between-flows properties of deployed AQM. Are you really advocating that we just do what pleases our whims right now, just because doing so is more convenient to us? I am puzzled, truly puzzled.

> clear how serious these problems really are.

	[SM] Fair enough, that is why I proposed to direct readers of rfc3168 to the L4S ops ID and leaving out the redirection via rfc8311. The point is, if we are going to encourage/permit end-points to behave in a way that violates the assumptions made by operators of rfc3168 AQMs the onus is on us to at least inform them about the fact, and what remedies we came up with.

> Only way forward I see is to

	[SM] Expect my, "I am running roughshod over L4S" ID any time soon, if all I need to do is to document issues... @chairs is this the official position of this WG and of the wider IETF, how to handle such conflicts of interest? I would be amazed if it would be...

Best Regards
	Sebastian

P.S.: I really tried to keep this focussed on how best to get the relevant information into the hands of potentially affected AQM operators; yet it turned again into a side-show about differences in opinion on how safe and sound engineering should be performed.

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 25 Mar 2021 15:34:50 +0200

> On 25 Mar, 2021, at 3:08 pm, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

Which, I think, just confirms that we have to consider what happens to operators and users who don't receive or act on that information.  How serious is the problem that is likely to result, how many people is it likely to affect, and are those people reasonably likely to be able to troubleshoot the problem and apply a suitable remedy?  Hence why I highlighted the principles of risk analysis in my talk.

That in turn is why it's much easier to standardise and deploy protocols that explicitly try to fit in with existing deployed specifications.  In that case, the answers to those questions are benign, and the protocol can be deployed without too much in the way of special safeguards - often, just an agreement to shut down experimental deployments if they prove to cause unexpected problems.

Unfortunately, the answers where L4S are concerned are *not* benign, because it is not designed to interoperate with a certain class of existing deployed infrastructure.  It therefore requires much more robust precautions surrounding any deployment - precautions which l4s-ops does not presently describe.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 25 Mar 2021 14:56:15 +0000

> I assume that would have to be by a PS RFC updating 3168, which would indeed provide a breadcrumb trail that could be followed.  However, I don't see very much support for that idea, so it's simply…

Just reading this: Yes. If the L4S deployment experiment results in 
widescale deployment and then it seems an update to a PS is the correct 
outcome, the WG can certainly consider this.

Gorry

---

From: "Livingood, Jason" <[email-masked]>
Date: Thu, 25 Mar 2021 14:58:40 +0000

[JL] Agree and in my day job we're keen to start more real-world L4S experiments. We already have identified some interested partners but if you have not yet spoken with me about it and are potentially interested, please hit me up 1:1 off-list.

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 25 Mar 2021 16:14:58 +0100

Hi Jason,

just to ridicule me, what kind of L4S experiments are you keen to start? Will you share the data and inform the WG about the nature and outcome of said experiments?
What mechanisms do you intend to employ to restrict the experiment to consenting adults^W^W knowing and willing partners?

Best Regards
	Sebastian

> On Mar 25, 2021, at 15:58, Livingood, Jason <Jason_Livingood=[email-masked]> wrote:
>

---

From: "Livingood, Jason" <[email-masked]>
Date: Thu, 25 Mar 2021 16:53:22 +0000

On 3/25/21, 11:16 AM, "tsvwg on behalf of Sebastian Moeller" <[email-masked] on behalf of [email-masked]> wrote:
>    just to ridicule me, what kind of L4S experiments are you keen to start?

[JL] Testing L4S on an E2E basis from the WLAN-based client all the way to the app provider on the other end. No particular details defined other than that - this is pretty early stage stuff but seems like the next logical step for this type of technology.

>Will you share the data and inform the WG about the nature and outcome of said experiments?

[JL] I would think so - we have previously done so with DNSSEC, IPv6, DoH, etc.

> What mechanisms do you intend to employ to restrict the experiment to consenting adults^W^W knowing and willing partners?

[JL] TBD as plans are worked out. But companies that operate infrastructure at scale are naturally focused on risk management of new technology testing & deployment, so I don't think there is cause for alarm in your part if that is what you are getting at.

[JL] BTW, if you know of anyone willing to develop code changes to AQM implementations in OpenWrt to handle ECT1-marked packets according to the L4S semantics, let me know as that seems something worth technically exploring as a related matter (noting that I chair a grants fund at my work that does underwrite open source dev and other applied research).

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 25 Mar 2021 20:41:50 +0100

Hi Jason,

thanks.

> On Mar 25, 2021, at 17:53, Livingood, Jason <[email-masked]> wrote:
> 
> [JL] Testing L4S on an E2E basis from the WLAN-based client all the way to the app provider on the other end. No particular details defined other than that -

	[SM] Given that you work for a big ISP that has many industry partners, certainly also in the content business, why do you need an RFC status to start such experiments, especially, if, as you indicate below, you know how to isolate such experiments from causing harm? If you ask me these two claims do not add up fully yet, sorry. 

> this is pretty early stage stuff but seems like the next logical step for this type of technology.

	[SM] I had expected there to be somewhat more precise plans with clearer hypothesis to test. I also assumed, that the DOCSIS world would start mass deployment of L4S network bits (under the name of low latency docsis) the moment the IETF ratifies the L4S RFCs, given for how long LLD is already advertised and marketed. 
	But I am happy if I a plainly wrong and the cable industry is planning a careful, experiment-driven staged test and deployment instead. Let's see which of the two alternatives will play out. ;)

> [JL] I would think so - we have previously done so with DNSSEC, IPv6, DoH, etc.

	[SM] Excellent, I welcome that. After asking here in the list for some time now, that parties that support L4S show which data convinced them to do so, I am looking forward to that data, whenever it will be ready.

> [JL] TBD as plans are worked out. But companies that operate infrastructure at scale are naturally focused on risk management of new technology testing & deployment, so I don't think there is cause…

	[SM] Mmmh appeal to authority? Really, I am willing to trust you personally, and do not expect malice here. But that does not fill me with confidence that your containment plans are ready for action tomorrow.

> [JL] BTW, if you know of anyone willing to develop code changes to AQM implementations in OpenWrt to handle ECT1-marked packets according to the L4S semantics, let me know as that seems something w…

	[SM] In all honesty, thanks, but before I convince anybody to work on that, I want to see data first that this is a good idea to do in the first place. Especially, is that change safe, and does this actually realize improvements for more than L4S' design sweet spot, the short RTT low-hop-count fast lane between eyeballs and near-by content. 
	Don't get me wrong I will not stop anybody doing that, but I will not actively work towards that goal until and unless I am convinced that this offers an improvement for the whole internet. And I am not interested in expediting/speeding up short RTT flow at the expense of longer RTT flows, but that is what L4S currently does.

Sidenote:, the best way of getting any capability into OpenWrt is to have your feature accepted and included into the mainline Linux kernel, then it will percolate into OpenWrt automatically.

Best Regards
	Sebastian

---

From: Steven Blake <[email-masked]>
Date: Thu, 25 Mar 2021 22:22:12 -0400

On Thu, 2021-03-25 at 09:26 +0000, Bob Briscoe wrote:

> might 

RFC-3168 ECN is a feature that router vendors have been shipping for
about two decades. Unless and until the IETF deprecates RFC-3168 (or
vendors EOL the feature), it should remain perfectly safe for operators
to enable it without having to track whatever experimental RFCs IETF
decides to publish, or without having to install patch releases.

All IMHO.

Regards,

// Steve

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 10:23:55 +0100

Hi Bob,

did not see this yesterday, so my belated response, prefixed [SM2]

> On Mar 25, 2021, at 13:19, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> [BB] An EXP can only update a PS in exceptional circumstances - called a downref.

	[SM2] Thanks I appreciate this process related information a lot. I am not hung up on the exact placement of a reference, as long as the intended audience will stumble upon it without having to do much reasearch, because that reasearch is not guaranteed to be invested.

> The only clue will be that RFC8311 updates RFC3168 to say that ECT(1) is available for experiments,

	[SM2] That is too subtle a clue for anybody wanting to deploy rfc3168 AQMs in their domain, I think...

> and then the IANA assignment for ECT(1) should eventually point to ecn-l4s-id (and RFC8311 already references ecn-l4s-id). There has been talk of a further update to RFC3168 (or RFC8311) to formally deprecate RFC3168 AQMs marking ECT(1) packets.

	[SM2] Which might only happen after the L4S experiment concludes successfully I would say; but the remedies/work-arounds of the L4S-ops draft seem to be important even while the experiment proceeds (unless we can make sure that the L4S experiment is inherently safe, e.g. by using a guard DSCP, in that case it suffices if networks actively participating in the L4S experiment read and heed the L4S-ops draft.). I do have my doubts about the effectiveness of requiring non-participating parties to change their ways, but if we as a WG beliefe this to be the way, I just want to make sure that connecting the relevant parties with the L4S-ops draft is as explicit and visible as possible (while still respecting IEF process). What I want to avoid is a situation like Douglas Adams described:

"“But the plans were on display…”
“On display? I eventually had to go down to the cellar to find them.”
“That’s the display department.”
“With a flashlight.”
“Ah, well, the lights had probably gone.”
“So had the stairs.”
“But look, you found the notice, didn’t you?”
“Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”"

> 
> 
> I think you're imagining that the world should be easy to follow for companies that don't put effort into following the world.

	[SM2] No, I envision a world where IFF rules and regulations get changed, a realistic effort is made to inform potentially affected parties of those changes. And here we already stipulate that someone is looking at rfc3168 already, so doing their due dilligence, IMHO.

> In practice, companies pay staff or consultants to brief them on what's going on in standards. And there are plenty of people in any networking company who will follow what's going on sufficiently to have an idea where to look. 

	[SM2] If that works great! But as I said, when made decisions regarding the default use of ECN in sqm-scripts (affecting all SQM users not actively changing the defaults), I did not go on an extended RFC reading session before (I had read rfc3168 before, but simply assumed ECN to be deployable as is). Yes, that was my sub-optimal action, not one of a bigger layer, but I think this still anecdotally illustrates how important it is to make these things as visible as possible.

> Nonetheless, you are right to be concerned, because many companies do a very poor job of tracking RFC updates, not helped by all the outdated tutorials on the web. That's why s**t happens.

	[SM2] Realistically, changing an existing RFC post-hoc, forces an outreach effort on the party changing the RFC, not because that is fair or just, but simply because they are the only ones that really know...

> 
> For instance, try an image search for "IPv4 header" and nearly every highly ranked image still show the ToS field. Even the Wikipedia page about the IPv6 header still says it has a traffic class field, despite RFC3260 being 19 years old.

	[SM] Good example. It supports my point above that outreach is necessary, as a change in an RFC != change in behavior of deployed instances based on that RFC...

> 
> Nonetheless, your own efforts to create controversy around L4S have greatly helped to raise its profile in this respect :)

	[SM2] I would not call it "creat[ing] controversy", but rather "emphasizing short-comings and pointing out where improvements seem advised".

Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 26 Mar 2021 15:46:59 +0000

Sorry, missed the deadline, but I also support adoption of draft-white-tsvwg-l4sops and will further review and contribute text.

Regards,
Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Gorry Fairhurst
Sent: Tuesday, March 9, 2021 5:06 PM
To: [email-masked]
Subject: [tsvwg] Adoption call for draft-white-tsvwg-l4sops - to conclude 24th March 2021

TSVWG,

Best wishes,
Gorry, Wes and David
(tsvwg co-chairs)