Subject: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

From: "Black, David" <[email-masked]>
Date: Mon, 22 Mar 2021 19:30:43 +0000

Following up on Pete's comment:

> Along these lines, there is nothing stopping anyone from using DSCP (as
> an appropriate risk control within participating ASs), to perform
> experiments to reproduce issues that can be demonstrated in the lab.

That is a commendable approach, particularly as it is effectively recommended by RFC 4774 (Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field) - see Section 3 [1], and take note that RFC 4774 is a BCP (Best Current Practice) whose author was the late Sally Floyd.  It may be time for the WG to reconsider the L4S approach of using alternate ECN semantics without qualification by DSCP, at least to run experiments on actual networks.

My 0.02 is that an Experimental RFC that used a DSCP to signal the L4S ECN semantics (in participating networks) could have been published at least 2 years ago.
 
Thanks, --David (as a TSVWG co-chair, but not claiming to speak for other co-chairs)
[1] https://datatracker.ietf.org/doc/html/rfc4774#section-3

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Pete Heist
Sent: Monday, March 22, 2021 4:04 AM
To: Livingood, Jason
Cc: [email-masked]
Subject: Re: [tsvwg] L4S drafts: Next Steps

[EXTERNAL EMAIL] 

Hi Jason, one comment below...

On Sat, 2021-03-20 at 15:08 +0000, Livingood, Jason wrote:
> > The lab studies have shown
> 
> IMO the challenge with lab studies is that there are a lot of variables
> that are artificial and/or the test environment or test traffic does
> not fully reflect reality (production). 10 or 15 years ago I would have
> been very focused on extensive QA testing and an elaborate lab test
> environment. These days its more how to do controlled testing directly
> in the production network, with a controlled/minimized blast radius in
> case of problems, easy/quick rollback, rapid code/config iteration.
> 
> > far from a promising system that might be ready for a field trial,
> > one that shows worrying failure modes that primarily affect innocent
> > bystanders.
> >  What we're asking for is a system that at least behaves reasonably
> > under lab conditions, chosen to represent realistic challenge
> > scenarios likely to occur in the real world.  Only then can we have
> > any confidence that L4S will not cause problems anywhere and
> > everywhere that it is deployed.
> 
> Only one way to find out - test it in a real production experiment with
> appropriate risk controls.

Along these lines, there is nothing stopping anyone from using DSCP (as
an appropriate risk control within participating ASs), to perform
experiments to reproduce issues that can be demonstrated in the lab.
Here are a few: https://github.com/heistp/l4s-tests/

It would be nice if someone took the time to set up some production
gear with fq_codel built into it, like routers and/or WiFi access
points, and test various kinds of real-world traffic, through tunnels
and not. One could combine traffic to L4S servers within the
experimental network, along with conventional traffic to anywhere on
the Internet. That seems like a useful step before starting an
Internet-wide experiment. Perhaps even better would be to fix what
safety issues are known so that they can't occur in the first place.

Regards,
Pete

>  The status quo seems to suggest years more debate without (IMO)
> sufficient data. An alternative is a controlled production experiment,
> appropriately risk-managed, between willing participants. I see no
> downside to allowing that to occur via experimental RFC. We're setting
> the bar at the proposed standard or standard level, which I think is
> incorrect. I would say instead enable the experiment to accelerate the
> learning & improvement & drive data-driven decisions.
> 
> Jason
>

---

From: Steven Blake <[email-masked]>
Date: Wed, 24 Mar 2021 01:52:55 -0400

On Mon, 2021-03-22 at 19:30 +0000, Black, David wrote:
> That is a commendable approach, particularly as it is effectively
> recommended by RFC 4774 (Specifying Alternate Semantics for the
> Explicit Congestion Notification (ECN) Field) - see Section 3 [1],
> and take note that RFC 4774 is a BCP (Best Current Practice) whose
> author was the late Sally Floyd.  It may be time for the WG to
> reconsider the L4S approach of using alternate ECN semantics without
> qualification by DSCP, at least to run experiments on actual
> networks.
> 
> My 0.02 is that an Experimental RFC that used a DSCP to signal the
> L4S ECN semantics (in participating networks) could have been
> published at least 2 years ago.

I agree wholeheartedly (as I have said multiple times in the past).

As it stands, the -l4ops draft puts a burden on operators who are
minding their own business to evaluate and potentially make operational
changes to their network to avoid potential disruptions to their
network performance, for the benefit of someone else's experiment. That
is not appropriate IMHO nor is it necessary when an alternative (DSCP
isolation) is available.

Regards,

// Steve

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 24 Mar 2021 07:34:47 +0000

Hi Steve, David + others

I believe that it is good to refer back to 
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14#appendix-B.4 
before taking this discussion further.

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Steven Blake
> Sent: den 24 mars 2021 06:53
> To: Black, David <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)
>
> > > Along these lines, there is nothing stopping anyone from using DSCP
> > > (as an appropriate risk control within participating ASs), to
> > > perform experiments to reproduce issues that can be demonstrated in
> > > the lab.
> >
> > Explicit Congestion Notification (ECN) Field) - see Section 3 [1], and
> > take note that RFC 4774 is a BCP (Best Current Practice) whose author
> > was the late Sally Floyd.  It may be time for the WG to reconsider the
> > L4S approach of using alternate ECN semantics without qualification by
> > DSCP, at least to run experiments on actual networks.
> >
> > My 0.02 is that an Experimental RFC that used a DSCP to signal the L4S
> > ECN semantics (in participating networks) could have been published at
> > least 2 years ago.
>
> As it stands, the -l4ops draft puts a burden on operators who are minding
> their own business to evaluate and potentially make operational changes to
> their network to avoid potential disruptions to their network performance,
> for the benefit of someone else's experiment. That is not appropriate IMHO
> nor is it necessary when an alternative (DSCP
> isolation) is available.
>
>
> Regards,
>
> // Steve
>
>
>

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 24 Mar 2021 09:41:38 +0200

> On 24 Mar, 2021, at 9:34 am, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

We're all familiar with that.  It was clearly written, however, in the context of deploying L4S end-to-end across the global Internet, which in its current form is not feasible.  In the context of a controlled and confined experiment, and with L4S' designed-in preference for short paths, at least half the arguments given there simply go away.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Mar 2021 08:28:13 +0000

I agree Ingemar, 

This really  is not a new topic, and the WG should only be revisiting this if there is a new requirement - the design to allow the traffic to be DSCP-marked was not taken without thought and  visibility.

... If an operator wishes to do this internally, that’s fine and they can use a private DSCP to do just that.

Gorry

> On 24 Mar 2021, at 07:35, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi Steve, David + others
>

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 24 Mar 2021 10:12:21 +0000

Hi Gorry,

At the time of that design decision, we were still promised that
a solution to the 3168 interop problem would be forthcoming, but
that seems to have fallen through as not tractably implementable
and is now proposed to be handled chiefly by operational
considerations, in light of the poor performance of the proposed
algorithms in lab testing.

That change in scope over the course of that discovery merits a
re-evaluation of the design decisions that compromise on safety
considerations, IMO.

(Though it's still listed as a requirement in A.1.4 of l4s-id
and references provided as though it's a reasonably well-solved
problem, so maybe I'm misunderstanding the outcome of some of
the discussions since November 2019?  I'm not clear on whether an
update is planned to that section...)

Anyway, I believe we saw a call or 2 for further discussion of this
question in comments from several people in the last year.

I'll agree this is not a new topic, and that it would be great if
anyone who wants to comment would read through B.4 of l4s-id to
refresh their memory and spend some effort speaking to the points it
raises, but I also think there's a fair bit of unexplored discussion
space here that's worth re-opening if the "fallback on classic ecn"
requirement is to be dropped or weakened.

Best regards,
Jake

On 3/24/21, 1:29 AM, "Gorry Fairhurst" <[email-masked]> wrote:

I agree Ingemar, 

Gorry

> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-14*appendix-B.4__;Iw!!GjvTz_vk!He0Lu9NS80YwQ0QKP_9uL9JSDpboI_ZjeH98P87VfX06JeFArdwFisa8UcOXx0U$

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 24 Mar 2021 12:17:44 +0200

> On 24 Mar, 2021, at 10:28 am, Gorry Fairhurst <[email-masked]> wrote:
> 

I agree that it's not a new topic.  However, it has renewed relevance with the introduction of the l4s-ops draft, since L4S has failed to adequately address the open issues which prevent its being deployed as an Internet-wide experiment (which is the predicate for many of the arguments against DSCP in B.4).

Since David Black brought up RFC-4774 (a remarkably prescient document), I will note here that it divides alternate-ECN proposals into three possible categories:

Option 1: cannot be deployed on the Internet.  This clearly describes DCTCP, for example.

Option 2: can reliably detect whether it is receiving RFC-3168 or "alternate" signals and switch modes gracefully.

Option 3: is inherently compatible with RFC-3168 traffic and infrastructure.

There is no Option 4 contemplated; certainly none where harm to or required action by innocent bystanders is accepted.

I appreciate that L4S intended to improve DCTCP sufficiently to merit an Option 2 classification.  However, it has clearly failed to do so to date, despite man-years of effort from its proponents and many hours of WG meeting time expounding its presumed benefits (while ignoring warnings of its downsides).  This leaves L4S as an Option 1 proposal, though with less restricted applicability than DCTCP since it does at least carry an identifier; it can only safely be deployed on networks that have been prepared for it.

This means a containment mechanism is required, and DSCP is perfect for the job.  The only alternative I see is to write l4s-ops - or include language in the other drafts - such that it requires deployment only in similarly isolated circumstances as DCTCP.  The latter option would be unpalatable, I'm sure, but people really have painted themselves into a corner on this issue.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 24 Mar 2021 11:58:47 +0100

Hi Gory,

I think these are different issues. To my knowledge the WG never discussed using DSCPs to at least guard L4S signaling DURING the experiment, 

To recap:

   Cons:

   Consumes DSCP pairs:  A DSCP is by definition not orthogonal to
      Diffserv.  Therefore, wherever the L4S service is applied to
      multiple Diffserv scheduling behaviours, it would be necessary to
      replace each DSCP with a pair of DSCPs.

Given that during the experiment we expect willingly participating networks that actually are prepared to measure and monitor L4S behavior, employing two DSCPs will not be a blocker.

   Uses critical lower-layer header space:  

Seems to apply to ECT(1) as well.

   Not end-to-end (host-network): 
and
   Not end-to-end (network-network):  

In the context of the experiment this is exactly part of what makes DSCPs a useful safety mechanism, as we can hope for little leakage of L4S signalling unless all participating nodes have been made L4S-aware, exactly what seems appropriate for an experimental roll-out, no?

      Local-use DSCP:  
and
      Recommended standard DSCP:  

Are not relevant at all, and seem to indicate a confusion between PHB and DSCP on the part of the writers of said paragraphs.

   Not all tunnels:  

Again for an experiment, it seems a big enough task to start out with not handling all tunnels. After all L4S is building a speed-way between near data sources and data consumers so the amount of to-be-expected tunneling will be relative small. And in all honestly, the not-all-tunnels argument ALSO applies to ECT(1), so not a unique con.

   ECN hard in some lower layers::  

Again, the primary roll-out of L4S AQM will probably be close to the edges, that is close to data sources and data destinations, it is IMHO quite likely that both ends "speak" mostly IP so this con should not stop initial experiments.

   Hard to distinguish Classic ECN AQM:  
In the scope of the experiment with willingly participating networks, that could simply be ruled out a-priori and hence should be a no issue for an experiment.

   Could migrate to e2e:  

This is not even a cogent con of using DSCP as identifier. I propose that we delete that from the draft completely.

IMHO this section of the draft is not at odds with requiring that an L4S experimental roll-out is guarded behind a DSCP to make sure that only parties that know what they are getting themselves into will participate. In fact a number of these "cons" are IMHO actual "pros" for requiring a DSCP.

Best Regards
	Sebastian

> On Mar 24, 2021, at 09:28, Gorry Fairhurst <[email-masked]> wrote:
> 
> I agree Ingemar, 
>

---

From: Pete Heist <[email-masked]>
Date: Wed, 24 Mar 2021 12:01:44 +0100

I'll just add to the sentiment that I think the use of DSCP is still
worthy of consideration. Beyond the use of a single DSCP on all
traffic, there may be other alternatives that address at least some of
the concerns in B.4.

Pete

On Wed, 2021-03-24 at 12:17 +0200, Jonathan Morton wrote:
> > This really  is not a new topic, and the WG should only be
> > revisiting this if there is a new requirement - the design to allow
> > the traffic to be DSCP-marked was not taken without thought and 
> > visibility.
> 
> I agree that it's not a new topic.  However, it has renewed relevance
> with the introduction of the l4s-ops draft, since L4S has failed to
> adequately address the open issues which prevent its being deployed
> as an Internet-wide experiment (which is the predicate for many of
> the arguments against DSCP in B.4).
> 
> Since David Black brought up RFC-4774 (a remarkably prescient
> document), I will note here that it divides alternate-ECN proposals
> into three possible categories:
> 
> Option 2: can reliably detect whether it is receiving RFC-3168 or
> "alternate" signals and switch modes gracefully.
> 
> There is no Option 4 contemplated; certainly none where harm to or
> required action by innocent bystanders is accepted.
> 
> I appreciate that L4S intended to improve DCTCP sufficiently to merit
> an Option 2 classification.  However, it has clearly failed to do so
> to date, despite man-years of effort from its proponents and many
> hours of WG meeting time expounding its presumed benefits (while
> ignoring warnings of its downsides).  This leaves L4S as an Option 1
> proposal, though with less restricted applicability than DCTCP since
> it does at least carry an identifier; it can only safely be deployed
> on networks that have been prepared for it.
> 
> This means a containment mechanism is required, and DSCP is perfect
> for the job.  The only alternative I see is to write l4s-ops - or
> include language in the other drafts - such that it requires
> deployment only in similarly isolated circumstances as DCTCP.  The
> latter option would be unpalatable, I'm sure, but people really have
> painted themselves into a corner on this issue.
> 
>  - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Mar 2021 12:46:33 +0000

So injecting a little more history here.

Some previous discussions are:

The earliest discussion of L4S in the IETF that I recall was in 2015. 
When discussing, it was known that RFC4774 would allow experimentation 
with a diffserv domain (e.g., as in RFC 6660), but the practicalities of 
deploying an end-to-end Internet transport required an RFC to use a 
method that was not protected by a DSCP. The discussion on whether the 
ECN Plus a Diffserv Codepoint (DSCP) was also noted in the appendix of 
draft-briscoe-tsvwg-ecn-l4s-id-00 in 2015.  R

Tradeoffs that lead to using ECT(1) were described in the proposal to 
the AQM working group in draft-briscoe-aqm-dualq-coupled-00 in 2015.

The interaction with DSCPs was revisited in 2018 
(draft-briscoe-tsvwg-l4s-diffserv-00), after RFC8311 enabled such a 
deployment experiment using ECT(1).  That which explains how the two 
approaches interact, how they can be arranged to complement each other 
and in which cases one can stand alone without needing the other.

Gorry

---

From: Kyle Rose <[email-masked]>
Date: Wed, 24 Mar 2021 09:11:38 -0400

On Wed, Mar 24, 2021 at 8:46 AM Gorry Fairhurst <[email-masked]>
wrote:

> method that was not protected by a DSCP.

Understood. But I think the point that several folks have been making is
that an experimental RFC based solely on DSCP could have been published,
and opt-in experiments within and across guinea pig networks could have
begun, years ago. By now we'd have real world data about the behaviors of
L4S and TCP Prague that would likely have clarified many of the
long-running unknowns (e.g., regarding RTT sensitivity, behavior in the
presence of classic traffic, etc.) that are unlikely to be resolved by
simply thinking hard about the problem. Instead we've spun on a single
question, the allocation of an internet-wide code point to L4S, for several
years during which we could have been doing the work and gathering the data
required for understanding the likelihood of success in making that
allocation. Until we have real world data, those questions remain
unanswered and consensus cannot be achieved.

Kyle

---

From: "Black, David" <[email-masked]>
Date: Wed, 24 Mar 2021 13:53:35 +0000

> and in which cases one can stand alone without needing the other.

My recollection is that this draft was not taken forward due to lack of interest at the time.

Thanks, --David

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Gorry Fairhurst
Sent: Wednesday, March 24, 2021 8:47 AM
To: Pete Heist; [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

[EXTERNAL EMAIL] 

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Mar 2021 13:57:26 +0000

> My recollection is that this draft was not taken forward due to lack of interest at the time.

True, the discussion of Diffserv goes has to date gone in cycles - with 
few (if any) operators stepping-up and saying that they would like to 
deploy L4S with a DSCP.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 24 Mar 2021 15:19:32 +0100

Hi Gory,

> On Mar 24, 2021, at 14:57, Gorry Fairhurst <[email-masked]> wrote:
> 
> True, the discussion of Diffserv goes has to date gone in cycles - with few (if any) operators stepping-up and saying that they would like to deploy L4S with a DSCP.

	[SM] Could you clarify please, if there is zero interest by operators to participate in the L4S experiment if DSCPs will be required, or requiring DSCPs will be a (potential) hurdle for an internet wide deployment of L4S? 
	Side-question, if you are asked between two options with the same benefit for you, which would you select the one requiring more effort to implement or the easier one? In other words is relative convenience of deployment for operators really the ideal yard-stick for selecting safety mechanisms?
	As much as I personally dislike the overloaded ECT(1)/CE design of L4S, for running an experiment, guarding the current design with an additional layer of DSCP to assure that unwitting bystanders are not affected by the experiment, seems like a way forward to starting an experiment, without completely re-opening the "classifier" question.
	And if I understand the L4S position correctly, all reasonable tests that do not require real networks have been exhausted, and hence further testing needs to be performed on lie networks, and a guard DSCP would allow exactly that.
	What a guard DSCP would not easily allow is internet-wide roll-out of the L4S network bits, but given that the experiment has not been run yet and hence no feed-back from the acquired data back into design and implementation has been possible, such a deployment would seem probably a bit premature today, no? 

Best Regards
	Sebastian

---

From: Pete Heist <[email-masked]>
Date: Wed, 24 Mar 2021 15:38:54 +0100

Thanks, draft-briscoe-tsvwg-l4s-diffserv is one that I hadn't seen
before. It seems to discuss more the general implications of using DSCP
in combination with the proposed ECT(1) as L4S-ID classifier, rather
than using DSCP as a way to address the safety concerns that result
from the ambiguous definition of CE that ECT(1) as L4S-ID introduces.

I did notice that section 7.2 of this draft might become less
complicated if the L4S-ID were a DSCP, but that's an aside really.

Pete

---

From: <[email-masked]>
Date: Wed, 24 Mar 2021 15:04:20 +0000

Hi Gorry,

My take was that a CDN or content owner might be interested in L4S and start experiments without asking operators for co-operation. Then innocent bystanders may get hit and a specified DSCP might allow for protection.

An operator testing L4S is likely able to find an environment where L4S doesn't create harm. A DSCP isn't required for such an experiment and if the operator desires to use one, standardization isn't a requirement, I guess.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von Gorry Fairhurst
Gesendet: Mittwoch, 24. März 2021 14:57
An: Black, David <[email-masked]>; Pete Heist <[email-masked]>; [email-masked]
Betreff: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

On 24/03/2021 13:53, Black, David wrote:
>> approaches interact, how they can be arranged to complement each 
>> other and in which cases one can stand alone without needing the other.

Gorry

> with a diffserv domain (e.g., as in RFC 6660), but the practicalities 
> of deploying an end-to-end Internet transport required an RFC to use a 
>> traffic, there may be other alternatives that address at least some 
>> of the concerns in B.4.
>>
>> Pete

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Mar 2021 15:33:19 +0000

> Hi Gorry,
Hi
> My take was that a CDN or content owner might be interested in L4S and start experiments without asking operators for co-operation.

Sure. So in this case, bottlenecks in that operator network could then 
receive ECT(1) traffic and not expect this.

> Then innocent bystanders may get hit and a specified DSCP might allow for protection.

The bystanders are those impacted by a "CE-marking from a non-L4S 
router" that is congested, but where the other traffic responds using L4S.

Is this any different to the first operator not forwarding the ECT(1) 
codepoint?

I do wonder if this proves a real problem whether the community might 
deprecate CE-marking of ECT(0) in non-L4S devices, I wonder in future 
how much hardship this would cause, especially if L4S were to be widely 
deployed.

> An operator testing L4S is likely able to find an environment where L4S doesn't create harm. A DSCP isn't required for such an experiment and if the operator desires to use one, standardization isn…
True, if an operator really wanted to use a DSCP, they could use a 
private one. I think that operator would be better likely to decide to 
differentiate on the ECN codepoint.
> Ruediger

Thanks,

Gorry

---

From: <[email-masked]>
Date: Wed, 24 Mar 2021 16:05:22 +0000

Hi Gorry,

One detail only, otherwise we are in line otherwise, I think

> Then innocent bystanders may get hit and a specified DSCP might allow for protection.

[GF] The bystanders are those impacted by a "CE-marking from a non-L4S router" that is congested, but where the other traffic responds using L4S.

[GF] Is this any different to the first operator not forwarding the ECT(1) codepoint?

[GF] I do wonder if this proves a real problem whether the community might deprecate CE-marking of ECT(0) in non-L4S devices, I wonder in future how much hardship this would cause, especially if L4S were to be widely deployed.

[RG] In the case of an operator sending L4S traffic, the source of a "CE-marking from a non-L4S router" must be upstream of the terminal. That is likely a policy point of the operator or an external network. The former doesn't apply and I doubt that the latter is to be expected. I'd expect the terminals to be deployed for some experiment (and users are either operator employees or friendly volunteers), so the contact with external networks sending "CE-marking from a non-L4S router" might be rare (I'd wonder about that - where's the bottleneck, a peering interface?).

Regards,

Ruediger

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Mar 2021 16:06:31 +0000

See below,

>>>> traffic, there may be other alternatives that address at least some of
The people advocating for L4S have asked for something different.

so, what I am saying is that, as far as I know: no operators have 
submitted an ID describing how *they* would like to use DSCPs to protect 
their network/traffic. Such an ID, if written, might explain how this 
relates to existing use of DSCPs; explain how operators and host stacks 
would interact with the Endpoint use of DSCPs and ECN codepoints; and 
how they would see the interdomain case.  If someone else wishes to 
contribute that ID, then let it be done without delay, so we can see it.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 24 Mar 2021 17:31:37 +0100

Hi Ruediger,

> On Mar 24, 2021, at 17:05, <[email-masked]> <[email-masked]> wrote:
> 
> Hi Gorry,
> 

	[SM] Almost, CableLabs has made L4S style signaling and AQM mandatory as part of Low Latency DOCSIS 3.1, the mandate seems to cover both the CPE as well as the CMTS. If the L4S experimental RFCs will be released it is very likely that DOCSIS ISPs will enable these features pretty quickly. After all, marketing for low latency docsis started years ago. IMHO this will en-roll quite a number of users into the experiment that are neither "operator employees or friendly volunteers". In that case A DSCP can do wonders to restrict the reach of L4S signaling to participating / L4S aware networks, as SLAs/TCAs are likely required to make sure that tose DSCP are passed on.

	As I have been told, the experiment in the 4S experimental drafts is not about safety or functionality, but about deployment, I expect mass deployment pretty soon. A DSCP during the experiment to reduce the "blast radius" seems like a cautious engineering approach to me, but then I am not an engineer. Also it will be considerably easier, in case I was too pessimistic, to remove the guarding DSCP requirement if L4S is successful and should be elevated to proposed standard status, than introducing one at that point, when mass deployment is already in process or even mostly finished.

Best Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 24 Mar 2021 21:35:57 +0000

Pete,

> from the ambiguous definition of CE that ECT(1) as L4S-ID introduces.

[BB] You're correct. draft-briscoe-tsvwg-l4s-diffserv isn't really 
relevant to this thread.

Bob

>> of
>> other
>>> of

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 24 Mar 2021 22:10:06 +0000

Can someone explain for the record how using a DSCP as an L4S identifier 
would solve the CE ambiguity problem?

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 25 Mar 2021 00:23:46 +0200

> On 25 Mar, 2021, at 12:10 am, Bob Briscoe <[email-masked]> wrote:
> 
> Can someone explain for the record how using a DSCP as an L4S identifier would solve the CE ambiguity problem?

It doesn't.  It only allows containing the resulting harm to participating networks, thereby protecting (most) innocent bystanders.  It's enough, I think, to conduct a field experiment to discover just how bad the problem really is in practice.

This is predicated on the notion that the L4S-ID DSCP(s) are bleached or dropped at the participating networks' border, and in the case of border bleaching, that L4S endpoints interpret CE marks *not* carrying the L4S-ID DSCP as per RFC-3168 or RFC-8511.

I refer you also to my recent post titled "L4S, DSCP, and RFC-4774 Option 2" in which I give a more detailed explanation of a two-DSCP proposal.  The short version is that a version of L4S using that scheme would have *some* positive confidence whether or not they were receiving L4S or conventional marking, without the need for heuristic probing or monitoring.

 - Jonathan Morton

---

From: <[email-masked]>
Date: Thu, 25 Mar 2021 07:17:13 +0000

Hi Sebastian,

My comment refers to experimental deployment, not to network roll-out. It must not be interpreted otherwise.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Mittwoch, 24. März 2021 17:32
An: Geib, Rüdiger <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; [email-masked]
Betreff: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

Hi Ruediger,

Best Regards
	Sebastian

---

From: Pete Heist <[email-masked]>
Date: Thu, 25 Mar 2021 12:45:47 +0100

Just to wrap this up, I fully support the idea that using DSCP to flag
the L4S ECN semantics is a productive way forward for now. IMO, it's
never too late to start down the right path, and it seems like we
should be able to pass drafts that do this relatively quickly.

Beyond what we've thought of to test in the lab, there may well be
things we haven't thought of, both in terms of safety and performance.
It makes sense to look at both on real-world traffic before the
allocation of ECT(1) for a potentially long Internet-wide experiment,
especially given the lab results we do have.

On Thu, 2021-03-25 at 00:23 +0200, Jonathan Morton wrote:
> > Can someone explain for the record how using a DSCP as an L4S
> > identifier would solve the CE ambiguity problem?
> 
> It doesn't.  It only allows containing the resulting harm to
> participating networks, thereby protecting (most) innocent bystanders. 
> It's enough, I think, to conduct a field experiment to discover just
> how bad the problem really is in practice.
> 
> This is predicated on the notion that the L4S-ID DSCP(s) are bleached
> or dropped at the participating networks' border, and in the case of
> border bleaching, that L4S endpoints interpret CE marks *not* carrying
> the L4S-ID DSCP as per RFC-3168 or RFC-8511.
> 
> I refer you also to my recent post titled "L4S, DSCP, and RFC-4774
> Option 2" in which I give a more detailed explanation of a two-DSCP
> proposal.  The short version is that a version of L4S using that scheme
> would have *some* positive confidence whether or not they were
> receiving L4S or conventional marking, without the need for heuristic
> probing or monitoring.
> 
>  - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 25 Mar 2021 14:00:58 +0000

Jonathan, David,

> This is predicated on the notion that the L4S-ID DSCP(s) are bleached or dropped at the participating networks' border, and in the case of border bleaching, that L4S endpoints interpret CE marks *n…

[BB] "Predicated on the notion" is posh for "it depends on a string of 
leaps of faith" (LoFs). Specifically:

LoF (1) The receiver somehow feeds back the arriving DSCP to the sender, 
which would require new IETF work in QUIC, TCPM, RTCP, etc. The only 
use-case to justify DSCP feedback would be to make this L4S-ID DSCP idea 
work. And this work hasn't started anywhere yet. That's not just a leap 
of faith. It's a leap into the abyss.

LoF (2) There are no FIFO RFC3168 AQMs a) upstream of the participating 
network, b) downstream of it, or c) within it (see details below).

LoF (3) That's before we get to all the other reasons for why using two 
fields (ECN and Diffserv) as a classifier would rarely work for those 
who /do/ want to use L4S (in Appx B.4 of ecn-l4s-id already pointed to 
on this thread).

Now let's see how realistic LoF (2) is by putting this DSCP idea to the 
test - by hiding these needles in each type of haystack:

a). Upstream:
Imagine a FIFO RFC3168 AQM in the queue of a home router in the upstream 
direction. An L4S client sends L4S packets through this AQM, which marks 
some as CE. Then the network participating in the L4S experiment 
delivers them, with the L4S-ID DSCP intact, to an on-net server. But the 
CE markings were RFC3168 not L4S.

b) Downstream:
Consider an RFC3168 AQM in the network of a corporate site or campus 
network receiving traffic from a connected L4S-participating network. Or 
perhaps on the access into a data centre connected to the participating 
L4S network. Same problem; the DSCP says its L4S, but the CE was RFC3168.

c) Within:
Consider an RFC3168 AQM within the network participating in the L4S 
experiment. Same problem. The participating network delivers the packet 
with the L4S-ID DSCP intact, but the CE marking is RFC3168, not L4S.

In this last case, the L4S-paticipating network has to use techniques 
like those in l4sops to make sure any FIFO RFC3168 AQMs don't mark 
ECT(1). That's no different to if ECT(1) alone was the identifier.

ecn-l4s-id correctly summarizes all the above with the following para:

    Hard to distinguish Classic ECN AQM:  Defining a DSCP to indicate L4S
       is a way to help network nodes identify L4S packets (albeit
       unreliable due to the likelihood of bleaching - see above).
       However, it does not help hosts distinguish between ECN markings
       from L4S and Classic AQMs.  This is because Classic AQMs would
       have been implemented without any logic to recognize an L4S DSCP
       or apply L4S marking behaviour.

More frankly, the amount of time wasted discussing this idea is down to 
an extremely sloppy characterization of the problem in some people's 
minds as something vaguely to do with classifiers. Added to extreme 
vagueness in the distinction between receivers and senders. This is not 
how the IETF should be.

> I refer you also to my recent post titled "L4S, DSCP, and RFC-4774 Option 2" in which I give a more detailed explanation of a two-DSCP proposal.  The short version is that a version of L4S using th…

This requires the same 3 leaps of faith as above.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 25 Mar 2021 15:40:56 +0100

Hi Bob,

I top post, because this is related, but not a point by point response to your argument why you consider Jonathans proposal unpalatable.

Here is my back of the envelop version of how to use a guard DSCP to secure an L4S experiment over existing networks without increasing the risk for innocent bystanders.

1) make the reference AQM configurable to require both the guard DSCP and ECT(1) (or CE) to steer a packet into the LL-queue.

2) make the reference protocols configurable to only negotiate "L4S-mode" if the initiator packets also carried the guard DSCP.

3) make the reference protocols configurable to only treat CEs as HF-CC signals if they are accompanied with the guard DSCP, otherwise treat them as a drop

4) optionally, configure the non-LL queue to not use ECN, so that any ECN marking in packet captures will come from upstream for unguarded packets.

Require for the duration of the experiment (by a MUST in the ID/RFC), that all L4S-protocols and L4S-AQMs ofparticipating networks/entities be configured to follow rules 1 to 3, and also require participating networks to use SLAs/TCAs to make sure to only pass the guard DSCP into networks that are also participating in the experiment, otherwise that DSCP should be re-marked to something else.

Given that neither the L4S AQM nor the protocols are finished and deployed yet, adding these features for the sake of a safe experiment seems not unfeasible, no?

The important idea here is not to reach 100% certainty, but really to use the DSCP as a tool for all participating networks to signal " I am L4S-aware and so is the next hop". It is still expected that these networks then follow the L4S-ops draft. What this should achieve is to avoid leaking L4S signaling into networks that are not prepared to handle L4S behavior.

This will allow to finally perform all those safety and functionality measurements and tests, that you believe can only be won over live networks, without externalizing the risk to innocent bystanders significantly.

This is also orthogonal to the question what identifier a L4S PS might use in the end.
I note your objections to Jonathan's proposal of incorporating DSCPs into L4S proper (which I think would be a good idea, but orthogonal to my idea here). I commented upon these in the context of this limited guard DSCP proposal (seatch for text prefixed [SM]) and think none of them would be show-stopper, but all are addressed. No leaps of faith required...

Best Regards
	Sebastian

> On Mar 25, 2021, at 15:00, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan, David,
> 
> [BB] "Predicated on the notion" is posh for "it depends on a string of leaps of faith" (LoFs). Specifically:
> 
> LoF (1) The receiver somehow feeds back the arriving DSCP to the sender, which would require new IETF work in QUIC, TCPM, RTCP, etc. The only use-case to justify DSCP feedback would be to make this L4S-ID DSCP idea work. And this work hasn't started anywhere yet. That's not just a leap of faith. It's a leap into the abyss.
> 
> LoF (2) There are no FIFO RFC3168 AQMs a) upstream of the participating network, b) downstream of it, or c) within it (see details below).
> 
> LoF (3) That's before we get to all the other reasons for why using two fields (ECN and Diffserv) as a classifier would rarely work for those who /do/ want to use L4S (in Appx B.4 of ecn-l4s-id already pointed to on this thread).
> 
> 
> Now let's see how realistic LoF (2) is by putting this DSCP idea to the test - by hiding these needles in each type of haystack:
> 
> a). Upstream:
> Imagine a FIFO RFC3168 AQM in the queue of a home router in the upstream direction. An L4S client sends L4S packets through this AQM, which marks some as CE. Then the network participating in the L4S experiment delivers them, with the L4S-ID DSCP intact, to an on-net server. But the CE markings were RFC3168 not L4S. 

	[SM] In my proposal that only happens if the end-network invalidly claims to be L4S-aware, by asserting the guard DSCP while employing an rfc3168 AQM. This clearly would be the responsibility of the end-point network to remedy. 

> 
> b) Downstream: 
> Consider an RFC3168 AQM in the network of a corporate site or campus network receiving traffic from a connected L4S-participating network. Or perhaps on the access into a data centre connected to the participating L4S network. Same problem; the DSCP says its L4S, but the CE was RFC3168. 

	[SM] Unless the receiving network declared L4S awareness via an SLA/TCA to conserve the guard DSCP un-mapped, the L4S-participating network would remap the guard DSCP and de-fang the L4S signaling. The whole idea is to make sure that the L4S experiment happens between/in L4S-participating network...

> 
> c) Within: 
> Consider an RFC3168 AQM within the network participating in the L4S experiment. Same problem. The participating network delivers the packet with the L4S-ID DSCP intact, but the CE marking is RFC3168, not L4S. 

	[SM] The L4S-participating network needs to follow the L4S-ops ID/RFC to make sure this can not happen.

> 
> In this last case, the L4S-paticipating network has to use techniques like those in l4sops to make sure any FIFO RFC3168 AQMs don't mark ECT(1). That's no different to if ECT(1) alone was the identifier.
> 
> 

	[SM] In the context of guarding an L4S experiment bleaching is part of the safety mechanism and not an issue.

>       or apply L4S marking behaviour.

	[SM] For the scope of an experiment, it would be in the responsibility of participating networks to assure the lack of sensitive rfc3168 AQMs, side-stepping this issue as well.

> 
> 
> More frankly, the amount of time wasted discussing this idea is down to an extremely sloppy characterization of the problem in some people's minds as something vaguely to do with classifiers. Added to extreme vagueness in the distinction between receivers and senders. This is not how the IETF should be.
> 
> 
>>

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 25 Mar 2021 18:58:12 +0200

> [BB] "Predicated on the notion" is posh for "it depends on a string of leaps of faith" (LoFs).

I really don't like your tone with this phrasing.  But let's put that aside.

> Specifically:

Are you arguing here that the current L4S design is so thoroughly baked into existing standardisation work and/or implementations that not even minor changes can be made?  It sounds very much like it to me, and that is very troubling for the prospect of making L4S safe to deploy beyond an Option 1 (completely isolated) context.

In any case, I have already shown you a way to feed back the presence or absence of a particular DSCP.  Simply refrain from emitting the special DSCP, except on the initial SYN, unless you received the appropriate special DSCP on all valid packets received on the connection.  That gives you a very easy way to switch the connection into "Classic mode" if it strays outside the approved path.

> LoF (2) There are no FIFO RFC3168 AQMs a) upstream of the participating network, b) downstream of it, or c) within it (see details below).

Incorrect.  Remember that when using a DSCP as a containment mechanism, it is assumed (as I spelled out previously) that the borders of the network at least bleach that DSCP, and may even block traffic carrying it.  That's how containment works.  So if the L4S traffic crosses a border of the participating network, it automatically switches to Classic mode and there is no more problem with RFC-3168 AQMs on any part of the path.

Which leaves open two scenarios where an RFC-3168 could be encountered without crossing a border:

1: Within the participating network.  But all RFC-3168 AQMs are supposed to have been removed from such a network before L4S was deployed on it.  So either the onus is on the network operator (who is at least an "interested observer") to deal with the problem, *or* the RFC-3168 AQM is present specifically for the purpose of experimentation, eg. to find out how the combination with L4S traffic really works in the real world.

2: The path runs entirely outside the participating network.  This implies that two communicating L4S endpoints were deployed outside the experimental network, which would violate the spirit of the experimental deployment and should not be allowed.  The two-DSCP scheme I outlined in my other post specifically attempts to detect and protect against this eventuality.

> LoF (3) That's before we get to all the other reasons for why using two fields (ECN and Diffserv) as a classifier would rarely work for those who /do/ want to use L4S (in Appx B.4 of ecn-l4s-id alr…

I do not accept Appendix B.4 to be a valid set of arguments against using DSCP for this purpose.  If nothing else, it does not appear to consider any form of the two-DSCP scheme I recently described.  I would encourage you to give it a serious, principled consideration.

> In this last case, the L4S-paticipating network has to use techniques like those in l4sops to make sure any FIFO RFC3168 AQMs don't mark ECT(1). That's no different to if ECT(1) alone was the ident…

Thank you for confirming that you understand that using ECT(1) alone as the identifier suffers from all of these problems.  That was a big sticking point over the past two years, and I'm glad we can now get past that.

> This requires the same 3 leaps of faith as above. 

No, it does not.  I encourage you to re-read that and carefully absorb my explanation of how it works.  I think you will find the exercise very enlightening.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 25 Mar 2021 17:54:20 +0000

If using DiffServ is considered as a widely deployable technique on the internet, then we should simply use DiffServ as the "input" L4S identifier and use SCE as the congestion "output". I don't think we need more complex experimental schemes, especially if they end up needing a final DiffServ codepoint anyway even after the experiment (like in Jonathan's proposal).

As such it would be the perfect marriage of both L4S and SCE. So an AQM can only mark SCE if it has the DiffServ L4S-id and can protect and isolate the L4S-SCE flows from flows not responding to the SCE marks (in a DualQ or FQ, or any other scheme detecting presence of non-L4S flows).

This was one of the options that was considered before and which would be the simplest solution involving DiffServ.

The question was and still is if DiffServ "is" the perfect wedding proposal for Internet wide traffic. How many more hurdles do we need for deploying scalable and smooth congestion control (which is I assume the common goal of both L4S and SCE).

Main question is: Is the risk (pushing away Classic flows on a RFC3168 ECN bottleneck after tens of seconds) worth the extra deployment obstacles (practically making its deployment as constrained as an end-to-end managed service).

If the answer is yes, then we should define a DiffServ codepoint that identify L4S and only use SCE for those. If the risk is assumed minimal and comparable with other experimental CCs (see ICCRG https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-the-great-internet-tcp-congestion-control-census-00), then we live with the (small?/controlled?) impact.

I understood the latter had the biggest support in the input/output vote during the previous interim. I think if L4S flows are not used for long time downloads (non-application-limited flows), the impact is minimal and a 3168 detection and fallback would never be needed. It also was known that Classic bottleneck detection would be a challenge, but I think Asad/Bob showed that a (maybe not 100% perfect, but already) very good implementation is possible if needed for the problem conditions that are known then. These could be used (experimented with) when using L4S for downloads.

So bottom-line question: Who believes an end-to-end DiffServ solution is realistically deployable???
Second sub question: who believes L4S is a bigger risk than all other CCs "experiments" on the Internet??

Regards,
Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
Sent: Wednesday, March 24, 2021 11:24 PM
To: Bob Briscoe <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 25 Mar 2021 21:11:42 +0200

> On 25 Mar, 2021, at 7:54 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

That's certainly an interesting idea, and one that we have at least partly considered as part of developing SCE.

In the SCE context, DSCP would be used first as an experiment protection mechanism (if the WG decided that was necessary), and then as a way to handle some of SCE's relatively few failure modes that exist when deployed *without* a DSCP.  It would not be necessary to use DSCP as a containment mechanism, only an enabler.  Some details below.

> Main question is: Is the risk (pushing away Classic flows on a RFC3168 ECN bottleneck after tens of seconds) worth the extra deployment obstacles (practically making its deployment as constrained a…

First let me clear up a small point:  SCE's typical failure mode is the opposite to L4S's typical failure mode.  When the two inadvertently end up in the same queue and managed by the same AQM instance, L4S squashes the RFC-3168 traffic, but SCE would be the one that gets squashed by RFC-3168 in the same scenario (if and only if the AQM is SCE enabled).

This clearly changes the risk profile, since the risk is now to the involved participants in SCE, rather than to the innocent bystanders obeying existing specifications.  This makes the risk much easier to manage when it arises, and we can legitimately engage in exercises which reduce that risk (as a performance enhancement in specific scenarios) but don't eliminate it entirely - and there is actually some incentive to do so, because the risk is not externalised.

That is where the DSCP comes in.  Using the DSCP, the sender advertises that it understands SCE, while traffic not carrying the DSCP might not.  An SCE-aware middlebox can use that as extra information when distinguishing flows from each other, and thus avoid placing SCE traffic in the same queue as non-SCE traffic.  That's not the only strategy possible, but an illustrative one; we can discuss others later.

At a non-SCE middlebox, there is no need for it to understand or react to the DSCP.  SCE transports react to the CE marks that RFC-3168 AQMs produce in the same way as conventional traffic, so there is no incompatibility.  This also makes losing the DSCP en route relatively safe, even if an SCE-aware AQM is encountered subsequently.

> So bottom-line question: Who believes an end-to-end DiffServ solution is realistically deployable???

It is well-known that DSCPs get altered in various ways across general Internet paths.  What matters here is whether losing the DSCP indicating L4S or SCE support is fail-safe.

In current L4S it is not fail-safe, because the receiver doesn't know (and can't tell the sender) what type of AQM applied the mark.  The DSCP schemes that Sebastian and I described are capable of being fail-safe if correctly implemented.

In SCE, it is inherently fail-safe since the receiver and sender both know what type of AQM applied the marks.

> Second sub question: who believes L4S is a bigger risk than all other CCs "experiments" on the Internet??

L4S in its current form has the same risk profile as DCTCP, which is explicitly classified as NOT suitable for deployment on the Internet, because it is so much more harmful to AIMD traffic than most of the other experiments out there.  With good DSCP protection, that could be improved to an RFC-4774 Option 2 standard, but I think not to Option 3.

I note that some other very aggressive TCPs that predated CUBIC are also not considered suitable for deployment (I suspect their RFCs have been retired, but haven't checked).  By contrast, most other congestion controls currently in development are either closely tied to datacentre technology and are thus naturally restricted to that environment, or are going to considerable lengths to compete reasonably with the status quo, especially CUBIC, or even to behave like low-priority scavengers.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Thu, 25 Mar 2021 22:13:12 -0400

On Thu, 2021-03-25 at 17:54 +0000, De Schepper, Koen (Nokia -
BE/Antwerp) wrote:
> If using DiffServ is considered as a widely deployable technique on
> the internet, then we should simply use DiffServ as the "input" L4S
> identifier and use SCE as the congestion "output". I don't think we
> need more complex experimental schemes, especially if they end up
> needing a final DiffServ codepoint anyway even after the experiment
> (like in Jonathan's proposal).
> 
> As such it would be the perfect marriage of both L4S and SCE. So an
> AQM can only mark SCE if it has the DiffServ L4S-id and can protect
> and isolate the L4S-SCE flows from flows not responding to the SCE
> marks (in a DualQ or FQ, or any other scheme detecting presence of
> non-L4S flows).
> 
> This was one of the options that was considered before and which
> would be the simplest solution involving DiffServ.
> 
> The question was and still is if DiffServ "is" the perfect wedding
> proposal for Internet wide traffic. How many more hurdles do we need
> for deploying scalable and smooth congestion control (which is I
> assume the common goal of both L4S and SCE).

Why are you discussing deployment? I thought this exercise was about
enabling experiments? 

> Main question is: Is the risk (pushing away Classic flows on a
> RFC3168 ECN bottleneck after tens of seconds) worth the extra
> deployment obstacles (practically making its deployment as
> constrained as an end-to-end managed service).

Worth it to whom? A general principle is that people conducting
experiments shouldn't impose burdens on third parties.

> If the answer is yes, then we should define a DiffServ codepoint that
> identify L4S and only use SCE for those. If the risk is assumed
> minimal and comparable with other experimental CCs (see ICCRG 
> https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-the-great-internet-tcp-congestion-control-census-00
> ), then we live with the (small?/controlled?) impact.
> 
> I understood the latter had the biggest support in the input/output
> vote during the previous interim. I think if L4S flows are not used
> for long time downloads (non-application-limited flows), the impact
> is minimal and a 3168 detection and fallback would never be needed.
> It also was known that Classic bottleneck detection would be a
> challenge, but I think Asad/Bob showed that a (maybe not 100%
> perfect, but already) very good implementation is possible if needed
> for the problem conditions that are known then. These could be used
> (experimented with) when using L4S for downloads.
> 

This should not be an issue if and until L4S experiments prove that it
is a beneficial technology to deploy.

> CCs "experiments" on the Internet??

No one can offer an informed opinion until there is experimental data.
The existing simulation data is not promising, to put it mildly.

Regards,

// Steve

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 08:53:39 +0100

Hi Steven,

> On Mar 26, 2021, at 03:13, Steven Blake <[email-masked]> wrote:
> 
> is a beneficial technology tfor details.o deploy.
> 

	[SM] I think Pete's data, while from an artificial testbed and not from a live production network, counts as more than a mere simulation, but the conclusion "not promising" still holds.See https://github.com/heistp/l4s-tests#rtt-unfairness 

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 09:06:01 +0100

Hi Koen,

I think none of this addresses my simple proposal to use a guard DSCP for the duration of the L4S experiment. Looking at the NQB ID (https://tools.ietf.org/html/draft-ietf-tsvwg-nqb-05) and the low latency docsis (LLD) ID (https://tools.ietf.org/id/draft-white-tsvwg-lld-00.html) I think I can even propose a candidate DSCP pattern: 0x2A 
(for the use as guard DSCP the fact that this has a lower per-chance traversal likelihood than say 0x02 is actually a feature in the context of a guard DSCP). 

Here is cable labs position (NQB is used as synonym for the LL-queue and traffic having appropriate properties to qualify for the LL-queue):
"As of the writing of this draft, it is proposed that the DiffServ value 0x2A be standardized in IETF/IANA to indicate NQB [I-D.white-tsvwg-nqb]."

So it seems pretty clear that one of the biggest drivers behind the L4S effort already assumes that a L4S DSCP is viable, desirable and standardization is already in process. One could say they consider "DiffServ [...] as a widely deployable technique on the internet", no?

Since I have seen no arguments from you on the list arguing against the feasibility of an NQB DSCP/PHB, I wonder why you accept that as viable for the NQB draft and LLD, but not for L4S. Especially since LLD contains L4S as one of its building blocks?

Best Regards
	Sebastian

> On Mar 25, 2021, at 18:54, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 26 Mar 2021 11:07:59 +0000

Hi Sebastian,

The NQB codepoint is not needed to make L4S work. It could be interesting within small network segments (near the access edges: home and DC, not the core) to add more traffic into the L4S Q. I don't think anyone considers it very realistic using NQB as a widely deployable E2E feature, although it would be a bonus if it would finally, and RFCs should allow it to become that. That doesn't mean it will be a fact within the next decades.

I don't think your proposal is very simple, and has quite some impact on all parts of the architecture. If DiffServ would be in the picture, I see the DiffServ-input and ECN-output as the only simple straightforward solution. It removes the disadvantages of both while keeping the advantages of both. This was always my understanding, and I don't see any reason why this has changed. Let's keep things to the point.

Would you want SCE only to work if it has the right DiffServ codepoint on it? SCE would also need the DiffServ codepoint to protect SCE from non-SCE traffic. Even when using FQ, tunneled SCE gets trampled over by non-SCE traffic.

Again, the only question is, who wants to deploy a feature that relies on a DiffServ codepoint that needs end to end path support??? 
Let it be clear: I'm NOT. I think this is suicide... And I understood that was the main consensus during the input/output vote, and the main reason why people accepted L4S with the small risk of getting too high throughput, rather than SCE needing FQ or getting again the delay-based problems (getting trampled over by loss based and CE-only CCs).

Regards,
Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Friday, March 26, 2021 9:06 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; Bob Briscoe <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

Hi Koen,

I think none of this addresses my simple proposal to use a guard DSCP for the duration of the L4S experiment. Looking at the NQB ID (https://tools.ietf.org/html/draft-ietf-tsvwg-nqb-05) and the low latency docsis (LLD) ID (https://tools.ietf.org/id/draft-white-tsvwg-lld-00.html) I think I can even propose a candidate DSCP pattern: 0x2A (for the use as guard DSCP the fact that this has a lower per-chance traversal likelihood than say 0x02 is actually a feature in the context of a guard DSCP). 

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 12:54:43 +0100

Hi Koen,

more below, prefixed [SM]. But as a preface, it seems while you addresses this mail at me, you do not actually discuss my proposal at all. Why?
	While I am fully sympathetic to the proposal to re-jig L4S' signaling mechanism, all I was/am proposing is a method to contain potential fall-out during an L4S experiment to those voluntarily participating in the experiment, while keeping innocent bystanders reasonably safe. 
	Is my proposal 100% perfect or technically elegant? Sure not, but it does not need be, the initial "mandatory" life-time of this would only cover L4S experimental phase and a PS phase only IFF it had proved itself to be functional and feasible, so the containment mechanism would need to survive the same level of evaluation and scrutiny as L4S itself.  

> On Mar 26, 2021, at 12:07, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> The NQB codepoint is not needed to make L4S work.

	[SM] I know, but I did not claim that (the guard DSCP is about making the L4S experiment safe, not about L4S' core functionality). My claim is, that the biggest expected immediate user of L4S saw it fit to make sure that an L4S-DSCP exists. This indicates that they do/did not share your concerns about the applicability of such a DSCP. And in actual discussion Greg seemed to indicate that he sees the fact that SLA's/TCAs might be required to guarantee safe-passage of such an L4S-DSCP as acceptable and clearly not a show-stopper.

> It could be interesting within small network segments (near the access edges: home and DC, not the core) to add more traffic into the L4S Q. I don't think anyone considers it very realistic using NQB as a widely deployable E2E feature, although it would be a bonus if it would finally, and RFCs should allow it to become that. That doesn't mean it will be a fact within the next decades.

	[SM] Mind you, you are still discussing the "make DSCPs an integral part of L4S signaling" proposal. For my proposal of using a guard DSCP for the duration of the L4S experiment, your arguments do simply not hold. Not propagating by default E2E is part of what makes a guard DSCP actually helpful to contain an L4S experiment to participating networks. Also, the one use case where L4S is going to work for reasonably well, is short-RTT, low-hop count "fast-lanes", say, between eyeballs and content providers, a situation that typically involves a sufficient low number of AS/networks/actors to make the "negotioate DSCP-passage per contract" approach feasible.

> 
> I don't think your proposal is very simple,

	[SM] Well, the problem is complex, so any solution will have to have some complexity in itself, but conceptually my proposal is rather small; I do not claim that implementing it might not take work and effort though.

> and has quite some impact on all parts of the architecture.

	[SM] I disagree, adding configuration options that are required to be active for the experiment, will allow to require/allow to change these to inactive for full standards deployment. The impact would be to add these features/capabilities for the duration of the experiment sure. IMHO adding these sounds preferable than not running an experiment at all... As David indicated, with a safety proposal similar to my rough idea, the L4S experiment could have probably been started years ago, so is not changing the implementation really worth years of wall clock waiting time?

> If DiffServ would be in the picture, I see the DiffServ-input and ECN-output as the only simple straightforward solution.

	[SM] I agree that this sounds like a better design, than what L4S proposed right now, in regards to safety-by-design, but I maintain, that even my more modest guard-DSCP proposal might unlock the path to experimentation sooner. I would be delighted, if during experimentation such an "DiffServ-input and ECN-output" alternative would also be tested.

> It removes the disadvantages of both while keeping the advantages of both. This was always my understanding, and I don't see any reason why this has changed. Let's keep things to the point.

	[SM] Well, that is not my point, so if you will. I keep sticking to my point. If you want to deploy L4S signaling as currently designed and implemented for experimentation in live production networks soon, put L4S under the containment of a guarding DSCP would be a great way to convince me of the safety of doing so. And I assume that I am not alone in this.

> Would you want SCE only to work if it has the right DiffServ codepoint on it? SCE would also need the DiffServ codepoint to protect SCE from non-SCE traffic. Even when using FQ, tunneled SCE gets t…

	[SM] I defer to Jonathan's point regarding fitting SCE-style ECT(1) output signaling into L4S. This is orthogonal to the issue of how to make experiments with existing L4S design and implementation safe.

> Again, the only question is, who wants to deploy a feature that relies on a DiffServ codepoint that needs end to end path support??? 

	[SM] According to the NQG ID, Greg, assumes this to be an attractive feature. I assume that Greg speaks on behalf of CableLabs and also assume that CableLabs thinks that DOCSIS-ISPs will be interested in that feature as well. I did not notice you objecting to the NQB draft based on your doubts about DSCP passage... 

> Let it be clear: I'm NOT. I think this is suicide...

	[SM] Why? And "suicide" is not really an appropriate term here. That said IMHO, L4S is not going to fail because of a DSCP requirement; L4S is going to fail because L4S-compliant protocols fail to compete sufficiently well on non-L4S bottlenecks, and because L4S bottlenecks give such an undeserved advantage to L4S flows that folks will disable these pretty quickly. Whether or not you use a DSCP, is irrelevant to the sad fact that L4S is simply not ready for anything else than Eyeball<->nearCDN fast lanes (where ironically, DSCP can be reasonably expected to be E2E, either by default or facilitated by SLA/TCA).

> And I understood that was the main consensus during the input/output vote, and the main reason why people accepted L4S with the small risk of getting too high throughput, rather than SCE needing FQ or getting again the delay-based problems (getting trampled over by loss based and CE-only CCs).

	[SM] If we actually had asked folks about their motivations, we might know, but we didn't... so let's not go there, please.

Best Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 26 Mar 2021 12:12:19 +0000

Hi Steve, (and others that think this "experiment" should be about proving performance of L4S CCs)

The survey confirmed that none of the CC developers doubt that a frequent L4S-CE (being equal to SCE) signal is useful and will improve all kinds of CCs. That the current Prague implementation is not fully up to its optimal performance level yet is because there is not much incentive for the real beneficiaries to work on that (can't be used anyway, as no L4S networks are available). Saying this, the outcome of the survey was also that many have implementations or plans. We all tent to forget that also BBRv2 is a very promising (partial) implementation that supports L4S, where only TCP-Prague seems to attract attention. With the 2 publicly available implementations (Prague and BBRv2) joined together we cover all performance features.

I see the experiment mainly as a "DEPLOYMENT" facilitator, not a performance experiment. Only the need for a minimum window and Classic ECN bottleneck detection needs real live experience (what is the prevalence, its impact and what are the best control/detection mechanisms) and those would need to be further developed during the first deployment wave. I don't see the point of having an experiment in controlled (DiffServ) segments, as this could be done in labs too. If we have this many parties involved, we need to create an open ecosystem (or call it platform) where the solutions can be developed, like it is common practice today. Innovations in CC could up to now only happen from a one-party-only: "sender endpoint CC constrained by Classic CCs, better AQM for Classic CCs". L4S could provide an uncluttered platform for a multi-party innovation in the transport area. I think we should see it like that, and that justifies the "experimental" draft status, but we should go for real open deployments from day one, with as less extra hurdles added to this (ref other mails about end-to-end DiffServ).

If in IETF TSVWG we can't create these coordinated platforms, we will keep on discussing forever (it seems), while the Internet world will happily and freely (though sub-optimally) progress on actions from individual parties.

Regards,
Koen.

-----Original Message-----
From: Steven Blake <[email-masked]> 
Sent: Friday, March 26, 2021 3:13 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

BE/Antwerp) wrote:
> This was one of the options that was considered before and which would 
> be the simplest solution involving DiffServ.
> 

Why are you discussing deployment? I thought this exercise was about enabling experiments? 

> deployment obstacles (practically making its deployment as constrained 
> as an end-to-end managed service).

Worth it to whom? A general principle is that people conducting experiments shouldn't impose burdens on third parties.

> https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-th
> e-great-internet-tcp-congestion-control-census-00
> for long time downloads (non-application-limited flows), the impact is 
> minimal and a 3168 detection and fallback would never be needed.
> challenge, but I think Asad/Bob showed that a (maybe not 100% perfect, 
> but already) very good implementation is possible if needed for the 
> problem conditions that are known then. These could be used 

This should not be an issue if and until L4S experiments prove that it is a beneficial technology to deploy.

Regards,

// Steve

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 26 Mar 2021 12:30:17 +0000

Hi Sebastian,

I understood the goal of your proposal. But before diving into the details of a DiffServ-based proposals, I'm taking a step back asking: Is using DiffServ an option at all. I don't believe so (see other reply to Steve's mail related to the reason for deployment).

As a second step after this, "IF" using DiffServ is an option, then L4S and SCE can get married and we can benefit from both. If DiffServ cannot be used, we have to pick one, like we did up to now and trust as a community that we together can make that work. I think the IETF needs to provide the platform for multi-party innovations, but then needs to be able to let control go and to trust the professionalism of the individual parties.

Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Friday, March 26, 2021 12:55 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; Bob Briscoe <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

Hi Koen,

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 13:54:05 +0100

Hi Koen,

> On Mar 26, 2021, at 13:12, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> The survey confirmed that none of the CC developers doubt that a frequent L4S-CE (being equal to SCE) signal is useful and will improve all kinds of CCs.

	[SM] Sure, we also probably have no doubts that world peace would be great to hav, as well as a generic cure for cancer. But wishing alone will not make that a reality.

> That the current Prague implementation is not fully up to its optimal performance level yet is because there is not much incentive for the real beneficiaries to work on that (can't be used anyway, as no L4S networks are available).

	[SM] OR because it might be simply impossible to reach a better state than what little is achieved in TCP Prague today. The onus is on you to demonstrate that this new signaling works as advertised. Declaring all non AQM-parts as somebody else's problem does not instill confidence in me.

> Saying this, the outcome of the survey was also that many have implementations or plans.

	[SM] Great, but until any of those get realized and surpass TCP Prague's performance we have to accept that TCP Pragure might be as good as it gets.

> We all tent to forget that also BBRv2 is a very promising (partial) implementation that supports L4S, where only TCP-Prague seems to attract attention.

	[SM] Well, if you show data that confirms that BBR-L4S surpasses TCP Prague both in meeting L4S' requirements as well as performance I happily discuss that data. But until that point TCP Prague is all you have to show for... Well, that is not fair, I guess one outcome of L4S so far has been, pure DCTCP truly is not suited for the internet, even with an isolating AQM.

> With the 2 publicly available implementations (Prague and BBRv2) joined together we cover all performance features.

	[SM] Your challenge is to meet all protocol requirements, performance promises and safety grantees in a single implementation. Splitting these into two protocols is not going to help, so I would expect you to focus on getting on reference protocol up and running first.

> 
> I see the experiment mainly as a "DEPLOYMENT" facilitator, not a performance experiment.

	[SM] That is what I feared. L4S is clearly neither safe nor functional for the existing internet, and instead of using a staged approach to run an experiment first whether your goals and promises will actually hold for normal internet conditions, where path RTT between different flows easily differs by multiple dozen or even 100ms, you want to mass deploy what you have right now. Knowing full well, that after a mass deployment getting any changes into the deployed instances is an uphill fight.
What makes you so sure that the network bits of L4S are truly ready for prime-time as they are, that you are willing to forego the additional round of changes after your baby had actual contact with the real internet?

> Only the need for a minimum window and Classic ECN bottleneck detection needs real live experience (what is the prevalence,

	[SM] I disagree. L4S still requires testing under those realistic conditions that I have been describing for some years now.

> its impact and what are the best control/detection mechanisms) and those would need to be further developed during the first deployment wave.

	[SM] Sorry, these should exist before the first deployment wave, what you have right now, is "thoughts and prayers" instead of robust control and detection mechanisms.

> I don't see the point of having an experiment in controlled (DiffServ) segments, as this could be done in labs too.

	[SM] No, I agree that L4S has not exhausted the development that can be done with the results from lab experiments. What exactly do you want to experiment with when your only question seems to be, on how many hosts can we deploy this? I will spell this out explicitly, deployment is not en experiment, at least not in any scientific meaning of the word experiment. A real experiment can contain deployment sure, but what is the hypothesis you want to test here?

> If we have this many parties involved, we need to create an open ecosystem (or call it platform) where the solutions can be developed, like it is common practice today.

	[SM] But you have for years now, objected to any changes to the broken designs and implementations of L4S, why is that going to change in the near future?

> Innovations in CC could up to now only happen from a one-party-only: "sender endpoint CC constrained by Classic CCs, better AQM for Classic CCs". L4S could provide an uncluttered platform for a multi-party innovation in the transport area.

	[SM] The operation word is "could", this is close to wishful thinking, not engineering, IMHO.

> I think we should see it like that, and that justifies the "experimental" draft status, but we should go for real open deployments from day one, with as less extra hurdles added to this (ref other mails about end-to-end DiffServ).

	[SM] I consider current L4S design and implementation to be comically and yet catastrophically broken, I object to deploying it as it is right now, and especially without any robust containments in place. 

> If in IETF TSVWG we can't create these coordinated platforms, we will keep on discussing forever (it seems), while the Internet world will happily and freely (though sub-optimally) progress on acti…

	[SM] Or L4S can simply be changed according to the ample feed-back you received to actually stand a better chance to achieve its lofty goals and promises. The point is, some of us have actual operational experience in getting latency lowering technology out to interested users in the field, an experience which guided some of the free advice and feed-back that L4S so far mainly decided to ignore.

Best Regards
	Sebastian

---

From: Kyle Rose <[email-masked]>
Date: Fri, 26 Mar 2021 08:54:58 -0400

On Fri, Mar 26, 2021, 8:30 AM De Schepper, Koen (Nokia - BE/Antwerp) <
[email-masked]> wrote:

> I understood the goal of your proposal. But before diving into the details
> of a DiffServ-based proposals, I'm taking a step back asking: Is using
> DiffServ an option at all.

Why wouldn't it be? The point of the proposal AIUI is to require networks
to take action to explicitly opt-in to this experiment. Given the DiffServ
bleaching that occurs by default at network boundaries, operators that take
no action will not find themselves ambushed by novel behavior.

This approach allows for experimenting with ECT(1) as a classifier in a
safe (contained, time-limited) way, thus allowing a path toward universal
deployment (i.e., removing the DSCP guard) if the experiment is successful
without requiring standardization and rollout of end-to-end DiffServ. And
it does so (a) without permanently burning the ECT(1) code point if the
experiment fails, and (b) in an opt-in manner that greatly reduces the
potential for unintended side effects on classic traffic.

This seems to me like the obvious path toward consensus on an experimental
deployment aimed at gathering real-world experience without first
committing to something unproven.

Kyle

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 14:09:51 +0100

Hi Koen,

> On Mar 26, 2021, at 13:30, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> I understood the goal of your proposal.

	[SM] Then why did you not actually address it? I am confused.

> But before diving into the details of a DiffServ-based proposals, I'm taking a step back asking: Is using DiffServ an option at all. I don't believe so (see other reply to Steve's mail related to the reason for deployment).

	[SM] Well, then you failed, because none of your abstract theoretical objections actually fit my proposal, you are free to slay any strawmen you like, but that does not show why my simple proposal would be unworkable. I tried to address yours and Bob's objections and think I showed why these, in the context of a guard DSCP are sufficientky met tp at east merit a discussion of the proposal. Sure, this proposal has been declared "crazy" by some, but if you feel the same than show my where and why, please.

> 
> As a second step after this, "IF" using DiffServ is an option,

	[SM] Sorry, this is trying to use a general challenge with Diffserv in a context where it does not apply. The core idea of my proposal is, the guard DSCPs will not escape participating networks, so the fact that DSCPs are unlikely to be E2E (your single most important objection) simply is not a logical argument against my proposal. 

> then L4S and SCE can get married and we can benefit from both.

	[SM] Different discussion. Also I do not accept for one minute that you are actually willing to entertain this as a real option. But I understood that when proposing a guard DSCP that leaves the core L4S machinery underneath in tact. Which is of essence, given that it took you like a decade to get to this point, so I accept that you want to test the design and implementation you have, and not that you might wish you had.

> If DiffServ cannot be used,

	[SM] There is zero indication that it can not be used as a guard DSCP, if you have real objections, please bring them, the proposal needs fleshing out (or shooting down, I might have overlooked a showstopper, but DSCP not naturally being E2E is not a showstopper here).

> we have to pick one, like we did up to now and trust as a community that we together can make that work.

	[SM] What in the L4S development in this WG for the last few years, makes you believe that?

> I think the IETF needs to provide the platform for multi-party innovations, but then needs to be able to let control go and to trust the professionalism of the individual parties.

	[SM] Hard to parse, but it explains why you essentially propose RFCs without teeth (like no mandatory policing that ECT(1) flows actually behave in an L4S compatible fashion, and no reliable guarantees for sharing between flows of different RTT or traffic class). Add to this the desire to inflict the current known broke state of L4S (it neither functions as promised, nor is t safe for the rest of the internet) onto the existing internet without any robust and reliable safety mechanisms, and I really doubt that I am willing to trust your professionalism, sorry. 

So after this philosophical aside, can we maybe discuss why my proposal is unacceptable, please? 

Regards
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 26 Mar 2021 13:43:54 +0000

On 26/03/2021 12:54, Kyle Rose wrote:
> On Fri, Mar 26, 2021, 8:30 AM De Schepper, Koen (Nokia - BE/Antwerp) 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I understood the goal of your proposal. But before diving into the
>     details of a DiffServ-based proposals, I'm taking a step back
>     asking: Is using DiffServ an option at all.
>
>
> Why wouldn't it be? The point of the proposal AIUI is to require 
> networks to take action to explicitly opt-in to this experiment.
And endpoints and a new way of using DSCPs with transports.
> Given the DiffServ bleaching that occurs by default at network 
> boundaries, operators that take no action will not find themselves 
> ambushed by novel behavior.
>
Does it?
> This approach allows for experimenting with ECT(1) as a classifier in 
> a safe (contained, time-limited) way, thus allowing a path toward 
> universal deployment (i.e., removing the DSCP guard) if the experiment 
> is successful without requiring standardization and rollout of 
> end-to-end DiffServ.
That didn't and doesn't require an RFC.
> And it does so (a) without permanently burning the ECT(1) code point 
> if the experiment fails, and (b) in an opt-in manner that greatly 
> reduces the potential for unintended side effects on classic traffic.
>
Actually I agree, and avoids potentially obsoleting ECT(0)? - which is 
another possible way out if we have a problem, as discussed earlier this 
week.
> This seems to me like the obvious path toward consensus on an 
> experimental deployment aimed at gathering real-world experience 
> without first committing to something unproven.
>
> Kyle

I'm getting frustrated by that inference that the WG has somejow failed 
to deliver something. What is unproven? - Is it that the advocates who 
wanted a backwards-compatibility mode have failed to produce a method? 
data on ECT(0) marking? - or where you refering to something else? 
Please be specific.

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 26 Mar 2021 15:59:42 +0200

> On 26 Mar, 2021, at 2:30 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> As a second step after this, "IF" using DiffServ is an option, then L4S and SCE can get married and we can benefit from both.

In recent days I have put forward two distinct ways to use Diffserv in this context, both of which I believe are workable.  The one I prefer is of course the one involving SCE-type signalling rather than L4S-type signalling; it consumes fewer DSCPs and has fewer failure modes.

In the SCE context, Diffserv is useful for three things:

1: Protection of early experimental deployments.  This is temporary.

2: Enabling dual-queue AQMs to perform SCE classification and signalling.

3: Allowing SCE AQMs to distinguish SCE and non-SCE traffic embedded in the same tunnel.

SCE itself is compatible with existing dumb FIFOs, policers, dropping AQMs, and RFC-3168 AQMs both single and multi-queued, without the assistance or need for any Diffserv mechanism.  Most SCE AQMs will probably have AF or FQ functionality to ensure good compatibility with conventional traffic.

Diffserv is strictly an enhancement to SCE, which can be deployed in those contexts which benefit from it.  The CDN -> ISP -> subscriber path is an ideal context for deploying a coherent Diffserv system.  I think that dovetails very nicely with the technical needs of DOCSIS Low Latency.

 - Jonathan Morton

---

From: Martin Duke <[email-masked]>
Date: Fri, 26 Mar 2021 10:15:06 -0700

Speaking as an individual,

Bob has identified all of the problems with using DSCP for L4S. Among them
is running L4S through a tunnel.

Another axis of discussion was changing the L4S signal to be ECT(1)->ECT(0)
and preserving the CE signal (thread: [1]), which comprehensively solves
the coexistence issues. But this fell down, partially because of the
(needless) RFC6040 tunnel decap behavior, which will revert any
outer-header changes. There are some other problems, of course, they don't
affect bystanders and I don't see them as likely to be fatal.

Honestly, if we're going to break L4S in tunnels anyway, I would just as
soon break it with the semantic change and not take all the other DSCP
baggage that Bob describes. We could even bis the mistake in 6040 so that
"eventually" the needless behavior goes away.

This is not a statement in favor of 1->0 marking. It is a statement that if
the WG is going to insist on a DSCP to ship this design, I would prefer
1->0 over that.

Sorry to complicate the discussion.

Martin

[1] https://mailarchive.ietf.org/arch/msg/tsvwg/TDMKRyH9E6zho6VkiXDYes7FZK0/

On Fri, Mar 26, 2021 at 7:00 AM Jonathan Morton <[email-masked]>
wrote:

> > On 26 Mar, 2021, at 2:30 pm, De Schepper, Koen (Nokia - BE/Antwerp) <
> details of a DiffServ-based proposals, I'm taking a step back asking: Is
> using DiffServ an option at all. I don't believe so (see other reply to
> Steve's mail related to the reason for deployment).
> >
> > As a second step after this, "IF" using DiffServ is an option, then L4S
> and SCE can get married and we can benefit from both.
>
> In recent days I have put forward two distinct ways to use Diffserv in
> this context, both of which I believe are workable.  The one I prefer is of
> course the one involving SCE-type signalling rather than L4S-type
> signalling; it consumes fewer DSCPs and has fewer failure modes.
>
> SCE itself is compatible with existing dumb FIFOs, policers, dropping
> AQMs, and RFC-3168 AQMs both single and multi-queued, without the
> assistance or need for any Diffserv mechanism.  Most SCE AQMs will probably
> have AF or FQ functionality to ensure good compatibility with conventional
> traffic.
>
> Diffserv is strictly an enhancement to SCE, which can be deployed in those
> contexts which benefit from it.  The CDN -> ISP -> subscriber path is an
> ideal context for deploying a coherent Diffserv system.  I think that
> dovetails very nicely with the technical needs of DOCSIS Low Latency.
>
>  - Jonathan Morton
>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 26 Mar 2021 17:34:34 +0000

Hi Sebastian,

To be clear: I assume (didn't bother to dive into the details) your proposal works for the purpose of what you intend to do: block L4S everywhere except for the paths that are explicitly allowing it. 

My first point is that requiring to set a DiffServ codepoint alone would already support that.

Second clarification on your quote: 
>> "Also I do not accept for one minute that you are actually willing to entertain this as a real option."
I don't have any objections to the added value that SCE would bring. I'm a fan of it 😊. For me this is not a religious war against SCE!!! Unfortunately SCE would also need a DiffServ (or other) L4S-ID to make it work in all cases. My proposal (which Jonathan acked by the way) was to combine it in that case. "IF" DiffServ is seriously considered to not being a show stopper we should use SCE as congestion mark and DiffServ as L4S-ID. I'm also in favor for the best technical solution, but it needs to have a realistic deployment chance...

Then the second point I tried to make is that I think DiffServ will largely block the deployment of L4S and/or SCE from the start. I know we could see this as a feature (like you do, I still assume well intended), but I think it will be its destruction/decline from the start. My fear is that this will cause the interest of investing in L4S/SCE from end-systems to drop to zero. Would you buy a car, if you would not be permitted to drive it from your driveway?

And why? Because there might be some classic ECN single Q bottlenecks that might get congested long enough by L4S flows and cause noticeable rate reductions for ECT(0) users? I don't know what your Internet experience is, but this is my everyday's reality, definitely working remotely from home now. To extend the analogy: would you leave your car on the driveway not using it, because you know there is a chance that you might hurt, even kill some innocent bystander by accident?

I wonder if there is some (accessible) L4S network support out there, who would want to use ECT(0) still? There is much more to win with L4S than with classic ECN.

Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Friday, March 26, 2021 2:10 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; Bob Briscoe <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

Hi Koen,

Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 26 Mar 2021 18:07:42 +0000

I’m open for discussing alternatives, but I think we need first to decide whether using DiffServ is a feature or suicide…

I’m afraid it will be suicide… I don’t think enabling end-to-end DiffServ pass-through is under control of a single party. So at the end L4S will be restricted to single operational domains. Of course allowing L4S without DiffServ won’t stop anyone using DiffServ.

Do people with experience in that field (operators) have a substantiated opinion on this?

Would application providers see using DiffServ as an obstacle?

Koen.

From: Martin Duke <[email-masked]>
Sent: Friday, March 26, 2021 6:15 PM
To: Jonathan Morton <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; Bob Briscoe <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

Bob has identified all of the problems with using DSCP for L4S. Among them is running L4S through a tunnel.

Another axis of discussion was changing the L4S signal to be ECT(1)->ECT(0) and preserving the CE signal (thread: [1]), which comprehensively solves the coexistence issues. But this fell down, partially because of the (needless) RFC6040 tunnel decap behavior, which will revert any outer-header changes. There are some other problems, of course, they don't affect bystanders and I don't see them as likely to be fatal.

Honestly, if we're going to break L4S in tunnels anyway, I would just as soon break it with the semantic change and not take all the other DSCP baggage that Bob describes. We could even bis the mistake in 6040 so that "eventually" the needless behavior goes away.

This is not a statement in favor of 1->0 marking. It is a statement that if the WG is going to insist on a DSCP to ship this design, I would prefer 1->0 over that.

Martin

On Fri, Mar 26, 2021 at 7:00 AM Jonathan Morton <[email-masked]<mailto:[email-masked]>> wrote:
> On 26 Mar, 2021, at 2:30 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]<mailto:[email-masked]>> wrote:
>

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 21:05:56 +0100

Hi Koen,

> On Mar 26, 2021, at 18:34, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> To be clear: I assume (didn't bother to dive into the details)

	[SM] It would considerably improve the quality of our discussion, if you would actually care enough to read my arguments. How are we to make any progress one way or the other, if you do not even give me to coutesy of hearing me out?

> your proposal works for the purpose of what you intend to do: block L4S everywhere except for the paths that are explicitly allowing it. 

	[SM] So to summarize, a considerably body of lab measurement indicate that nrither L4S's AQM nor its reference protocol are ready for testing over the open internet. The AQM is prone to throttle non-L4S flows close to starvation, and without an L4S AQM TCP Prague get easily driven close to starvation in traditional FIFO bottlenecks. In rfc3168 bottlenecks however TCP Prague will drive conventional traffic close ot starvation.
	All of this is known and well documented in the lab already. Now, you insist for further progress you need to run experiments over the existing internet with all its variable RTT paths and FIFO bottlenecks. I predict that these experiments are going to cause pain rto quite a number of parties that do not consent to the experiment and are unlikely to profit from it in any shape or form.
	My compromise proposal to bring your desire for internet scale experiments and safety for the rest of the internet together is simply to require you contain potential the fall-out of your experiment to consenting parties.
	Ideally, you would first go and fix design and implementation of L4S to not have such glaringly obvious failure modes, but I keep bringing the same issues up year after year only to see no substantial change for the better...
	Tl;dr: Yes the gist of my proposal is to contain the benefits and risks of the experiment to willing participants, something I would not have thought to be controversial, but simplest rules of prudent engineering.

> My first point is that requiring to set a DiffServ codepoint alone would already support that.

	[SM] Well, sure, but you came to the ECT(1) decision for a reason (a reason I do not agree with, but that is a different topic, I am trying to build you a bridge here how to run your experiments without taking the rest of the internet down), so I am not going to argue about that. My proposal adds an GUARD DSCP for the duration of the EXPERIMENT to allow your measurements in live production networks without negative side-effects for non-participants. This layer of safety can be removed if the experiment has run ist course and measurements in consenting networks has provided data that L4S is safe without the DSCP. At which point the DSCP could be dropped, try doing that if you only use a DSCP as L4S classifier... If ou still have questions about this, please just ask, instead of simply assuming what I meant.

> I don't have any objections to the added value that SCE would bring.

	[SM] Again, that seems orthogonal to me proposal, I did not really popose to replace the problematic L4S identifier here. Really, I am trying to help you get your experiments on the road, without taking down my internet.

> I'm a fan of it 😊. For me this is not a religious war against SCE!!!

	[SM] Great for me neither.

> Unfortunately SCE would also need a DiffServ (or other) L4S-ID to make it work in all cases. My proposal (which Jonathan acked by the way) was to combine it in that case. "IF" DiffServ is seriously considered to not being a show stopper we should use SCE as congestion mark and DiffServ as L4S-ID. I'm also in favor for the best technical solution, but it needs to have a realistic deployment chance...

	[SM] Which is again not about my proposal. Let's stick to that please, I am not requesting you change your signalling method, and you are not really open to do so anyway, so let's stick to where we might find common ground?

> Then the second point I tried to make is that I think DiffServ will largely block the deployment of L4S and/or SCE from the start. I know we could see this as a feature (like you do, I still assume well intended),

	[SM] To perform a controlled experiment you need knowledge about the parts involved, the feature is not to keep L4S restricted to as few ASs as possible, but to make sure that where L4S is experimented with there is proper containmeint. I have heard no counter proposal with similar containment properties (or any actually).

> but I think it will be its destruction/decline from the start.

	[SM] How that? Chatter in the WG tells me that there are several players that are already lining up to run L4S experiments as soon as the RFCs are ratified. Are you telling me that none of these parties will actually do so if we require a tiny bit of DSCP gymnastics to get the experiment started? Really?
	Let me ask again, is the interest in the L4S that conditional that even such small requirements will doom the experiment? And I am talking abou the experiment here, not a mass deployment, because a mass deployment is NOT an experiment...

> My fear is that this will cause the interest of investing in L4S/SCE from end-systems to drop to zero. Would you buy a car, if you would not be permitted to drive it from your driveway?

	[SM] Wrong analogy, we are at the stage where our prototype car needs to be taken to test drives under real road conditions, an Erlkoenig if you will, or a FUT (friendly user test). Let's walk before we try to run please.

> 
> And why? Because there might be some classic ECN single Q bottlenecks that might get congested long enough by L4S flows and cause noticeable rate reductions for ECT(0) users?

	[SM] THat is one part of the problem, the other problem is the massive bias for both L4S over classic and short RTTs over longer RTTs that make current L4S design and implementation not a good fit for the existing internet.

> I don't know what your Internet experience is, but this is my everyday's reality, definitely working remotely from home now. To extend the analogy: would you leave your car on the driveway not using it, because you know there is a chance that you might hurt, even kill some innocent bystander by accident?

	[SM] Are you using L4S at home right now? If not, why not, and what ese are you using. I offer to help install and configure OpenWrt with SQM to at least get you first hand experience with what I consider the status quo of low latency on the internet. Will it be 99.999% < 1 ms, ure as hell no, but I am confident it will improve your current set-up. Just holler or come over to the OpenWrt forum, where we help out often and happily.

> I wonder if there is some (accessible) L4S network support out there, who would want to use ECT(0) still? There is much more to win with L4S than with classic ECN.

	[SM] How that? For state of the art SQM all you need is to configure your home router and you are improving your latency under load experience massively without having to change transport protocols. Switch to L4S and effectively you are running a single queue modified PIE that will in all likelihood be less useful that say fq_codel. To actually gain something with L4S you will first need to deploy L4S-aware protocols on both ends of your connections. Why are we even having such surreal discussions? SQM exists today works wonders in todays internet versus PIE in the sky L4S (see what I did there) which might work even better some day in the future (if somebody actually bothers to cobbe together a competitive L4S protocol, because TCP Prague ain't).

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 26 Mar 2021 21:16:06 +0100

Hi Koen,

> On Mar 26, 2021, at 19:07, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

	[SM] I do not think that this phrasing is helpful. The proposals on the table are:
a) re-jig L4S' signaling to re-balanca the inherent safety of L4S
b) making sure that the existing L4S signaling can safely be experimented with in the existing internet

I can see, why you would not want to do a), but liking to B) suicide is hyperbole.

>  
> I’m afraid it will be suicide… I don’t think enabling end-to-end DiffServ pass-through is under control of a single party.

	[SM] Exactly, that is one of the tenets of my proposal, require that any L4S path needs to be negotiated, so that the networks involved can make sure that the required safety mechanisms are in place. So that is hardly an argument agsainst my guard DSCP proposal, unless inflicting L4S' side-effects on innocent bystanders is the goal of the experiment...

> So at the end L4S will be restricted to single operational domains.

	[SM] Ever heard of SLAs and TCAs? Differserv domains can and do negotiate safe passage of DSCPs or PHB already, so that operational knowledge and process probably can be leveraged an used to span safe L4S test domains spanning multiple AS.

> Of course allowing L4S without DiffServ won’t stop anyone using DiffServ.

	[SM] And does nothing to guarantee the safety of the L4S experiment, so 

> Would application providers see using DiffServ as an obstacle?

	[SM] Let me add the following refinement of the question (because any additional requirement is an "obstacle" the question is rather is this a show-stopper):
Would you parytcipate in the L4S experiment without a required DSCP but would pass if a guard DSCP would be required? And if so, how would you propose to keep the experiment under containment in your live production network instead?

Best Regards
	Sebastian

> On Fri, Mar 26, 2021 at 7:00 AM Jonathan Morton <[email-masked]> wrote:

---

From: "C. M. Heard" <[email-masked]>
Date: Fri, 26 Mar 2021 20:00:00 -0700

On Fri, Mar 26, 2021Sebastian Moeller wrote:

> On Mar 26, 2021, at 18:34, De Schepper, Koen wrote:
>  [SM] Well, sure, but you came to the ECT(1) decision for a reason (a
> reason I do not agree with, but that is a different topic, I am trying to
> build you a bridge here how to run your experiments without taking the rest
> of the internet down), so I am not going to argue about that. My proposal
> adds an GUARD DSCP for the duration of the EXPERIMENT to allow your
> measurements in live production networks without negative side-effects for
> non-participants. This layer of safety can be removed if the experiment has
> run ist course and measurements in consenting networks has provided data
> that L4S is safe without the DSCP. At which point the DSCP could be
> dropped, try doing that if you only use a DSCP as L4S classifier.

This point sure seems to have been lost in much of the discussion. Note
again: GUARD DSCP for the duration of the EXPERIMENT. NOT DSCP instead of
ECT(1).

I've read several comments objecting that using DSCP to distinguish L4S
traffic is undeployable end-to-end on the general Internet. Well, according
to the intended status of the drafts, L4S is starting life as an
EXPERIMENT, and as such is not expected (or at least should not be
expected) to be deployed on the general Internet. If the WG's desire is to
have L4S deployed on the general Internet as soon as the specs are
approved, then the WG should take Steven Blake's advice and
deprecrate/obsolete RFC 3168 and change the intended status of the drafts,
or else use a mechanism that is backward compatible with RFC 3168, such
Jake Holland's proposal to use ECT(1) -> ECT(0) as the L4S congestion
signal, leaving the semantics of CE unchanged.

Mike Heard

---

From: Pete Heist <[email-masked]>
Date: Sat, 27 Mar 2021 14:33:01 +0100

...

On Fri, 2021-03-26 at 08:54 -0400, Kyle Rose wrote:
> <[email-masked]> wrote:

I agree- the topic of this thread is the use of a guard DSCP for
experiment. Sorry if I contributed to mixing in other possible uses
early on.

> universal deployment (i.e., removing the DSCP guard) if the
> experiment is successful without requiring standardization and
> rollout of end-to-end DiffServ. And it does so (a) without
> permanently burning the ECT(1) code point if the experiment fails,
> and (b) in an opt-in manner that greatly reduces the potential for
> unintended side effects on classic traffic.
> 

Although an RFC isn't technically required to do this, it should still
serve to codify how DSCP should be used and treated, and also to raise
visibility as a sanctioned IETF experiment. After a successful
experiment at "operator scope" (if that terminology captures it, but
the larger the better), it should be easier to justify and start an
experiment at Internet scope.

I also think that with codepoint space this tight, it's not just about
verifying safety, but also effectiveness. I feel the same way even for
proposals that fall under RFC4774 Option 3 ("Friendly Coexistence with
Competing Traffic").

Pete

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 27 Mar 2021 20:00:08 +0000

Hi Pete, Kyle, Charles, Sebastian, Jonathan

CC others

I have followed this discussion for a while now. 

It is possible in a 5G system to set up traffic filters that include 5-tuples and DSCP + more, so what you propose is possible. 

But with that said, I don’t at all see that this needs standardization and why this would be needed as part of the L4S work. In addition, your  intended purpose with such an guard DSCP is quite likely to be met with confusion among 3GPP operators as there is no evidence of RFC3168 AQM in 5G systems. 

I only see that this discussion goes at length to protect classic ECN enabled traffic in the possible cases where network nodes implement RFC3168 AQMs (or FQ-CoDel) and omits the fact that these are eventually either upgraded or replaced over time. 

IANA already lists ECT(1) as intended for experimental purposes only so I would say that it is a bad idea long term to continue to claim ECT(1) for RFC3168 AQMs. 

/I

From: tsvwg <[email-masked]> On Behalf Of Pete Heist
Sent: den 27 mars 2021 14:33
To: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] L4S DSCP (was: L4S drafts: Next Steps)

...

On Fri, Mar 26, 2021, 8:30 AM De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked] <mailto:[email-masked]> > wrote:

I understood the goal of your proposal. But before diving into the details of a DiffServ-based proposals, I'm taking a step back asking: Is using DiffServ an option at all.

Why wouldn't it be? The point of the proposal AIUI is to require networks to take action to explicitly opt-in to this experiment. Given the DiffServ bleaching that occurs by default at network boundaries, operators that take no action will not find themselves ambushed by novel behavior.

I agree- the topic of this thread is the use of a guard DSCP for experiment. Sorry if I contributed to mixing in other possible uses early on.

This approach allows for experimenting with ECT(1) as a classifier in a safe (contained, time-limited) way, thus allowing a path toward universal deployment (i.e., removing the DSCP guard) if the experiment is successful without requiring standardization and rollout of end-to-end DiffServ. And it does so (a) without permanently burning the ECT(1) code point if the experiment fails, and (b) in an opt-in manner that greatly reduces the potential for unintended side effects on classic traffic.

This seems to me like the obvious path toward consensus on an experimental deployment aimed at gathering real-world experience without first committing to something unproven.

Although an RFC isn't technically required to do this, it should still serve to codify how DSCP should be used and treated, and also to raise visibility as a sanctioned IETF experiment. After a successful experiment at "operator scope" (if that terminology captures it, but the larger the better), it should be easier to justify and start an experiment at Internet scope.

I also think that with codepoint space this tight, it's not just about verifying safety, but also effectiveness. I feel the same way even for proposals that fall under RFC4774 Option 3 ("Friendly Coexistence with Competing Traffic").

Pete

Kyle

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 27 Mar 2021 22:34:38 +0200

> On 27 Mar, 2021, at 10:00 pm, Ingemar Johansson S <[email-masked]> wrote:
> 
> In addition, your intended purpose with such an guard DSCP is quite likely to be met with confusion among 3GPP operators as there is no evidence of RFC3168 AQM in 5G systems.

To paraphrase Barnes Wallis:  if you told them that it was to protect *other* networks from the effects of L4S traffic, do you think that might help?

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 28 Mar 2021 17:12:41 +0200

Hi Ingemar,

> On Mar 27, 2021, at 21:00, Ingemar Johansson S <[email-masked]> wrote:
> 

	[SM] Excellent news, thanks for this information.

> But with that said, I don’t at all see that this needs standardization and why this would be needed as part of the L4S work.

	[SM] A guard DSCP would need to be unambiguously agreed on by all participants (or rather a PHB, different DSCPs could take that role). IMHO a standards document is exactly the place to lay down the "law" on how to do this. And in addition the L4S-ops ID/RFC should spell this out in detail to facilitate such an approach. I wonder how issues affecting interoperability between entities are regulated on the 3GPP world, but I naively assume that this is handled in standards documents, no?

> In addition, your  intended purpose with such an guard DSCP is quite likely to be met with confusion among 3GPP operators as there is no evidence of RFC3168 AQM in 5G systems. 

	[SM] Hmm, I am puzzled why you see it a problem to explain in detail to them what risk an unguarded L4S deployment will carry, but let me help you out here. As long as 3GPP operators can guarantee that L4S packets never ever leave their domain, they are already free to use what ever method they want, the do not need the IETF's blessing to use L4S. I assume however, that data packets carried over 3GPP member networks, might "occasionally" actually get sent to end-hosts outside of the operators network. It is that second case in which containment/informed consent becomes important. Why, because L4S traffic (as seen in TCP Prague) behaves considerably different under a number of conditions quite common on the existing internet, like:
a) running rough-shod over other glows in shared rfc3168 queues,
b) introducing a hitherto unseen level of RTT bias which will significantly change the sharing behaviour between short RTT (i.e. eyeball<->CDN) and longer RTT flows, with RTT distributions over the internet well within the range from single low double digits, to well above 100ms this is an important change that might require operator opt-in.
c) introducing an inherent disadvantage for existing standards compliant traffic versus the new comer, which also might require operator opt-in.

Especially b) and c) seem like POLICY issues, that operator will likely not accept getting dictated from an RFC, but consider to be under their sole direction?

 I guess you might direct confused operators to the tsv mailing list, where we can discuss such issues and patiently will explain the risk of the L4S experiment as well as possible mitigation strategies.

> I only see that this discussion goes at length to protect classic ECN enabled traffic in the possible cases where network nodes implement RFC3168 AQMs (or FQ-CoDel) and omits the fact that these ar…

	[SM] Well, if you are wiling to WAIT with the L4S experiment UNTIL these are upgraded and/or replaced that sounds like a viable plan... if however you want an earlier experiment, maybe come up with a counter-proposal for a mitigation strategy that will also allow willing parties to participate in the experiment without force-recruiting those parts of the internet into the experiment that have not opted in. I do not claim my sketch is complete or without challenges, or that it the best possible mitigation/containment strategy, but so far we only have two proposals on the table, and mine seems less ambitious (and I believe fixing L4S' issues for good, like in Jonathan's proposal would be better, I see no way forward, where L4S proponents will agree to that, fallacy of sunk cost and all)...

> IANA already lists ECT(1) as intended for experimental purposes only so I would say that it is a bad idea long term to continue to claim ECT(1) for RFC3168 AQMs. 

	[SM] If the RFCs are changed that is a valid stance for the future, but will not help with legacy deployment, will it?  Then again, if you are willing to give notice now and wait for, say, a decade for deployments to change to the new mode, I see no issue with that. But then that would mean the L4S experiment would need to wait just as long (no objections from my side ;) ).
	To repeat, my proposal of using a guard DSCP only serves the purpose of allowing an L4S experiment in live production networks in the near future. If L4S aims for the long game with deployment in 10 years, we are probably better off with actually fixing its trouble spots for good ;)

Best Regards
	Sebastian

> On Fri, Mar 26, 2021, 8:30 AM De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 28 Mar 2021 19:01:45 +0000

Hi Sebastian+Jonathan + others

See inline [IJ] 
/Ingemar

> Sent: den 28 mars 2021 17:13
> To: Ingemar Johansson S <[email-masked]>
> Cc: Pete Heist <[email-masked]>; Kyle Rose <[email-masked]>; C. M.
> Heard <[email-masked]>; Jonathan Morton <[email-masked]>;
> De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-
> labs.com>; tsvwg IETF list <[email-masked]>
> > On Mar 27, 2021, at 21:00, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > Hi Pete, Kyle, Charles, Sebastian, Jonathan CC others
> >
> > It is possible in a 5G system to set up traffic filters that include 5-tuples and
> DSCP + more, so what you propose is possible.
> 
> > But with that said, I don’t at all see that this needs standardization and why
> this would be needed as part of the L4S work.
> 
> 	[SM] A guard DSCP would need to be unambiguously agreed on by all
> participants (or rather a PHB, different DSCPs could take that role). IMHO a
> standards document is exactly the place to lay down the "law" on how to do
> this. And in addition the L4S-ops ID/RFC should spell this out in detail to
> facilitate such an approach. I wonder how issues affecting interoperability
> between entities are regulated on the 3GPP world, but I naively assume that
> this is handled in standards documents, no?
[IJ] Like said before by others, if DSCP would have been this easy to use, then we would have had this discussion. Operators can use DSCP in L4S experiments but that of course requires that they see a specific benefit with this. So far I have not seen anybody except you fighting so hard for this and that should tell a little about the magnitude of the problem you are trying to solve.  

> 
> 
> > In addition, your  intended purpose with such an guard DSCP is quite likely
> to be met with confusion among 3GPP operators as there is no evidence of
> RFC3168 AQM in 5G systems.
> 
> 	[SM] Hmm, I am puzzled why you see it a problem to explain in detail
> to them what risk an unguarded L4S deployment will carry, but let me help
> you out here. As long as 3GPP operators can guarantee that L4S packets
> never ever leave their domain, they are already free to use what ever
> method they want, the do not need the IETF's blessing to use L4S. I assume
> however, that data packets carried over 3GPP member networks, might
> "occasionally" actually get sent to end-hosts outside of the operators
> network. It is that second case in which containment/informed consent
> becomes important. Why, because L4S traffic (as seen in TCP Prague)
> behaves considerably different under a number of conditions quite common
> on the existing internet, like:
> b) introducing a hitherto unseen level of RTT bias which will significantly
> change the sharing behaviour between short RTT (i.e. eyeball<->CDN) and
> longer RTT flows, with RTT distributions over the internet well within the
> range from single low double digits, to well above 100ms this is an important
> change that might require operator opt-in.
> c) introducing an inherent disadvantage for existing standards compliant
> traffic versus the new comer, which also might require operator opt-in.
> 
> Especially b) and c) seem like POLICY issues, that operator will likely not
> accept getting dictated from an RFC, but consider to be under their sole
> direction?
[IJ] See below about the use of the ECT(1) code point

> 
>  I guess you might direct confused operators to the tsv mailing list, where we
> can discuss such issues and patiently will explain the risk of the L4S
> experiment as well as possible mitigation strategies.
> 
> 
> >
> > I only see that this discussion goes at length to protect classic ECN enabled
> traffic in the possible cases where network nodes implement RFC3168 AQMs
> (or FQ-CoDel) and omits the fact that these are eventually either upgraded
> or replaced over time.
> 
> 	[SM] Well, if you are wiling to WAIT with the L4S experiment UNTIL
> these are upgraded and/or replaced that sounds like a viable plan... if
> however you want an earlier experiment, maybe come up with a counter-
> proposal for a mitigation strategy that will also allow willing parties to
> participate in the experiment without force-recruiting those parts of the
> internet into the experiment that have not opted in. I do not claim my sketch
> is complete or without challenges, or that it the best possible
> mitigation/containment strategy, but so far we only have two proposals on
> the table, and mine seems less ambitious (and I believe fixing L4S' issues for
> good, like in Jonathan's proposal would be better, I see no way forward,
> where L4S proponents will agree to that, fallacy of sunk cost and all)...
> 
> 
> > IANA already lists ECT(1) as intended for experimental purposes only so I
> would say that it is a bad idea long term to continue to claim ECT(1) for
> RFC3168 AQMs.
> 
> 	[SM] If the RFCs are changed that is a valid stance for the future, but
> will not help with legacy deployment, will it?  Then again, if you are willing to
> give notice now and wait for, say, a decade for deployments to change to the
> new mode, I see no issue with that. But then that would mean the L4S
> experiment would need to wait just as long (no objections from my side ;) ).
[IJ] Even legacy equipment is upgraded every once in a while or gets replaces because it gets too old or just breaks down. 

> 	To repeat, my proposal of using a guard DSCP only serves the
> purpose of allowing an L4S experiment in live production networks in the
> near future. If L4S aims for the long game with deployment in 10 years, we
> are probably better off with actually fixing its trouble spots for good ;)
[IJ] Collected answer to all the above. All I see is that this RFC3168 AQM is brought up repeatedly, when prompted about the possibility that even RFC3168 AQMs can be updated (the obvious first step is to make these follow https://www.iana.org/assignments/dscp-registry/dscp-registry.xml ) , it is only met with all kinds of arguments like we don't do this until L4S is deemed safe for deployment and we wont allow the experiment to happen until [TSVWG formally says whatever...], this is probably as close to a catch 22 situation that you can get in IETF. 
This is to me just smokes and mirrors with FUD as a main ingredient and it is getting harder and harder to keep  a serious discussion.

> > I understood the goal of your proposal. But before diving into the details of
> a DiffServ-based proposals, I'm taking a step back asking: Is using DiffServ an
> option at all.
> >
> > I agree- the topic of this thread is the use of a guard DSCP for experiment.
> Sorry if I contributed to mixing in other possible uses early on.
> >
> > This approach allows for experimenting with ECT(1) as a classifier in a safe
> (contained, time-limited) way, thus allowing a path toward universal
> without requiring standardization and rollout of end-to-end DiffServ. And it
> does so (a) without permanently burning the ECT(1) code point if the
> experimental deployment aimed at gathering real-world experience without
> first committing to something unproven.
> >
> > Although an RFC isn't technically required to do this, it should still serve to
> codify how DSCP should be used and treated, and also to raise visibility as a
> sanctioned IETF experiment. After a successful experiment at "operator
> scope" (if that terminology captures it, but the larger the better), it should be
> easier to justify and start an experiment at Internet scope.
> >
> > I also think that with codepoint space this tight, it's not just about verifying
> safety, but also effectiveness. I feel the same way even for proposals that
> fall under RFC4774 Option 3 ("Friendly Coexistence with Competing Traffic").
> >
> > Pete
> >
> > Kyle

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 28 Mar 2021 22:33:53 +0300

> On 28 Mar, 2021, at 10:01 pm, Ingemar Johansson S <[email-masked]> wrote:
> 
> All I see is that this RFC3168 AQM is brought up repeatedly, when prompted about the possibility that even RFC3168 AQMs can be updated…

The point here is that even if *some* existing AQMs can be updated easily, there will be some which cannot or will not.  These are what you must pessimistically take into account when analysing the risk of deploying L4S.  Until you absorb that knowledge and start taking it into account, you're not going to make any headway.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 28 Mar 2021 20:03:06 +0000

Hi
See below [IJ]

> From: Jonathan Morton <[email-masked]>
> Sent: den 28 mars 2021 21:34
> Cc: Sebastian Moeller <[email-masked]>; Pete Heist <[email-masked]>; Kyle
> Rose <[email-masked]>; C. M. Heard <[email-masked]>; De Schepper,
> Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg
> IETF list <[email-masked]>
> > On 28 Mar, 2021, at 10:01 pm, Ingemar Johansson S
> > All I see is that this RFC3168 AQM is brought up repeatedly, when prompted
> about the possibility that even RFC3168 AQMs can be updated…
> 
> The point here is that even if *some* existing AQMs can be updated easily,
> there will be some which cannot or will not.  These are what you must
> pessimistically take into account when analysing the risk of deploying L4S.  Until
> you absorb that knowledge and start taking it into account, you're not going to
> make any headway.

[IJ] And these that "can or will not.." will then continue to run across the internet 10 years from now ?, 20 years ?.. Network gear is constantly being replaced with the reason being that they either become too old to be economically feasible to maintain or that they just break down. Home gateways breakdown or are just been replaced because they don't meet new demands. You don't normally keep a PC for more than ~5 years for a bunch of reasons. Why would you keep networking gear for decades ?

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 28 Mar 2021 22:04:01 +0200

Hi Ingemar,

> On Mar 28, 2021, at 21:01, Ingemar Johansson S <[email-masked]> wrote:
> 
> [IJ] Like said before by others, if DSCP would have been this easy to use, then we would have had this discussion.

	[SM] Getting an experiment with known negative side-effects grated by the IETF does not hinge upon "ease of use" or convenience to the experimenter. If you have a better proposal how to secure/contain L4S for the duration of the experiment, I am all ears. Just not bothering for the safety of the existing internet however, is IMHO not an option, sorry.

> Operators can use DSCP in L4S experiments but that of course requires that they see a specific benefit with this. So far I have not seen anybody except you fighting so hard for this and that should tell a little about the magnitude of the problem you are trying to solve.  

	[SM] Or about the lack of fore-sight in those that think deploying existing L4S without any safe guards onto the existing internet. I have a pretty clear prediction about how the L4S experiment is going to proceed unless properly contained, and I want to save us all unnecessary hassle to post-hoc explain how this WG could allow such unfinished engineering to be released upon the rest of the internet... I am a bit puzzled, that after all the data that clearly shows L4S already failing hard in the lab, you consider an "experimental" deployment without any safety mechanisms as advisable...

> [IJ] Even legacy equipment is upgraded every once in a while or gets replaces because it gets too old or just breaks down. 

	[SM] Yes, are you willing to put a number on that, and more importantly, are you willing to hold the L4S experiment until that point? I do not consider that to be a proposal that is on the table, not from you, not from the rest of team L4S, nor the wider WG. So let's please come back to the concrete proposal how to get your experiment started, okay?

> [IJ] Collected answer to all the above. All I see is that this RFC3168 AQM is brought up repeatedly, when prompted about the possibility that even RFC3168 AQMs can be updated (the obvious first ste…

	[SM] Nobody is stopping you from sending patches e.g. to the Linux kernel mailing list, so if you consider this to be an important point, you now the address to send your patches to. The power to resolve that catch22 is in your hand, which is to say, it is not a catch 22 situation at all.

> This is to me just smokes and mirrors with FUD as a main ingredient and it is getting harder and harder to keep  a serious discussion.

	[SM] Ironically, this argument comes as response to a concrete proposal how to get the L4S experiment air-borne. Maybe you re-rad my proposal and come back to the discussion without the assumption I would be out to get you/L4S? This is about how to contain the predicted fall-out of the experiment to those parties that have accepted that fall-out might happen and (hopefully) are prepared to clean up if something goes wrong. You know, really just standard cautious engineering practices.

@chairs, what am I to make out of the fact, that none of the L4S proponents seems to find the time to actually read my modest sketch of a proposal and discuss it on its merits?
Against my better judgment about the quality and readiness of L4S, I am trying hard to build a bridge, and all I receive in return is gross generalizations...

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 28 Mar 2021 22:08:20 +0200

Hi ingemar,

let's take your "let it age out" proposal at face value. How long do you estimate will it take for those problematic nodes to age out of the system after the IETF has signaled that rfc3168 has changed?
And let's assume you come up with a number of X years, are you really willing to put the L4S experiment on hold until that time has passed?

Unless the answer to that question is YES, let's just drop this proposal, okay? This is not a path to a productive discussion/outcome.

Best Regards
	Sebastian

> On Mar 28, 2021, at 22:03, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> See below [IJ]
>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 29 Mar 2021 00:30:01 +0300

> On 28 Mar, 2021, at 11:03 pm, Ingemar Johansson S <[email-masked]> wrote:
> 
> And these that "can or will not.." will then continue to run across the internet 10 years from now ?, 20 years ?

Yes.  Exactly.

I mentioned before that Pete's Czech ISP is still running 15-year-old routers that date from their founding.  I personally still have a "UFO" Apple Airport Base Station - the original one with 802.11b, 56-bit WEP, and a 56K analogue modem - which doesn't get much *use* these days, but it still *works* and is potentially useful.  Heck, I still write code for 40-year-old computers as a hobby.

These old devices lack support for newer specifications.  But they still work just fine in an environment that does use them, because backwards compatibility was designed in when the new specifications were published.  ECN Capable transports still work with dumb drop-tail FIFOs, and Ethernet frames carrying IPv6 packets still traverse Layer 2 devices that were in the shops before IPv6 was finalised, and which require IPv4 (or maybe AppleTalk) packets to configure the settings.  That is how Internet engineering works.  In general, old network devices are expected to continue working as before, and you only need to upgrade if you want to benefit from the new technology.

But L4S is *not* designed that way.  If it is deployed as-is without effective guarding, it will break quite a lot of stuff that isn't even very old.  Your best proposal is to scrap or materially change an existing specification that is currently rapidly deploying in the Internet because people see real benefits from it.  It's entirely conceivable that Pete's Czech ISP will choose this year to finally upgrade those old routers, *specifically* for the purpose of deploying fq_codel, now that they have some experience with it - and that would start a new 15-year timer, wouldn't it?

I've personally put forward two different ways to make L4S safely deployable in the past week - one which requires only minimal technical changes to at least try out, and one which changes much more but which I think will work better overall.  As before, it looks like those suggestions are being rejected out of hand (though Koen's attitude appears to be more balanced, for a change).  This is precisely what has delayed L4S for two years and counting, and there is no end in sight.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Sun, 28 Mar 2021 17:37:05 -0400

On Sun, 2021-03-28 at 19:01 +0000, Ingemar Johansson S wrote:

> [IJ] Collected answer to all the above. All I see is that this
> RFC3168 AQM is brought up repeatedly, when prompted about the
> possibility that even RFC3168 AQMs can be updated (the obvious first
> step is to make these follow 
> https://www.iana.org/assignments/dscp-registry/dscp-registry.xml ) ,
> it is only met with all kinds of arguments like we don't do this
> until L4S is deemed safe for deployment and we wont allow the
> experiment to happen until [TSVWG formally says whatever...], this is
> probably as close to a catch 22 situation that you can get in IETF. 
> This is to me just smokes and mirrors with FUD as a main ingredient
> and it is getting harder and harder to keep  a serious discussion.

Operators deploying RFC 3168 ECN AQMs have absolutely no incentive to
disable or upgrade anything unless and until the IETF moves RFC 3168 to
historic and (potentially) replaces it (at PS level) with something
else. For the moment neither step is on the table. That's the catch-22.

And it is disingenuous to claim that anyone is trying to prevent the
L4S experiment from happening. People have been patiently trying to
explain how it can be conducted in a way that doesn't negatively impact
non-participants.

Regards,

// Steve

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 29 Mar 2021 01:12:24 +0100

Jonathan,

I was trying to tease apart the merits of the L4S + DSCP approach for a 
/single/ network that you were proposing. So I'd like to go back to 
before the point where Koen sent the thread off in a different direction 
(to do with the merits of a DSCP for end-to-end signalling).

Although David started this thread, I don't think he had a detailed 
proposal. And I think Sebastian's  proposal is slightly different from 
yours (using the DSCP as a classifier at an L4S AQM, which I think is 
unnecessarily restrictive). I'd like to take one step at a time and 
focus on your approach first. Inline tagged [BB]...

On 25/03/2021 16:58, Jonathan Morton wrote:
>> [BB] it depends on a string of leaps of faith (LoFs).
>> Specifically:
>>

[BB] There is no L4S design 'baked in' to existing standardization work. 
L4S is on the experimental track, and designed to build /on/ ECN 
feedback in existing standards track IETF transport protocols. 
Specifically QUIC, RTCP and TCP.

An L4S congestion control acts at the sender in one direction. The other 
direction might use a completely different congestion control. This is 
outlined in l4s-arch, but it's no different from the architecture of any 
CC. For instance, if Cubic is acting in one direction, you can have 
Vegas or BBR or whatever in the other.

You are proposing that, in a connection between host X and host Y, if X 
sends using a scalable CC, it has to set both ECT(1) and a defined L4S 
DSCP. Then when peer Y sends, it has to set the same DSCP on the packets 
it sends to X. There are a number of problems with this:
a) There is no reason or need for peer Y to know anything about L4S. So 
how does it know to set this DSCP? A host doesn't normally even check 
what the incoming DSCP is.
b) This currently would only be useful to make the L4S DSCP scheme work. 
So it would seem unlikely to be adopted in any of the busy WGs working 
on these transport protocols. Particularly given it's unorthodox 
architecture (overloading a L3 protocol with an additional L4 meaning)
c) You seem to be assuming (or perhaps requiring) that the peer at one 
end of a connection can only use a scalable CC if the other peer also 
uses a scalable CC.
d) This also means that a scalable CC cannot be tested with a Classic CC 
in the reverse direction.
e) Requiring a 2-ended deployment greatly reduces the chances that any 
flow will use L4S.

Let's say there are 10 million hosts on a network participating in the 
L4S experiment. Even once 10,000 hosts deploy L4S with your DSCP scheme, 
they will still find 9.99M hosts without L4S support. In your scheme, 
both ends have to support L4S, which is a probability of 1 in a million 
(1 in 1000 squared) that any particular flow will support L4S. That 
makes the deployment incentives awful. And it's not such a great 
experiment either with so little L4S traffic.

Because AccECN for TCP is still being held back, we will anyway have 
this 2-ended deployment problem over TCP. But we don't want to 
unnecessarily create a 2-ended deployment over the other transports 
(QUIC, RTCP, etc).

> Incorrect.  Remember that when using a DSCP as a containment mechanism, it is assumed (as I spelled out previously) that the borders of the network at least bleach that DSCP, and may even block tra…

[BB] The a/b/c examples I gave (still below) solely involved /customers/ 
attaching to a single ISP participating in the L4S experiment. If an L4S 
network bleaches at the border where its own customers attach, how can 
any L4S traffic ever enter or leave the network? I'm not understanding 
you. Specifically:
* at the first IP hop controlled by the L4S network, if it bleaches the 
L4S DSCP from attached L4S sending customers, how does any L4S traffic 
ever enter the network?
* at the last IP hop controlled by the L4S network, if it bleaches the 
L4S DSCP before forwarding to attached customers, how does any host ever 
receive L4S traffic?

Please don't answer this here. Instead, please address the specific 
examples below (a/b/c), to keep the discussion concrete.

> 1: Within the participating network.  But all RFC-3168 AQMs are supposed to have been removed from such a network before L4S was deployed on it.  So either the onus is on the network operator (who…

[BB] It is good practice to read someone else's email right through 
before you start answering it. Here, you are telling me what I told you 
further down this email, but without responding to the follow-on point I 
made about this (find "l4sops" below).

> I do not accept Appendix B.4 to be a valid set of arguments against using DSCP for this purpose.  If nothing else, it does not appear to consider any form of the two-DSCP scheme I recently describe…

[BB] I am trying to first understand the single DSCP approach you appear 
to be describing here. But I will not be able to understand you if you 
don't explain how your approach will work in the examples I give at 
a/b/c below.

And no-one is going to write your idea into a working group draft unless 
you individually address each of the criticisms of DSCP-based schemes in 
Appx B.4. You cannot seriously expect to be granted a free pass to 
ignore all the problems with DSCPs by simply pronouncing that every one 
of the arguments is invalid.

Now, please address the a/b/c points below.

> Thank you for confirming that you understand that using ECT(1) alone as the identifier suffers from all of these problems.  That was a big sticking point over the past two years, and I'm glad we ca…

[BB] We identified these potential problems before we brought L4S to the 
IETF, and at the very first ad hoc meeting about L4S (Prague Jul 2015), 
we proposed that this should be on the list of requirements that would 
need to be solved.

Here, we are putting your idea for using a DSCP to the test. So please 
address the point about how an L4S-participating network still has to 
use the techniques in l4sops, whether it adds a DSCP or not.

> No, it does not.  I encourage you to re-read that and carefully absorb my explanation of how it works.  I think you will find the exercise very enlightening.

[BB] Let's stick to one scheme at a time for now.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 29 Mar 2021 01:35:43 +0100

Mike,

Many of the arguments for why a DSCP is not useful e2e also apply when 
it is used as a guard DSCP in a single network. For instance, when an IP 
header with a DSCP is tunnelled in pipe mode, it is not propagated to 
the outer. So it won't be visible, won't be bleached, etc.

There's far too much vagueness in this thread. In the email I just sent 
to Jonathan, I asked him to address each of the critiques of a DSCP in 
Appx B.4, not just dismiss them all as invalid, which is untenable. 
Certainly, some of the critiques are not relevant to the way the DSCP is 
used in Jonathan's scheme, but many are like the tunnel issue above. So 
far, we've only just started on the basics like whether the feedback 
approach is even practical.

Bob

On 27/03/2021 03:00, C. M. Heard wrote:
>      [SM] Well, sure, but you came to the ECT(1) decision for a reason
>     (a reason I do not agree with, but that is a different topic, I am
>     trying to build you a bridge here how to run your experiments
>     without taking the rest of the internet down), so I am not going
>     to argue about that. My proposal adds an GUARD DSCP for the
>     duration of the EXPERIMENT to allow your measurements in live
>     production networks without negative side-effects for
>     non-participants. This layer of safety can be removed if the
>     experiment has run ist course and measurements in consenting
>     networks has provided data that L4S is safe without the DSCP. At
>     which point the DSCP could be dropped, try doing that if you only
>     use a DSCP as L4S classifier.
>
>
> This point sure seems to have been lost in much of the discussion. 
> Note again: GUARD DSCP for the duration of the EXPERIMENT. NOT DSCP 
> instead of ECT(1).
>
> I've read several comments objecting that using DSCP to distinguish 
> L4S traffic is undeployable end-to-end on the general Internet. Well, 
> according to the intended status of the drafts, L4S is starting life 
> as an EXPERIMENT, and as such is not expected (or at least should not 
> be expected) to be deployed on the general Internet. If the WG's 
> desire is to have L4S deployed on the general Internet as soon as the 
> specs are approved, then the WG should take Steven Blake's advice and 
> deprecrate/obsolete RFC 3168 and change the intended status of the 
> drafts, or else use a mechanism that is backward compatible with RFC 
> 3168, such Jake Holland's proposal to use ECT(1) -> ECT(0) as the L4S 
> congestion signal, leaving the semantics of CE unchanged.
>
> Mike Heard

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 29 Mar 2021 06:23:22 +0300

> On 29 Mar, 2021, at 3:12 am, Bob Briscoe <[email-masked]> wrote:

> There is no L4S design 'baked in' to existing standardization work. L4S is on the experimental track, and designed to build /on/ ECN feedback in existing standards track IETF transport protocols. Specifically QUIC, RTCP and TCP.

Well, in that case it should still be feasible to improve L4S' design and implementation before deployment.  Of course, the longer you spend resisting any and all efforts to suggest such improvements, the narrower that window of opportunity gets before the whole WG just gives up in disgust and refuses to publish.

> An L4S congestion control acts at the sender in one direction. The other direction might use a completely different congestion control. This is outlined in l4s-arch, but it's no different from the architecture of any CC. For instance, if Cubic is acting in one direction, you can have Vegas or BBR or whatever in the other.

That is indeed a case that would be problematic for the two-DSCP scheme as I presented it.  I think you could salvage it by adding explicit feedback of the relevant information to AccECN.  Or, for the purposes of the experiment, require that L4S signalling be used in both directions or not at all.  Or - and this is my preferred option - switch to a non-ambiguous signalling scheme where these complications are far less necessary.  Your choice.

Using a single DSCP would be adequate for preventing L4S traffic originating within the participating network from escaping it, provided the appropriate filtering is actually implemented at the network border.  It would prevent, for example, L4S endpoints in two *different* participating networks from accidentally making an L4S connection over the non-participating path between those networks.  But it would not prevent L4S endpoints from being set up outside the participating network entirely and communicating with each other over unprepared networks.  It does have the virtue of being very easy to set up.

> You are proposing that, in a connection between host X and host Y, if X sends using a scalable CC, it has to set both ECT(1) and a defined L4S DSCP. Then when peer Y sends, it has to set the same DSCP on the packets it sends to X. There are a number of problems with this:
> a) There is no reason or need for peer Y to know anything about L4S.

If a host is participating in an L4S-TCP transport connection then *of course* it needs to know about L4S, if nothing else to be able to provide the correct feedback.  Maybe that is different for QUIC et al.

> So how does it know to set this DSCP? A host doesn't normally even check what the incoming DSCP is.

Not reflecting the DSCP as required by the experimental protocol would indicate that it's not participating in the experiment.  That should be a strong indicator to the sender that it must switch out of L4S mode immediately.

> b) This currently would only be useful to make the L4S DSCP scheme work. So it would seem unlikely to be adopted in any of the busy WGs working on these transport protocols. Particularly given it's unorthodox architecture (overloading a L3 protocol with an additional L4 meaning)

Protecting an "alternate ECN" scheme with a DSCP is explicitly called out as a possibility in published RFCs.  The fact that is hasn't actually been done before should not be a barrier in itself.

> c) You seem to be assuming (or perhaps requiring) that the peer at one end of a connection can only use a scalable CC if the other peer also uses a scalable CC.
> d) This also means that a scalable CC cannot be tested with a Classic CC in the reverse direction.
> e) Requiring a 2-ended deployment greatly reduces the chances that any flow will use L4S.

That last one is not an argument.  We explicitly *want* to limit use of L4S signalling to connections where both endpoints know what's going on.

> The a/b/c examples I gave (still below) solely involved /customers/ attaching to a single ISP participating in the L4S experiment. If an L4S network bleaches at the border where its own customers attach, how can any L4S traffic ever enter or leave the network? I'm not understanding you.

Obviously.

In your scenario, the customers are clearly within the participating network as defined by the border at which filtering of the DSCPs occurs.  Which means that any RFC-3168 equipment those customers happen to use, and the consequent effects, are automatically part of the experiment.

I hope you have plans to inform and educate those customers of what ill effects are possible, how to opt out of the experiment to avoid any possible ill effects, and how to inform relevant people about ill effects they might observe in practice during the experiment.

> Specifically:
> * at the first IP hop controlled by the L4S network, if it bleaches the L4S DSCP from attached L4S sending customers, how does any L4S traffic ever enter the network?
> * at the last IP hop controlled by the L4S network, if it bleaches the L4S DSCP before forwarding to attached customers, how does any host ever receive L4S traffic?
> 
> Please don't answer this here. Instead, please address the specific examples below (a/b/c), to keep the discussion concrete.

Please don't deliberately misunderstand me for the sake of argumentation.  Instead please re-read points 1 and 2 below from my earlier reply, in light of the brief clarification above.

> We identified these potential problems before we brought L4S to the IETF, and at the very first ad hoc meeting about L4S (Prague Jul 2015), we proposed that this should be on the list of requirements that would need to be solved.

Okay, so why have you not solved it in the intervening time?

> Here, we are putting your idea for using a DSCP to the test. So please address the point about how an L4S-participating network still has to use the techniques in l4sops, whether it adds a DSCP or not.

My point is that l4s-ops as currently written doesn't solve the problem at hand.  I'm trying to suggest things that should go into either l4s-ops or the core design of L4S, in order to solve the problem.  I'm a problem solver, you see.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 29 Mar 2021 07:36:47 +0000

Hi
Please see inline [IJ]

/Ingemar

> Sent: den 28 mars 2021 23:30
> Cc: Sebastian Moeller <[email-masked]>; Pete Heist <[email-masked]>;
> Kyle Rose <[email-masked]>; C. M. Heard <[email-masked]>; De
> Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-
> > On 28 Mar, 2021, at 11:03 pm, Ingemar Johansson S
> I mentioned before that Pete's Czech ISP is still running 15-year-old
routers
> that date from their founding.  I personally still have a "UFO" Apple
Airport
> Base Station - the original one with 802.11b, 56-bit WEP, and a 56K
analogue
> modem - which doesn't get much *use* these days, but it still *works* and
is
> potentially useful.  Heck, I still write code for 40-year-old computers as
a
> hobby.
> 
> These old devices lack support for newer specifications.  But they still
work
> just fine in an environment that does use them, because backwards
> compatibility was designed in when the new specifications were published.
> ECN Capable transports still work with dumb drop-tail FIFOs, and Ethernet
> frames carrying IPv6 packets still traverse Layer 2 devices that were in
the
> shops before IPv6 was finalised, and which require IPv4 (or maybe
> AppleTalk) packets to configure the settings.  That is how Internet
> engineering works.  In general, old network devices are expected to
> continue working as before, and you only need to upgrade if you want to
> benefit from the new technology.
[IJ] OK, Sure, you can bring up all sorts of gear that can run like forever.
I have a 40 years old cassette player that will only need a new rubber drive
belt to be functional again. But my take is that we are not designing
protocols mainly for a few thousand netheads who can tweak old gear to work
indefinitely long. 

> 
> But L4S is *not* designed that way.  If it is deployed as-is without
effective
> guarding, it will break quite a lot of stuff that isn't even very old.
Your best
> proposal is to scrap or materially change an existing specification that
is
> currently rapidly deploying in the Internet because people see real
benefits
> from it.  It's entirely conceivable that Pete's Czech ISP will choose this
year to
> finally upgrade those old routers, *specifically* for the purpose of
deploying
> fq_codel, now that they have some experience with it - and that would
start
> a new 15-year timer, wouldn't it?
[IJ] Not necessarily so. 
1) FQ-CoDel as well as other AQMs can be updated to follow
https://www.iana.org/assignments/dscp-registry/dscp-registry.xml and as a
first step treat ECT(1) as Not-ECT. 
2) There are good reasons to make these devices upgradeable e.g for security
patches, a full L4S upgrade could go in there.

> 
> I've personally put forward two different ways to make L4S safely
deployable
> in the past week - one which requires only minimal technical changes to at
> least try out, and one which changes much more but which I think will work
> better overall.  As before, it looks like those suggestions are being
rejected
> out of hand (though Koen's attitude appears to be more balanced, for a
> change).  This is precisely what has delayed L4S for two years and
counting,
> and there is no end in sight.
> 
>  - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 29 Mar 2021 07:44:04 +0000

Hi
/Inline

> Sent: den 28 mars 2021 22:08
> Cc: Jonathan Morton <[email-masked]>; Pete Heist
> <[email-masked]>; Kyle Rose <[email-masked]>; C. M. Heard
> <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp)
> <[email-masked]>; tsvwg IETF list <[email-masked]>
> let's take your "let it age out" proposal at face value. How long do you
> estimate will it take for those problematic nodes to age out of the system
> after the IETF has signaled that rfc3168 has changed?
> And let's assume you come up with a number of X years, are you really willing
> to put the L4S experiment on hold until that time has passed?
[IJ] Things does not work that way. There are absolutely no incentives to adapt nodes on the internet that does feature A to work in concert with feature B, if feature B is put on hold until feature A adapts. 
This would call for a flag day and these have so far not worked well on the internet.

> 
> Unless the answer to that question is YES, let's just drop this proposal, okay?
> This is not a path to a productive discussion/outcome.
> 
> Best Regards
> 	Sebastian
> 
> 
> 
> > On Mar 28, 2021, at 22:03, Ingemar Johansson S
> >> Cc: Sebastian Moeller <[email-masked]>; Pete Heist
> >> <[email-masked]>; Kyle Rose <[email-masked]>; C. M. Heard
> >> <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp)
> >> <[email-masked]>; tsvwg IETF list
> >> <[email-masked]>
> >> The point here is that even if *some* existing AQMs can be updated
> >> easily, there will be some which cannot or will not.  These are what
> >> you must pessimistically take into account when analysing the risk of
> >> deploying L4S.  Until you absorb that knowledge and start taking it
> >> into account, you're not going to make any headway.
> >
> > [IJ] And these that "can or will not.." will then continue to run across the
> internet 10 years from now ?, 20 years ?.. Network gear is constantly being
> replaced with the reason being that they either become too old to be
> economically feasible to maintain or that they just break down. Home
> gateways breakdown or are just been replaced because they don't meet
> new demands. You don't normally keep a PC for more than ~5 years for a
> bunch of reasons. Why would you keep networking gear for decades ?
> >
> >>
> >> - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 29 Mar 2021 10:10:16 +0200

Hi Ingemar,

more below, prefixed [SM].

> On Mar 29, 2021, at 09:44, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> /Inline
> 
> [IJ] Things does not work that way. 

	[SM] What does not work that way? Conceptually, a discussion works best, if all sides enter only proposals they consider actually workable... here I tried to lay out the common sense consequences of arguing that the natural replacement cycle will solve the L4S rfc3168 interoperability/safety issues. As I said before, I do not think that L4S is prepared to give notice now and wait for another X years of grace period, so why even bring this argument in the first place?

> There are absolutely no incentives to adapt nodes on the internet that does feature A to work in concert with feature B, if feature B is put on hold until feature A adapts. 

	[SM] I tend to agree, hence I considered your approach, "wait for the natural replacement cycle to run its course" as an attempt at de-railing the discussion of meaningful methods to allow an experiment in an internet where rfc3168 AQMs are known to exist, and where their deployers expect them to continue to work as they did before the start of any IETF experiment. 
	As a thought experiment, would you agree that after L4S gets deployed, I can just register another experiment, running rough-shed over L4S based on the fact that doing so would be more convenient to my experiment? If you consider that to be appropriate I woulld at least have expected L4S having a designed in sun-set method allowing quit replacement for something better... IMHO this is simply not an honest argument from your side, but an attempt at constructing a staw-man argument to tear down.

> This would call for a flag day and these have so far not worked well on the internet.

	[SM] +1; but again, why did you propose this in the context of discussing actual solutions to the current stale-mate? 

I also repeat my invitation to actually look at my (and/or Jonathan's more comprehensive) proposals and try to poke holes into and/or improve them. 

@chairs, I have a hard time still assuming good-faith in team L4S, the continued ignoring of the actual proposals on the table and introduction of distractions like Ingemar's "proposal" here do not help getting to any conclusion. While I am understanding, I have zero leverage here, I want to explicitly state, that if L4S was a scientific experiment, it would have a hard time passing peer review... (it never is a grear idea to alienating all possible reviewers).

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 29 Mar 2021 10:23:21 +0200

Hi Ingemar,

more below inline, prefixed [SM].

> On Mar 29, 2021, at 09:36, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> Please see inline [IJ]
> 
> /Ingemar
> 
> belt to be functional again.

	[SM] Your "walkman" is in all likelyhood not deployed over the internet, and hence this argument is not helping with the issue at hand. Sidenote, these "real life" examples tend to be really bad analogies for packet switched networks like the internet, can we, please keep these out of the discussion?

> But my take is that we are not designing
> indefinitely long. ld have 

	[SM] Correction, team L4S is not actually designing protocols at all, they only took on wok on TCP Prague, since without any theoretically deployable protocol it would have been even harder to get their ID's describing the network bits of L4S adopted... The current attempt is mainly about forcing the network bits onto the existing internet with as little restrictions and safety as possible, as apparently safety is considered only as a hindrance for L4S expected evolution... Why are we actually still contemplating such haphazard engineering, again?

> first step treat ECT(1) as Not-ECT. 

	[SM] Yes, I failed to see your patches so far on LKML though.

> patches, a full L4S upgrade could go in there.

	[SM] +1; but no amount of wishful thinking is going to solve the fact that currently deployed gear will stay for a long time on the net. And the security consequence of that is a real life nightmare. But even though this is potentially catastrophic it has NOT changed into a worls where automatic updates are the norm and where old insecure software gets replaced in a timely manner. IMHO the security consequences/side-effects of that are much more severe than the slight inconvenience this causes for L4S's potential deployment, and yet not enough parties care (including e.g. smartphone manufacturers that only supply OS updates for a short number of years, explicitly giving a shout-out to apple here, with ~7 years software support they are leading by good example, the android side is in a sorry state comparatively).
Tl;dr: Automatic updates and security fixes are not going to magically start working well enough any tie soon, to be a viable solution to L4S's coexistence issues with rfc3168. It would be great if that would be different, to that I agree.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 29 Mar 2021 11:47:37 +0200

Hi Bob,

> On Mar 29, 2021, at 02:12, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
> 
> I was trying to tease apart the merits of the L4S + DSCP approach for a /single/ network that you were proposing. So I'd like to go back to before the point where Koen sent the thread off in a different direction (to do with the merits of a DSCP for end-to-end signalling).
> 
> Although David started this thread, I don't think he had a detailed proposal. And I think Sebastian's  proposal is slightly different from yours (using the DSCP as a classifier at an L4S AQM, which I think is unnecessarily restrictive).

	[SM] Again I note, that my sketch does not seem to have been read... and still there is time for an unqualified side remark... I am still open to a discussion, please engage.

> I'd like to take one step at a time and focus on your approach first. Inline tagged [BB]...

	[SM] Conceptually, if you manage  to "shoot" down Jonathan's more detailed and comprehensive proposal, you still need to tackle mine because of its much smaller and more targeted scope... I gues your time your decision with what to engage...

> [BB] There is no L4S design 'baked in' to existing standardization work. L4S is on the experimental track, and designed to build /on/ ECN feedback in existing standards track IETF transport protocols. Specifically QUIC, RTCP and TCP.

	[SM] I begin to think that adopting L4S ID's without accompanying protocol IDs might not have been the most productive ad efficient move.

> An L4S congestion control acts at the sender in one direction. The other direction might use a completely different congestion control. This is outlined in l4s-arch, but it's no different from the…

	[SM] I beg to differ, L4S protocols like TCP Prague require specific feed-back from the receiver, so a TCP sender for example is not free to use TCP Prague on a half link unless the receiver agreed to it, that seems different from other TCP CCs. Sure the reverse path is not technically forced to use TCP Prague, but that seems academic. 
	Or let me ask, are you really proposing end-points should only negotiate L4S signaling in uni-directionally, and which direction would you recommend? I smell a stawman...

> a) There is no reason or need for peer Y to know anything about L4S.

	[SM] Pardon? For the only existing L4S protocol TCP Prague to work both sides are expected to negotiate AccECN, with TCP Prague being the only existing consumer of AccECN as far as I can tell (with a very weak definition of existence)...

> So how does it know to set this DSCP? A host doesn't normally even check what the incoming DSCP is.

	[SM] Yes, because normally this is not required, but an AQM normally does not check whether a packet carries ECT(0) or ECT(1) when deciding between setting CE or drop either, so why is a change on the end-host side unacceptable, given that you need a new protocol anyway, since mo existing protocol fulfills the L4S protocol requirements (this very much includes the "best" of the lot, TCP Prague).

> b) This currently would only be useful to make the L4S DSCP scheme work.

	[SM] Again, so is differentiating between EXT(0) and ECT(1) at the AQM level.

> So it would seem unlikely to be adopted in any of the busy WGs working on these transport protocols. Particularly given it's unorthodox architecture (overloading a L3 protocol with an additional L4 meaning)

	[SM] How about to negotiate that with the affected WGs first instead of jumping to conclusions, just because these conclusions are convenient for you? Sidenote, the same argument can be made about using ECT(1) as "classifier", both the ECN bits and the DSCP bits sit in the same layer and in both cases we expect the protocol layer to interact with them in some fashion.

> c) You seem to be assuming (or perhaps requiring) that the peer at one end of a connection can only use a scalable CC if the other peer also uses a scalable CC.

	[SM] No, both ends need to cooperate for an L4S CC (by virtue of the AccECN requirement and the other protocol requirements), but there is no need for them to use the Prague CC in both directions. But the fact that cooperation is needed will allow to introduce the DSCP based additional layers.

> d) This also means that a scalable CC cannot be tested with a Classic CC in the reverse direction.

	[SM] Well then, I am sure we could relax Jonathan's proposal to split out the two directions. But I severely doubt that L4S is going to be all that helpful unless both the forward and reverse ACK traffic are handled by L4S AQM, otherwise the reverse path jitter is going to pretty much destroy the jitter reduction that L4S in the forward path delivered... unless one naively assumed an uncongested reverse path.

> e) Requiring a 2-ended deployment greatly reduces the chances that any flow will use L4S.

	[SM] Bt you do that already, with TCP Prague, sorry. And you absolutely NEED to, you are trying to establish a new communication signal, so you need to coordinate all three: sender, receiver and signal (path). This not a logical argument to make, especially not in the context of an experiment.

> 
> Let's say there are 10 million hosts on a network participating in the L4S experiment. Even once 10,000 hosts deploy L4S with your DSCP scheme, they will still find 9.99M hosts without L4S support.

	[SM] If the L$S requirements include the DSCP as mandatiry element, these 10 million end-hosts will naturally acquire the necessary capability when they enter the experiment. This is also why it is essential to bake this into the design and implementation BEFORE beginning wide-scale deployment.
	You are already facing exactly that situation by the way, by virtue of only having a fig-leaf protocol in TCP Prague...

> In your scheme, both ends have to support L4S, which is a probability of 1 in a million (1 in 1000 squared) that any particular flow will support L4S. That makes the deployment incentives awful. And it's not such a great experiment either with so little L4S traffic.

	[SM] Sorry, again, since protocols need to take active steps to gain L4S compliance already, this is not a changed when an additional DSCP component gets added to the signaling mechanism.... 

> 
> Because AccECN for TCP is still being held back, we will anyway have this 2-ended deployment problem over TCP. But we don't want to unnecessarily create a 2-ended deployment over the other transports (QUIC, RTCP, etc).

	[SM] Convenience should not trump safety, but that is effectually what Koen and you are bringing as argument. In all honesty, if the interest in L4S will not survive the addition of a DSCP layer to make the design save, then I would humbly propose that the interest in L4S was not sincere enough to begin with. 

> [BB] The a/b/c examples I gave (still below) solely involved /customers/ attaching to a single ISP participating in the L4S experiment. If an L4S network bleaches at the border where its own customers attach, how can any L4S traffic ever enter or leave the network? I'm not understanding you. Specifically:

	[SM] Easy, during the experiment a participating network AS agreed to bleach the L4S DSCP at its diffserv boundaries, unless the other side agreed by SLA/TCA/verbal contract/whatever to knowingly accept that DSCP and follow the L4S requirements. This very much includes the end-points. Endpoint need to opt in as well, but they need to do this more or less already, since they would need to deploy L4S capable/compliant protocols to participate in the testing to begin with. My spider-sense how ever tells me, that you very much plan to enroll end-points into the experiment without first asking for consent....
	Could you please elaborate on L4S' official position on this, are end points expected to participate willingly and by active opt-in, ior are you envisioning end-point getting volunteered, say by their ISP without active consent?

> * at the first IP hop controlled by the L4S network, if it bleaches the L4S DSCP from attached L4S sending customers, how does any L4S traffic ever enter the network?

	[SM] Easy, it end points need to actively opt-in to the L4S experiment, an ISP supplying end customers could easily interpret the combination of the L4S-DSCP and ECT(1) (if that is still part of the proposal) as implicit decision for requesting L4S treatment if possible. But that place, ISP <-> enduser is special in that regards, for boundaries between commercial entities, like CDNs something like an SLA/TCA should e required (as these exist at hese boundaries anyway, unlike at the end-user <-> ISP boundary).

> * at the last IP hop controlled by the L4S network, if it bleaches the L4S DSCP before forwarding to attached customers, how does any host ever receive L4S traffic?

	[SM] Same argument, if the ISP it self participates in the experiment it might simply pass the DSCP to the end customer, as there still needs to be a protocol at the receiving end to actually turn that DSCP into an L4S signal (in communication it matters how a reciver interprets a message). Or the ISP could allow endusers to toggle their L4S preference in the on-line customer center that many ISPs already seem to use for other self-serv customization.

> 
> Please don't answer this here. Instead, please address the specific examples below (a/b/c), to keep the discussion concrete

	[SM] Why? These conceptual issues could be answered on the same conceptual level (see my response above).

> .
> [BB] It is good practice to read someone else's email right through before you start answering it.
> Here, you are telling me what I told you further down this email, but without responding to the follow-on point I made about this (find "l4sops" below).
> 
>> 
> [BB] I am trying to first understand the single DSCP approach you appear to be describing here. But I will not be able to understand you if you don't explain how your approach will work in the examples I give at a/b/c below.
> 
> And no-one is going to write your idea into a working group draft unless you individually address each of the criticisms of DSCP-based schemes in Appx B.4.

	[SM] Bob the issue with B.4 is it contains gems like"
"Not end-to-end (host-network):  Very few networks honour a DSCP set
      by a host.  Typically a network will zero (bleach) the Diffserv
      field from all hosts.  DSCP bleaching would turn an L4S ECN packet
      into a Classic ECN packet."

And one part of the DSCP proposals is to explicitly restrict accidental E2E transmission of L4S signals unless the full network path considered itself L4S aware... In other words B.4 is explicitly written from the perspective of the taken decision, to post-hoc make this seem like the only viable choice.

> You cannot seriously expect to be granted a free pass to ignore all the problems with DSCPs by simply pronouncing that every one of the arguments is invalid.

	[SM] True any DCSP proposal should be measured in how it achieves its explicit goals. The same is true for all RFC/ID including the L4S drafts, where, to repeat my self, there still are considerable differences between what is promised in the UIDs and what is actually delivered as seen in actual test data.

>>> Imagine a FIFO RFC3168 AQM in the queue of a home router in the upstream direction. An L4S client sends L4S packets through this AQM, which marks some as CE. Then the network participating in the…

	[SM] In that case the end-user is expected either to disable the rfc3168 component of that AQM and opt-into using L4S, or to make the opposite decision. Once you embrace informed consent and active opt-in, such issues simply disappear. If however you think you know better than the end-user and decide to set policy for their network by force, the onus is on you to make sure that observable behaviour does not regress. The fact that you consider this situation a potential problem at all ring all sorts of alarm bells on my side, but at least is consistent with the overall approach of L4S towards risk management (which I do not endorse).

>>> Consider an RFC3168 AQM in the network of a corporate site or campus network receiving traffic from a connected L4S-participating network. Or perhaps on the access into a data centre connected to…

	[SM] Nope, by admitting the L4S DSCP and passing it on, the network essentially will have declared that it will deal with the consequences/fall-out of L4S signaling. Again active opt-in trumps accidental interoperability questions any day. Active consent will not automatically fix all interoperability questions, but it will make sure that involved parties are conscious of the fact and places the responsibility explicitly at that place/entity that can actually take active measures to contain potential issues. One solution could be simply, assure no rfc3168 is still in use (or have those simply treat ECT(1) like NotECT), the L4S-ops draft contains a few of those issues, and anybody opting in ot the L4S experuiment can be well expected to read that ID, no?

>>> Consider an RFC3168 AQM within the network participating in the L4S experiment. Same problem. The participating network delivers the packet with the L4S-ID DSCP intact, but the CE marking is RFC3…

	[SM] Nope, the onus is on the participating network to either not let this happen, or deal with its consequences any way it sees fit.

>>> In this last case, the L4S-paticipating network has to use techniques like those in l4sops to make sure any FIFO RFC3168 AQMs don't mark ECT(1). That's no different to if ECT(1) alone was the ide…

	[SM] But an L4S flow traversing a non-paticipating network that employs an rfc3168 might well interpret CE without DSCP as classoc ECN signal and respond appropriately, which should be better than a interpreting as a drop (think ABE).

> [BB] We identified these potential problems before we brought L4S to the IETF, and at the very first ad hoc meeting about L4S (Prague Jul 2015), we proposed that this should be on the list of requirements that would need to be solved.

	[SM] Yes, but for 6 years you did not manage to find a solution for the rfc3168 conundrum, which indicates that the trade-of agreed on 2015 simply is not viable either. So maybe another trade-of needs to be found, preferably one that does not trade in others' safety for L4S' convenience?

> 
> Here, we are putting your idea for using a DSCP to the test.

	[SM] Well, really you are trying hard to tear this down, I see exactly zero attempts to propose what Jonathan's proposal would need in addition to make it operationally equivalet to what you have converged upon...

> So please address the point about how an L4S-participating network still has to use the techniques in l4sops, whether it adds a DSCP or not.
> 
> 

	[SM] Yes, preferably my much more restricted and targeted proposal, please. You get to do your experiment quicker, and we get the failure result quicjer and the whole thing can bu forgotten earlier.

QUESTION: has anybody discussed the current design with its massively increase in RTT bias with T1-ISPs that mainly deal in long-range transit? I might be wortwile getting ther opinion, on whether we should prioritize even more bandwidth for short range connections than TCP in FIFOs already does? (I will put this into ne thread as well, down here it will just be ignored).

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 29 Mar 2021 14:58:48 +0200

Hi Bob,

> On Mar 29, 2021, at 02:35, Bob Briscoe <[email-masked]> wrote:
> 
> Mike,
> 
> Many of the arguments for why a DSCP is not useful e2e also apply when it is used as a guard DSCP in a single network. For instance, when an IP header with a DSCP is tunnelled in pipe mode, it is not propagated to the outer. So it won't be visible, won't be bleached, etc.

	[SM] Excellent point, sothe  L4S-ops draft should probably a recommendation to also propagate DSCPs in and out while tunneling, if tunneling is something the diffserv domain wants to support for L4S, otherwise the recommendation might simply be to bleach on tunnel ingress (which is a also a naturally safe default, if the tunnel might terminate in another potentially unknown diffserve domain).

> 
> There's far too much vagueness in this thread.

	[SM] Not that much more than in the L4S drafts, where problems magically seem to dissolved when they are punted to the protocols that night want to use L4S signaling, really.

> In the email I just sent to Jonathan, I asked him to address each of the critiques of a DSCP in Appx B.4, not just dismiss them all as invalid, which is untenable.
> Certainly, some of the critiques are not relevant to the way the DSCP is used in Jonathan's scheme, but many are like the tunnel issue above.

	[SM] Odd that tunneling always moves into the spot-light if it helps your argument, but gets more or less ignored when it causes unsolved challenges for L4S, like a tunnel with mixed traffic running over one of the relative common (common as far as AQMs go) FQ rfc3168 compliant AQM. Is it really to much to ask for consistent application of the same yard stick to all proposals?

> So far, we've only just started on the basics like whether the feedback approach is even practical.

	[SM] Using DSCPs for feed-back is exactly what the "A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services" ID is all about. I have heard very little comment from your side about its practicability issues (I probably have missed those though). Interesting to cite from that draft:

"7.  Relationship to L4S
   Traffic flows marked with the NQB DSCP as described in this draft are
   intended to be compatible with [I-D.ietf-tsvwg-l4s-arch, with the
   result being that NQB traffic and L4S traffic can share the low-
   latency queue in an L4S dual-queue node
   [I-D.ietf-tsvwg-aqm-dualq-coupled].  Compliance with the DualQ
   coupled AQM requirements is considered sufficient to enable fair
   allocation of bandwidth between the QB and NQB queues."

and

"4.1.  End-to-end usage and DSCP Re-marking
   In contrast to the existing standard DSCPs, many of which are
   typically only meaningful within a DiffServ Domain (e.g. an AS or an
   enterprise network), this DSCP is expected to be used end-to-end
   across the Internet.  Some network operators typically bleach (zero
   out) the DiffServ field on ingress into their network
   [Custura][Barik], and in some cases apply their own DSCP for internal
   usage.  Networks that support the NQB PHB SHOULD preserve the NQB
   DSCP when forwarding via an interconnect from or to another network.
   Bleaching the NQB DSCP is not expected to cause harm to default
   traffic, but it will severely limit the ability to provide NQB
   treatment end-to-end."

And from Greg White directly on the topic of SLAs (https://mailarchive.ietf.org/arch/msg/tsvwg/NrWh_sfCBRcASHirmScjeIAHlFA/):

"When we discussed this concern before, it seemed to me that an SLA might be a good approach (at least in the near term).  So, perhaps we aren’t communicating well.  It is possible that a peering network is already using the value 42 (0b101010) - or for that matter any DSCP that IANA might assign for NQB - in a manner that is inconsistent with NQB, so default bleaching is probably the best near term situation.  For an ISP that wishes to support the NQB PHB, they can validate (and set up SLAs) that identify the NQB traffic from a peer, and treat it per the PHB, whichever DSCP is chosen.  Alternatively, if the ISP does not wish to support the NQB PHB at all, then they can choose to bleach it to Default, or pass it though with Default treatment, whichever they choose."

This is pretty much what you seem to argue for as being impractical, no?

Best Regards
	Sebastian

>>  [SM] Well, sure, but you came to the ECT(1) decision for a reason (a reason I do not agree with, but that is a different topic, I am trying to build you a bridge here how to run your experiments without taking the rest of the internet down), so I am not going to argue about that. My proposal adds an GUARD DSCP for the duration of the EXPERIMENT to allow your measurements in live production networks without negative side-effects for non-participants. This layer of safety can be removed if the experiment has run ist course and measurements in consenting networks has provided data that L4S is safe without the DSCP. At which point the DSCP could be dropped, try doing that if you only use a DSCP as L4S classifier.
>> 
>> This point sure seems to have been lost in much of the discussion. Note again: GUARD DSCP for the duration of the EXPERIMENT. NOT DSCP instead of ECT(1).
>> 
>> I've read several comments objecting that using DSCP to distinguish L4S traffic is undeployable end-to-end on the general Internet. Well, according to the intended status of the drafts, L4S is starting life as an EXPERIMENT, and as such is not expected (or at least should not be expected) to be deployed on the general Internet. If the WG's desire is to have L4S deployed on the general Internet as soon as the specs are approved, then the WG should take Steven Blake's advice and deprecrate/obsolete RFC 3168 and change the intended status of the drafts, or else use a mechanism that is backward compatible with RFC 3168, such Jake Holland's proposal to use ECT(1) -> ECT(0) as the L4S congestion signal, leaving the semantics of CE unchanged.
>> 
>> Mike Heard
> 
> --

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 29 Mar 2021 13:23:40 +0000 (UTC)

Pete, tsvwg,

I can see that the idea of using a guard DSCP is proposed as a pragmatic way to make progress. However, I think an experiment isolated from RFC3168 AQMs by a DSCP would not, of itself, resolve the problem of L4S and RFC3168 AQMs. 

Suppose the L4S folks go away and deploy L4S in such a DSCP-guarded domain. They  make progress on some of the experimental goals listed in draft-ietf-tsvwg-ecn-l4s-id, and come back and say it works really well, and now they want to deploy it on the full internet. At that point, the WG is pretty much in the same position it is now - not knowing how to resolve the RFC3168 issue. Except that the stakes would be even higher, because:

- the L4S folks would have made even more investment in L4S, in a production deployment across multiple companies
- Other folks would potentially have deployed more RFC3168 AQMs
So unfortunately it seems to me that, although it now seems pretty hard for the WG to reach consensus on  how to resolve the RFC3168 issue, waiting for a DSCP-isolated experiment will make this harder, not easier. I think if would be better for the WG to find a path to that resolution sooner rather than later. That's not to say that a DSCP-guarded experiment couldn't contribute to such a resolution, but without some agreement on what the path to a decision looks like, it's not obvious that one is essential.
Alex

    On Saturday, March 27, 2021, 2:20:07 PM GMT, Pete Heist <[email-masked]> wrote:  
 
 ...

Kyle

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 29 Mar 2021 17:02:56 +0300

> On 29 Mar, 2021, at 4:23 pm, [email-masked] wrote:
> 

I think I can agree with the above.

A single-DSCP containment can limit the harm directly caused by the experiment, but it's not a long-term solution since it still leaves L4S as an RFC-4774 Option 1 scheme which *needs* to be contained to specially prepared networks.  I note also that even this minimal protection is not being greeted with any enthusiasm by L4S proponents.  Your argument against it is, however, the most coherent that I've heard so far.

I suggested a two-DSCP scheme as something with more robustness and longevity than a single-DSCP scheme, while still fitting the "L4S way" of doing things and requiring only minor implementation changes to try out.  The aim here is RFC-4774 Option 2, ie. explicitly detecting which signalling scheme is used by the network.  It doesn't solve all of the problems, but I think it does illuminate one path towards success.

Since two years ago, SCE has been available as an RFC-4774 Option 3 solution, though it has been opposed by Team L4S as an existential threat.  Adding DSCP labelling to SCE allows it to be used with simple dual-queue middleboxes, so far as the DSCP survives in the network, as well as the preferred FQ/AF AQMs, and without the serious compatibility problems that seem to be inherent to the conditional redefinition of a CE mark.  The same DSCP label could be used temporarily as an experiment containment mechanism, though SCE doesn't really need it.

What do you think of the latter two approaches?

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 30 Mar 2021 09:21:12 +0100

Sebastian,

If you are not adding to a thread, please consider remaining silent.
I've corrected all your misunderstandings inline (I ignored the one that 
is just a restatement of an opinion), tagged [BB].

> 	[SM] Excellent point, sothe  L4S-ops draft should probably a recommendation to also propagate DSCPs in and out while tunneling, if tunneling is something the diffserv domain wants to support for L…

[BB] L4S does not involve a DSCP, so there is no need to discuss DSCPs 
in l4sops.

> 	[SM] Odd that tunneling always moves into the spot-light if it helps your argument, but gets more or less ignored when it causes unsolved challenges for L4S, like a tunnel with mixed traffic runni…

[BB] The tunnel issue with L4S in VPNs over FQ is addressed in l4sops.
https://datatracker.ietf.org/doc/html/draft-white-tsvwg-l4sops-02#section-3
A solution is mentioned that at least isolates L4S from Classic traffic 
within the same VPN. A later revision will give more details of this 
solution, which we have been working out off list.

The tunnel issue was central to the choice of the L4S identifier - you 
will find it mentioned all over the ecn-l4s-id draft.

> This is pretty much what you seem to argue for as being impractical, no?

[BB] None of these 3 quotes about NQB are anything to do with DSCP 
feedback, nor reading the DSCP at the receiver.

Perhaps the ambiguity of the word end-to-end is the misunderstanding here.
* When the term end-to-end is used by transport layer people, it means 
from one end-system process to another (and even then it does not imply 
anything about feedback unless feedback is being talked about).
* But, when the word end-to-end is used by network layer people, it 
means that a DSCP survives (possibly remapped) as it traverses the path 
across networks /at the network layer/.

Also NQB is not L4S, which is the subject of this thread.

Bob

---

From: Pete Heist <[email-masked]>
Date: Tue, 30 Mar 2021 11:22:36 +0200

Alex,

I agree with this point overall. Although, a well conducted experiment
in congestion control has a funny way of yielding new information that
may be more consequential than what you were trying to look at in the
first place.

It becomes important what is tested, by whom, and what the criteria for
success are. And among other questions- if nobody in the test group
reports anticipated problems, but they still exist, what then? I think
this is also in line with what you're suggesting.

It seems to me that, short of (and in addition to) a guard DSCP for
experiment, the fundamental issue of semantic compatibility with
RFC3168 will need to be addressed head on, somehow.

Pete

On Mon, 2021-03-29 at 13:23 +0000, [email-masked] wrote:
> 
> Pete, tsvwg,
> 
> I can see that the idea of using a guard DSCP is proposed as a
> pragmatic way to make progress. However, I think an experiment
> isolated from RFC3168 AQMs by a DSCP would not, of itself, resolve
> the problem of L4S and RFC3168 AQMs. 
> 
> Suppose the L4S folks go away and deploy L4S in such a DSCP-guarded
> domain. They  make progress on some of the experimental goals listed
> in draft-ietf-tsvwg-ecn-l4s-id, and come back and say it works really
> well, and now they want to deploy it on the full internet. At that
> point, the WG is pretty much in the same position it is now - not
> knowing how to resolve the RFC3168 issue. Except that the stakes
> would be even higher, because:
> 
> - the L4S folks would have made even more investment in L4S, in a
> production deployment across multiple companies
> So unfortunately it seems to me that, although it now seems pretty
> hard for the WG to reach consensus on  how to resolve the RFC3168
> issue, waiting for a DSCP-isolated experiment will make this harder,
> not easier. I think if would be better for the WG to find a path to
> that resolution sooner rather than later. That's not to say that a
> DSCP-guarded experiment couldn't contribute to such a resolution, but
> without some agreement on what the path to a decision looks like,
> it's not obvious that one is essential.
> 
> Alex
> 
> 
> > On Fri, Mar 26, 2021, 8:30 AM De Schepper, Koen (Nokia -
> > BE/Antwerp) <[email-masked]> wrote:
> > > I understood the goal of your proposal. But before diving into
> > > the details of a DiffServ-based proposals, I'm taking a step back
> > This approach allows for experimenting with ECT(1) as a classifier
> > in a safe (contained, time-limited) way, thus allowing a path
> > toward universal deployment (i.e., removing the DSCP guard) if the
> Although an RFC isn't technically required to do this, it should
> still serve to codify how DSCP should be used and treated, and also
> to raise visibility as a sanctioned IETF experiment. After a
> successful experiment at "operator scope" (if that terminology
> captures it, but the larger the better), it should be easier to
> justify and start an experiment at Internet scope.
> 
> I also think that with codepoint space this tight, it's not just
> about verifying safety, but also effectiveness. I feel the same way
> even for proposals that fall under RFC4774 Option 3 ("Friendly
> Coexistence with Competing Traffic").
> 
> Pete
> 
> > Kyle
> 
>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 30 Mar 2021 12:46:28 +0200

Bob,

> On Mar 30, 2021, at 10:21, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 

	[SM] This is a tad impolite; in a discussion every opinion voiced (with sufficient civility) has the same right to be expressed. Just as I a not the arbiter on whether your posts "add" to a thread you are not the arbiter for mine. The fact that I have to explicitly write such self-evident and obvious facts is a sad fact in itself.

> I've corrected all your misunderstandings inline (I ignored the one that is just a restatement of an opinion), tagged [BB].

	[SM] Bob, when I come to different conclusions from evaluation the same data as you, that does not automatically mean that I am misunderstanding something. there could be different reasonable interpretations of the data, or, god forbid, you might be wrong... 

> [BB] L4S does not involve a DSCP, so there is no need to discuss DSCPs in l4sops.

	[SM] Yet, if you read my proposal you would understand that a such modified L4S would very much contain a guard DSCP and hence L4S-OPs should be amended in that way. We can argue about the merits of my proposal and in the end reject it, but from my position and perspective adding the whole DSCP part to the OPs ID makes inherent sense. Also, an operator might want to use DSCPs for better isolation between network nodes with and without rfc3168, even if such a DSCP is not in the other L4S drafts\. And in fact most of the remedies/work-arounds in L4S-ops are not in the other drafts either.
	I can't shake the feeling, your are not actually trying to engage with my arguments, but rather try to swat them down with as little effort as possible...

	But I understand you as saying that you categorically reject adding any DSCP requirement to the L4S drafts whether transiently for the duration of the experiment or permanently as part of the signaling, and based on that position you see no use for adding this to the OPs draft? If that is the case you might have started out with actually clearing up that assumption first, and then as a consequence rejecting discussing DSCPs as part of the OPs draft...

> A solution is mentioned that at least isolates L4S from Classic traffic within the same VPN. A later revision will give more details of this solution, which we have been working out off list.

	[SM] The thing is, there is no generic robust and reliable solution, neither off-list, nor on-list.... as I said, tunnels are only show-stoppers when L4S has no issues with them, and where it does it becomes magically manageable (somewhen in the future*).

> 
> The tunnel issue was central to the choice of the L4S identifier - you will find it mentioned all over the ecn-l4s-id draft.

	[SM] IMHO at the expected main deployment points, at the DOCSIS-ISP<-> end-user edge tunneling will simply be a minor issue that could for the first iteration of L4S be ignored, and for deeper bottlenecks like transit-links L4S simply is not suited (due to the ungodly increase in RTT-bias and bias against non-L4S traffic).

> [BB] None of these 3 quotes about NQB are anything to do with DSCP feedback, nor reading the DSCP at the receiver.

	[SM] Well, these are about enabling NQB end to end, which would be e necessary condition for using DSCPs in the context I proposed, these quotes indicate that the sponsors of the NQB draft see this as feasible enough.
	DSCP feedback is really a question about setting DSCPs and reading the DSCP at the reciever should be no issue, given that most protocols either live inside the kernel (where you just need to ask nicely) or in userspace, where raw_sockets seem to offer all the acces you might ever need.
	Honestly, Bob, these to questions indicate that you have not even started to look into how feasible a guard DSCP would be, but just trying to muddy the waters with under-researched objections, mostly based on lack of knowledge, no?

> * When the term end-to-end is used by transport layer people, it means from one end-system process to another (and even then it does not imply anything about feedback unless feedback is being talked about).
> * But, when the word end-to-end is used by network layer people, it means that a DSCP survives (possibly remapped) as it traverses the path across networks /at the network layer/.

	[SM] I thougth that it was clear that I was talking about the NECESSARY condition of DSCPs traversing fully between the end-nodes. Reading them there and constructing a feed-back loop out of that signal, is something I propose, well within the spirit of how L4S tackles issues, to formulate as a requirement for L4S-compatible protocols.

> Also NQB is not L4S, which is the subject of this thread.

	[SM] Reading the relevant CableLabs DOCSIS standards and the Low Latency DOCSIS White paper (where you were one of the co-authors) clearly indicates, that NQB is very much intended as the canonical DSCP in DOCSIS networks to request scheduling in the L4S-LL (or on DOCSISese immediate AQM). For all means and purposes, NQB seems to be the "missing" DSCP, even though it is not explicitly acknowledged as such in the L4S IDs yet. But again, ID text can (and in this case should) be changed, arguing that NQB is not L4S is technically true, but irrelevant. I understand you know this, and frankly am getting a bit tired of any discussion with you turning into a debate.

Again, here is the text from the NQB draft:
"9.  Relationship to L4S
   Traffic flows marked with the NQB DSCP as described in this draft are
   intended to be compatible with [I-D.ietf-tsvwg-l4s-arch], with the
   result being that NQB traffic and L4S traffic can share the low-
   latency queue in an L4S dual-queue node
   [I-D.ietf-tsvwg-aqm-dualq-coupled]. "

If you do not think that text will make NQB the missing L4S DSCP, then I would expect you object to that paragraph immediately. Just because the L4S DSCP is not explicitly introduced and mentioned in the core L4S IDs does not change the fact, that if the NQB draft succeeds with its goal, NQB will be the de facto DSCP for L4S, no?
I often wonder about the level of language lawyering happening in drafts and discussions, I would assume what matters more is what happens/is going to happen in reality than any amount of word smithing put into one draft, but what do I know.

Sebastian

*) Or rather not, as the failed attempts at fixing up dualQ's RTT-bias inflation and L4S' general incompatibility with rfc3168 AQMs in TCP Prague demonstrated. All things promised in draft and on list and things that petered out falling way short of the promises and what should be required.

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 30 Mar 2021 14:16:13 +0200

Bob,

since you asked,

> [....]

In Linux look at include/net/dsfield.h (https://elixir.bootlin.com/linux/latest/source/include/net/dsfield.h#L16)
This will give the ability inside the kernel to get and set DSCPs for IPv4 and IPv6...
Given that TCP Prague lives inside the kernel itself that should not be to hard to use.
That will not help for other OSs, but since these do not have L4S compatible protocols today (with TCP Prague being the closest to compatibility) that should not be a show stopper.

Best Regards
	Sebastian

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Tue, 30 Mar 2021 06:26:57 -0700 (PDT)

> That will not help for other OSs, but since these do not have L4S compatible protocols today (with TCP Prague being the closest to compatibility) that should not be a show stopper.

Sebastian, Bob, twvwg,
I am aware that FreeBSD has similiar features, and some of the userland
daemons and clients can even be configured to use DSCP's.

Regards,
-- 
Rod Grimes                                                 [email-masked]

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 30 Mar 2021 16:58:25 +0200

Thanks Rodney,

this information is appreciated.

Sebastian

> On Mar 30, 2021, at 15:26, Rodney W. Grimes <[email-masked]> wrote:
> 
>> Bob,
>> 
>> since you asked,
>>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 30 Mar 2021 20:47:00 +0300

> On 30 Mar, 2021, at 11:21 am, Bob Briscoe <[email-masked]> wrote:
> 
> L4S does not involve a DSCP, so there is no need to discuss DSCPs in l4sops.

Okay, so what mechanism do you propose to contain the L4S experiment to the participating networks?

> The tunnel issue was central to the choice of the L4S identifier - you will find it mentioned all over the ecn-l4s-id draft.

Yes, you've gone to a lot of trouble to make sure that L4S traffic is identified to the network in many different situations.  But the problems all have to do with networks that don't know to look for that identifier, even though it is present and correct on the traffic in question.  That's why the experiment has to be contained to networks that do.

Ultimately the problems arise because the network does not identify to the receiver what type of congestion signal it is producing.  You can only detect that indirectly through heuristics and probing, and (so far) not at all reliably.  We have been waiting for a solution to that deficiency since it was formally raised at Montreal.

Focus on these problems, and we might get somewhere.  Dismiss them, and you will fail.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 30 Mar 2021 23:07:36 +0100

Jonathan,

In May 2020, a very intense period of focus on the question of the L4S 
identifier came to a close. The possibility of a FIFO RFC3168 ECN AQM 
that you are focusing on (to the exclusion of all else) was one of many 
issues that made the choice of identifier difficult. In this thread, I 
have been digging into this DSCP proposal from you and David to show 
that, when another candidate identifier is investigated more deeply, 
different concerns of feasibility, safety, etc. emerge.

We can now summarize the impact of this proposal to use a DSCP to 
confine the L4S experiment to one network:
* You've agreed that the DSCP scheme doesn't itself avoid any FIFO 
RFC3168 ECN AQMs in the network participating in the experiment, or in 
its customer networks upstream or downstream. It still requires a 
participating L4S network to follow the l4sops draft to bypass these 
AQMs, as if they were using the original L4S ECN scheme.
* You've also agreed that the DSCP scheme needs changes to all standards 
track transport protocols, so that they somehow feed back the DSCP as 
well as the ECN field, even though bits are scarce, and even though the 
only motivation for these changes would be to make this DSCP idea work 
for L4S.
* In addition, the DSCP will not necessarily even serve its confinement 
function if it is within a pipe-mode tunnel.

In summary, this DSCP proposal adds very little to safety and it 
subtracts greatly from viability of the experiment.

To you and one or two others, destroying the viability of an L4S 
deployment experiment doesn't seem to matter. However, the point of that 
session in 2020 was for people to choose /on balance/ which identifier 
they wanted the WG to focus its work on, taking account of the 
compromises that each choice would entail.

On a question of balance between competing requirements, we were never 
going to get unanimity. Nonetheless, a very large majority of the people 
polled last May indicated that they wanted to move forward with the L4S 
identifier as it stood. This reaffirmed the same decision in Nov 2015. 
Also, at that meeting in May 2020, a very  large number of companies had 
confirmed they had plans to deploy L4S (all as a company, not just 
individuals in those companies). In contrast, of the three companies 
that you volunteered, two publicly admonished you for implying they 
supported you, and it turned out the other one had not actually 
expressed support as a company.

As a result, the chairs then set the agenda such that the WG would focus 
only on getting L4S through the WG stage; with L4S ECT(1) as the 
identifier. All other identifier proposals are currently off the table. 
You have no standing to demand that L4S uses a different identifier, let 
alone that L4S-specific variants of standards track transport protocols 
have to be invented to make your ideas work.

I only dug in depth into this DSCP scheme to prove how little it adds 
and how much it subtracts. This was primarily for those who joined David 
Black's discussion on the list such as Steven Blake, Jake, Kyle, and 
Ruediger, who otherwise might have continued to think it might be a 
feasible approach. I suspect there was some confusion that the DSCP was 
being used as a classifier in the network, whereas you are proposing it 
for a per-connection cross-layer path traversal feedback loop, without 
its classifier function at all.

Like this DSCP proposal, the SCE proposal also subtracted greatly from 
viability of the experiment. But at least it had the potential to 
improve safety if FIFO 3168 AQMs turned up. (However, as you admitted, 
SCE traffic gets squashed by legacy traffic in the same queue). So this 
is the second of your proposals that has not stood up to scrutiny.

See inline for detailed responses, tagged [BB]...

> Well, in that case it should still be feasible to improve L4S' design and implementation before deployment.  Of course, the longer you spend resisting any and all efforts to suggest such improvemen…

[BB] See above. The WG decision has been made (twice). This entitles me 
and others to resist further debate about the L4S identifier. It is 
those like you and David who are holding back the WG - by still 
proposing different identifiers. The question of the relative importance of:
     a) addressing RFC3168 AQMs
versus
     b) viability of the chosen identifier
was well understood by all those who made a choice last May.

> That is indeed a case that would be problematic for the two-DSCP scheme as I presented it.  I think you could salvage it by adding explicit feedback of the relevant information to AccECN.

[BB] AccECN is TCP only - you are ignoring QUIC and RTCP. And there is 
extremely limited space in TCP anyway.

We all understand that you are quite prepared to make L4S depend on 
ridiculous additions to standards track transport protocols, because you 
care if L4S fails to deploy.

> Or, for the purposes of the experiment, require that L4S signalling be used in both directions or not at all.  Or - and this is my preferred option - switch to a non-ambiguous signalling scheme where these complications are far less necessary.  Your choice.

[BB] You need to understand that others are entitled to weigh the 
importance of different issues differently to you. They have made their 
choices already (twice), and you are in the rough.

>
> Using a single DSCP would be adequate for preventing L4S traffic originating within the participating network from escaping it, provided the appropriate filtering is actually implemented at the network border.

[BB] Correct. The "appropriate filtering" condition is one of the major 
problems (see later).

> It would prevent, for example, L4S endpoints in two *different* participating networks from accidentally making an L4S connection over the non-participating path between those networks.

[BB] Correct. This is the /only/ case where the idea adds anything.

> But it would not prevent L4S endpoints from being set up outside the participating network entirely and communicating with each other over unprepared networks.

[BB] Correct.

> It does have the virtue of being very easy to set up.

[BB] Incorrect. Adding scarce bits to every standard transport protocol 
when they have no purpose other than to make an experimental idea work 
is not easy, let alone very easy.

> If a host is participating in an L4S-TCP transport connection then *of course* it needs to know about L4S, if nothing else to be able to provide the correct feedback.  Maybe that is different for Q…

[BB] No. The receiver doesn't know the congestion control being used by 
the sender.

The IETF builds protocols as independent components wherever possible. 
An AccECN receiver rightly knows whether the sender supports AccECN, but 
not what CC it is using. AccECN is intended to update RFC3168, and can 
be used for any congestion control: Reno, Cubic, BBR, DCTCP, Prague, 
whatever.

QUIC, RTCP and DCCP already provide general-purpose ECN feedback. AccECN 
makes TCP's ECN feedback similarly generic, replacing the feedback in 
RFC3168, which was specific to only one class of congestion controls. 
SACK did a similar job for loss feedback.

Dumb reflection from the receiver is a lower layer facility than flow 
regulation, which can evolve on top of it. it can be thought of as the 
second-lowest sub-layer of Layer 4, as follows:

| Application Layer         |
+---------------------------+
| Other Transport functions |2-ended
| Flow Regulation           |1-ended (sender)
| Feedback                  |2-ended
| Endpoint Identification   |2-ended
+---------------------------+
| Internetwork Layer        |

This influential paper about transport sub-layers separated out flow 
regulation as a sub-layer: 
http://conferences.sigcomm.org/hotnets/2008/papers/15.pdf
Taking sub-layering further, feedback (of delivery sequence and timing 
as well as ECN) is actually a sub-layer below flow regulation, because 
other higher transport functions depend on it, such as loss recovery, 
setting of timeouts, and so forth.

> Not reflecting the DSCP as required by the experimental protocol would indicate that it's not participating in the experiment.  That should be a strong indicator to the sender that it must switch o…

[BB] As above, making the L4S experiment depend on L4S deployment at the 
receiver is only of interest to those who don't care if the L4S 
experiment fails to deploy.

> Protecting an "alternate ECN" scheme with a DSCP is explicitly called out as a possibility in published RFCs.  The fact that is hasn't actually been done before should not be a barrier in itself.

[BB] The DSCP has been used for confinement - of pre-congestion 
notification (PCN):
     https://tools.ietf.org/wg/pcn/
Indeed Ruediger, Steven Blake and myself were involved in that.

PCN was one of a number of proposals for two ECN severity levels that 
motivated the publication of RFC4774  (all cited in the 2nd para of its 
introduction). PCN went on to be standardized in the IETF but only 
implemented by three router developers (AFAIK).

RFC4774 uses a DSCP in its traditional role as a classifier, as an 
example of a confinement mechanism. RFC4774 was not thinking of using a 
DSCP in the way you are proposing, as part of a per-connection 
cross-layer path traversal feedback loop, without its classifier 
function at all.

> That last one is not an argument.  We explicitly *want* to limit use of L4S signalling to connections where both endpoints know what's going on.

[BB] Inventing an unnecessary dependency to turn a 2-party deployment 
into a 3-party deployment is only of interest to someone who doesn't 
care about the viability of L4S.

> In your scenario, the customers are clearly within the participating network as defined by the border at which filtering of the DSCPs occurs.  Which means that any RFC-3168 equipment those customer…

[BB] Incorrect. I'm afraid it is just plain wrong to imagine that 
customers are inside the DSCP 'filtering' border of their ISP. An ISP 
re-marks DSCPs ('filters' in your language) at the IP hop that is 
closest to the attached customer but still under its own control. In the 
upstream of an access network, this is the Broadband Network Gateway 
(BNG), Cable Modem Termination System (CMTS), or equivalent. In the 
jurisdictions I know of, competition law allows a customer to source 
their own home gateway. So no ISP relies on DSCP bleaching at a home 
gateway.

> Please don't deliberately misunderstand me for the sake of argumentation.  Instead please re-read points 1 and 2 below from my earlier reply, in light of the brief clarification above.

[BB] No-one is /deliberately/ misunderstanding you. Everyone is going to 
misunderstand someone whose definition of an ISP's border is so 
"unorthodox".

>
>> [BB] Now, please address the a/b/c points below.
>>

[BB] Now that you've been corrected on where the ISP's borders are, 
these points are valid critique of your DSCP proposal. So you do need to 
address them. I notice you had snipped them, so I've reinstated.

>
>>>>
>>>> [BB] Now let's see how realistic LoF (2) is by putting this DSCP 
>>>> idea to the test - by hiding these needles in each type of haystack:
>>>>
>>>> a). Upstream:
>>>> Imagine a FIFO RFC3168 AQM in the queue of a home router in the 
>>>> upstream direction. An L4S client sends L4S packets through this 
>>>> AQM, which marks some as CE. Then the network participating in the 
>>>> L4S experiment delivers them, with the L4S-ID DSCP intact, to an 
>>>> on-net server. But the CE markings were RFC3168 not L4S.
>>>>
>>>> b) Downstream:
>>>> Consider an RFC3168 AQM in the network of a corporate site or 
>>>> campus network receiving traffic from a connected L4S-participating 
>>>> network. Or perhaps on the access into a data centre connected to 
>>>> the participating L4S network. Same problem; the DSCP says its L4S, 
>>>> but the CE was RFC3168.

[BB] The point in case (b) is that a provider network rarely (if ever) 
bleaches DSCPs at egress. And a receiving customer network will only 
bleach DSCPs at its ingress if it's using Diffserv itself. Many do not. 
But your DSCP proposal relies on bleaching at every border.

>>>> experiment. Same problem. The participating network delivers the 
>>>> packet with the L4S-ID DSCP intact, but the CE marking is RFC3168, 
>>>> not L4S.
>>>>
>>>> In this last case, the L4S-paticipating network has to use 
>>>> techniques like those in l4sops to make sure any FIFO RFC3168 AQMs 
>>>> don't mark ECT(1). That's no different to if ECT(1) alone was the 
>>>> identifier. 

[BB] We're trying to find someone interested in continuing this work. 
You already mentioned at the mic in IETF-110 that you'd read my new 
ideas on how to solve this in-band. But unfortunately, I can't find 
anyone who wants to devote their time to evaluating solutions to a 
probably hypothetical problem. As soon as a set of results is produced, 
everyone knows that you will contrive cases where it doesn't work and 
try to convince everyone that these are critical cases for the safety of 
the Internet. So I'm hardly surprised that no-one wants to run around on 
your playing field. But I'll keep on trying to find someone.

>> Here, we are putting your idea for using a DSCP to the test. So please address the point about how an L4S-participating network still has to use the techniques in l4sops, whether it adds a DSCP or…

[BB] I can confirm that you've accepted this point.

Regards

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 31 Mar 2021 01:41:00 +0300

> On 31 Mar, 2021, at 1:07 am, Bob Briscoe <[email-masked]> wrote:
> 
> We can now summarize the impact of this proposal to use a DSCP to confine the L4S experiment to one network:
> * You've agreed that the DSCP scheme doesn't itself avoid any FIFO RFC3168 ECN AQMs in the network participating in the experiment, or in its customer networks upstream or downstream. It still requires a participating L4S network to follow the l4sops draft to bypass these AQMs, as if they were using the original L4S ECN scheme.

Correct, although you mischaracterise this as a disadvantage.  The point of "containment" is to divide the Internet into a small number of networks that follow l4s-ops and participate in the L4S experiment, versus a much larger number of networks which do not need to know or care about L4S.  This makes deploying your experiment easier, not harder, so IMHO you should be welcoming it.

> * You've also agreed that the DSCP scheme needs changes to all standards track transport protocols, so that they somehow feed back the DSCP as well as the ECN field, even though bits are scarce, and even though the only motivation for these changes would be to make this DSCP idea work for L4S.

No, I have not agreed anything of the sort.  I have instead pointed out that because both endpoints of an L4S connection need to be L4S-aware for it to *be* an L4S connection, building in a DSCP reflection to these endpoints is not a showstopper - even if it means that both ends must be using an L4S congestion control for either end to do so.

Also, this does not prevent using ECT(1) as an indicator that one end is using L4S congestion control while the other is not.  See, there is a workaround.

> * In addition, the DSCP will not necessarily even serve its confinement function if it is within a pipe-mode tunnel.

I assume you mean "a tunnel that hides the DSCP field of its inner packets".  I don't know how prevalent these are, but if they are a feature of the network rather than the endpoint, they should be considered as part of the network border.

> In summary, this DSCP proposal adds very little to safety and it subtracts greatly from viability of the experiment.

 - Jonathan Morton

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 30 Mar 2021 19:12:52 -0400

> Okay, so what mechanism do you propose to contain the L4S experiment to the participating networks?

I think the notion of a strong containment guarantee to participating 
networks that you mention is a bit extreme in contrast to just reducing 
the likelihood or severity of issues. There has been a lot of congestion 
control experimentation on Internet paths that strong containment has 
not been expected of.

I think the sense of the working group has been to take an approach of 
collecting and recommending reasonable measures to identify and avoid 
issues when experimenting with L4S.  This is mainly in Section 4 of the 
ops draft right now.  I have not understood the general working group 
direction to be asking for strong containment guarantees though, and I 
don't think that needs to be a goal.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 31 Mar 2021 05:48:49 +0300

> On 31 Mar, 2021, at 2:12 am, Wesley Eddy <[email-masked]> wrote:
> 
> I think the notion of a strong containment guarantee to participating networks that you mention is a bit extreme in contrast to just reducing the likelihood or severity of issues.

Certainly I would prefer to see the underlying problems fixed, rather than adding safety barriers around the edges. Does that open the door to SCE-type signalling?  If not, why not?

> There has been a lot of congestion control experimentation on Internet paths that strong containment has not been expected of.

That's likely because the expected problems from most well-designed congestion controls, which are explicitly intended to coexist with existing traffic, are quite minor.  Once upon a time, TCP CUBIC was such an experiment.

I can also point to a major counterexample in the form of DCTCP.  L4S is a direct descendant of DCTCP, requiring a similar level of caution.  The main difference between the two is that L4S is capable of coexisting with conventional congestion controls in a correctly prepared network - but the "correctly prepared" part is crucial.

> I think the sense of the working group has been to take an approach of collecting and recommending reasonable measures to identify and avoid issues when experimenting with L4S.  This is mainly in Section 4 of the ops draft right now.  I have not understood the general working group direction to be asking for strong containment guarantees though, and I don't think that needs to be a goal.

My best read of L4S proponents' intentions is that the very moment they have an Experimental RFC published that sanctions a deployment of L4S (as required by RFC-8311), they will start pushing firmware updates and new kernels across a wide variety of cable networks in order to deploy "Low Latency DOCSIS", which includes L4S functionality as a core feature.  Far from a small-scale, carefully controlled experiment, this would be a large-scale deployment with the explicit intention of being hard to roll back.

I believe this has a strong likelihood of causing L4S traffic to flow not only within these networks and their attached CDNs, but as an unintended side effect, over other networks as well.  We have already discussed at length the problems this is likely to cause to uninvolved bystanders, which may even include unsuspecting cable ISP subscribers who suddenly start having L4S-enabled set-top boxes alongside their conventional-traffic computers.

Nothing I have heard in the past week has reassured me on those points.  Rather, the most basic of precautions is rejected as unworkable, and the simplest of modifications to L4S to enable them dismissed as infeasible.  Bob Briscoe even seems to think simultaneously that an ISP's customers are outside the "prepared network" for the experiment, and that putting a firewall to prevent L4S traffic reaching them would be absurd.  The cognitive dissonance is strong with that one.

Hence the question at the top.  I think it has reached maximum relevance.

- Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Tue, 30 Mar 2021 23:19:34 -0400

On Tue, 2021-03-30 at 19:12 -0400, Wesley Eddy wrote:
> > Okay, so what mechanism do you propose to contain the L4S
> > experiment to the participating networks?
> 

How many of those experimental congestion control protocols set a RFC
3168-defined ECT marking on their packets and then failed to respond to
a RFC 3168-defined CE congestion signal by triggering multiplicative
decrease?

> don't think that needs to be a goal.

The recommended measures in Sec. 5 (not 4) of l4sops-02 are directed at
the operators deploying RFC 3168 and *not participating in the L4S
experiment* and consist of either disabling RFC-3168 ECN or of
instantiating configurations that depend on the capability to classify
on ECT(1) markings specifically, a configuration feature that doesn't
appear to be universally available in deployed software. Opinions
differ on whether these measures are "reasonable" or are directed at
the appropriate parties.

Regards,

// Steve

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 31 Mar 2021 09:57:26 +0200

Dear Bob,

see more below, prefixed [SM]. Since your post did not contain anything of sensitive nature, I opted for moving this back on list.

> On Mar 31, 2021, at 00:24, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> I understand differences of opinion are valid. But all your three points were off-topic (and incorrect, as it turns out).

[SM] I understand your mail, as a tacit acknowledgment that you accept my arguments as you did NOT respond to my arguments or cited facts in any substantial fashion.

	You are certainly free to argue that my points are incorrect, but then it is incumbent on you to actually show, preferably by citing facts, how you came to that assessment.
Otherwise it just reduces your argument to an unsubstantiated statement of opinion. Which I add, says a lot about you, but little about the question at hand.
	Now, I understand what and why you are doing this, you are trying to shift the discussion away from points where your argument don't hold water. This is a time-honored debating strategy, used when the goal is to "score points" rather than as a discussion strategy that aims at actually solving problems by findig compromises (where they are possible).
	So let me get back to the question, what in the guard DSCP proposal is unfeasible, EXCEPT for not fulfilling your desire to force-recruit all internet users at the leisure of the L4S experiments participants? So, please tell me, why you consider requiring an explicit opt-in by those that will have to deal with the consequences (both positive and negative) as being out of the question? I can not help but repeating, that if you yourself consider L4S benefits so weak, that they will not survive such a common sense requirement, it might be time to drop the IDs. 

Best Regards
	Sebastian

> 
> 
> Bob
> 
> On 30/03/2021 11:46, Sebastian Moeller wrote:
>> Bob,
>> 
>> 
>>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 31 Mar 2021 10:26:32 +0200

Hi Bob

since this is relevant as a document on how little research you use to back up your FUD, I moved this back on list.

> On Mar 31, 2021, at 00:32, Bob Briscoe <[email-masked]> wrote:
> 
> Rod, Sebastian,
> 
> You're correct that a receiver with an open source OSs like Linux or FreeBSD can access the DSCP.

	[SM] A fact that you could have researched yourself before asking your broad question "is DSCP feed-back/reception actually possible". The fact that you did not, makes clear that your argument was not in good faith, but intended to paint any DSCP proposal as untenable, independent on the underlaying facts.

> But the main closed source ones cannot.

	[SM] Can not, or, Bob has still not done his hoemwork?
Have a look at https://www.fekt.vut.cz/conf/EEICT/archiv/sborniky/EEICT_2012_sbornik/03doktorskeprojekty/10pocitacovesystemy/01-xkrkos04.pdf:
"This paper describes the design and implementation of a driver for network traffic mark- ing to ensure Quality of Service provision on Microsoft Windows based terminal devices."
It seems quite obvious that to achieve that the developer has had to access the DSCP bits from within kernel space, and lo and behold, it was possible ot do so from a kernel space driver.

BUT the bigger issue is, that currently no TCP Prague prototype exists for anything but Linux, so it is a bit disingenuous to argue about platforms you do not even considered so far at all.
So, please strike windows of your list for now.

> In Windows,

	[SM] See above, it seems your assessment of what is possible in windows might have been driven more by your desired outcome than windows capabilities, sorry.

> MacOS and iOS,

	[SM] Mmmh, given that Macos (and iOS) inherited a lot of their networking facilities from FreeBSD if I am not mistaken, are you sure that the method that works in FreeBSD will not also work in Apple's OSs?

> the sending app cannot even choose the DSCP it uses.

	[SM] That is different from what we actually require here, for an in-kernel protocol driver, it seems irrelevant whether user space can read/set DSCPs no? That fact that you confuse this makes me believe you are not aiming at learning something here, but simply are trying to muddy the waters.

> The API only allows it to select from a list of application types, then the stack determines which DSCP to use for this application (dependent on uPNP queries to the access router, or defaults otherwise).

	[SM] Again assuming that is true, how is that relevant to an in-kernel protocol?

> 
> Whatever, I've responded off-list, because the higher-order point that I already stated on the list is that the receiving application doesn't know anything about the CC that the sender is using, so it won't have any logic to go looking for this DSCP.

	[SM] Except that even unidirectional L4S CC will require AccECN negotiation for the link, which requires an updates protocol at the other end, as no major operating system supports AccECN by default yet. This is again, smoke and mirrors instead of a honest discussion. And again I see that you plan to force-recruit all internet users into your little experiment, without first asking for their consent... that is not going to work well, given the sorry state of the L4S core technologies.

Sebastian

P.S.: I really dislike the fact tat you turn any attempt of en open discussion into a debate instead.

> 
> 
> Bob
> 
> On 30/03/2021 14:26, Rodney W. Grimes wrote:
>>> Bob,
>>> 
>>> since you asked,
>>>

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 31 Mar 2021 11:51:24 +0000

Sebastian

My understanding is that _nobody_ outside your quite small group is asking
for this extra DSCP dependency for the L4S experiment. 
Furthermore I don't see how you are about to get support and energy to
standardize/implement DSCP feedback in QUIC and RTCP (and TCP) in the near
future. 
So my conclusion is that the only objective with this lengthy discussion, as
well as the other dead cat recently placed on the table (deprecate RFC3168),
is to delay L4S as much as possible or just grind it down with repeated
discussions that go in circles. 

But now you got almost 2 years to express your opinion, it's time to move
on.

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
> Sent: den 31 mars 2021 10:27
> To: Bob Briscoe <[email-masked]>; tsvwg IETF list <[email-masked]>
> Subject: Re: [tsvwg] [on list again] [offlist] L4S DSCP (was: L4S drafts:
Next
> Steps)
> 
> Hi Bob
> 
> since this is relevant as a document on how little research you use to
back up
> your FUD, I moved this back on list.
> 
> 
> 
> > You're correct that a receiver with an open source OSs like Linux or
FreeBSD
> can access the DSCP.
> 
> 	[SM] A fact that you could have researched yourself before asking
> your broad question "is DSCP feed-back/reception actually possible". The
> fact that you did not, makes clear that your argument was not in good
faith,
> but intended to paint any DSCP proposal as untenable, independent on the
> underlaying facts.
> 
> Have a look at
> https://www.fekt.vut.cz/conf/EEICT/archiv/sborniky/EEICT_2012_sbornik/03
> doktorskeprojekty/10pocitacovesystemy/01-xkrkos04.pdf:
> "This paper describes the design and implementation of a driver for
network
> traffic mark- ing to ensure Quality of Service provision on Microsoft
Windows
> based terminal devices."
> It seems quite obvious that to achieve that the developer has had to
access
> the DSCP bits from within kernel space, and lo and behold, it was possible
ot
> do so from a kernel space driver.
> 
> BUT the bigger issue is, that currently no TCP Prague prototype exists for
> anything but Linux, so it is a bit disingenuous to argue about platforms
you do
> not even considered so far at all.
> 	[SM] See above, it seems your assessment of what is possible in
> windows might have been driven more by your desired outcome than
> windows capabilities, sorry.
> 
> > MacOS and iOS,
> 
> 	[SM] Mmmh, given that Macos (and iOS) inherited a lot of their
> networking facilities from FreeBSD if I am not mistaken, are you sure that
the
> method that works in FreeBSD will not also work in Apple's OSs?
> 
> 
> 	[SM] That is different from what we actually require here, for an
in-
> kernel protocol driver, it seems irrelevant whether user space can
read/set
> DSCPs no? That fact that you confuse this makes me believe you are not
> aiming at learning something here, but simply are trying to muddy the
> waters.
> 
> > The API only allows it to select from a list of application types, then
the
> stack determines which DSCP to use for this application (dependent on uPNP
> queries to the access router, or defaults otherwise).
> 
> 	[SM] Again assuming that is true, how is that relevant to an
in-kernel
> protocol?
> 
> >
> > Whatever, I've responded off-list, because the higher-order point that I
> already stated on the list is that the receiving application doesn't know
> anything about the CC that the sender is using, so it won't have any logic
to
> go looking for this DSCP.
> 
> 	[SM] Except that even unidirectional L4S CC will require AccECN
> negotiation for the link, which requires an updates protocol at the other
end,
> as no major operating system supports AccECN by default yet. This is
again,
> smoke and mirrors instead of a honest discussion. And again I see that you
> plan to force-recruit all internet users into your little experiment,
without
> first asking for their consent... that is not going to work well, given
the sorry
> state of the L4S core technologies.
> 
> Sebastian
> 
> P.S.: I really dislike the fact tat you turn any attempt of en open
discussion
> into a debate instead.
> >>> In Linux look at include/net/dsfield.h
> >>> (https://protect2.fireeye.com/v1/url?k=28f09509-776bac22-28f0d592-86
> >>> 6038973a15-468a77759b2e090e&q=1&e=a4b34fc4-b4d7-4080-bca4-
> e40a6fc5d9
> >>>
> a8&u=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Flatest%2Fsource%2Fin
> >>> clude%2Fnet%2Fdsfield.h%23L16) This will give the ability inside the
> >>> kernel to get and set DSCPs for IPv4 and IPv6...
> >>> Given that TCP Prague lives inside the kernel itself that should not
be to
> hard to use.
> >>> That will not help for other OSs, but since these do not have L4S
> compatible protocols today (with TCP Prague being the closest to
> compatibility) that should not be a show stopper.
> >> Sebastian, Bob, twvwg,
> >> I am aware that FreeBSD has similiar features, and some of the
> >> userland daemons and clients can even be configured to use DSCP's.
> >>
> >>
> >>> Best Regards
> >>> 	Sebastian
> >> Regards,
> >
> > --
> >
> > Bob Briscoe
> https://protect2.fireeye.com/v1/url?k=023db00c-5da68927-023df097-
> 866038973a15-5a76d97f5aa5ed60&q=1&e=a4b34fc4-b4d7-4080-bca4-
> e40a6fc5d9a8&u=http%3A%2F%2Fbobbriscoe.net%2F
> >

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 31 Mar 2021 09:10:07 -0400

> a RFC 3168-defined CE congestion signal by triggering multiplicative

The issue is well understood.  The doc talks about detection and 
fallback, and has placeholders for more content in this area.

> differ on whether these measures are "reasonable" or are directed at

Section 4 has a number of precautions, as stated, for experimenting with 
L4S.

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 31 Mar 2021 09:17:55 -0400

On 3/30/2021 10:48 PM, Jonathan Morton wrote:
>> On 31 Mar, 2021, at 2:12 am, Wesley Eddy<[email-masked]>  wrote:
>>

I have not seen indication that the mass of people working on this are supportive of this type of change.

---

From: "Black, David" <[email-masked]>
Date: Wed, 31 Mar 2021 18:18:54 +0000

Ingemar,

> for this extra DSCP dependency for the L4S experiment.

I think that depends on exactly what is meant by "extra DSCP dependency" - I think there's broader interest in a Guard DSCP that does not entail L4S redesign (and I would characterize both Jonathan and Sebastian's proposals as redesigning one or more core elements of L4S).

My day job is making my life entirely too busy, but in the hope of fostering productive discussion, I'll try to put together a summary (as an individual) of what a Guard DSCP could look like whose sole purpose is to better equip operators to deal with things that might go wrong in an L4S experiment (without redesigning the experiment).  Patience, please ...

Thanks, --David

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Ingemar Johansson S
Sent: Wednesday, March 31, 2021 7:51 AM
To: Sebastian Moeller; tsvwg IETF list
Subject: Re: [tsvwg] [on list again] [offlist] L4S DSCP (was: L4S drafts: Next Steps)

Sebastian

/Ingemar

Next
back up
FreeBSD
faith,
network
Windows
access
ot
you do
the
in-
read/set
the
in-kernel
to
end,
again,
without
the sorry
discussion
be to

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 31 Mar 2021 22:28:43 +0100

Sebastian,

I have helped you read the specs below. Personally I don't think anyone 
else is under the same misapprehensions as you about what the specs are 
saying, so I don't think this ought to be on the list, but you insisted, 
so pls see inline [BB]...

> 	So let me get back to the question, what in the guard DSCP proposal is unfeasible, EXCEPT for not fulfilling your desire to force-recruit all internet users at the leisure of the L4S experiments p…
[BB] Insults ignored.
more...

>>> 	[SM] Bob, when I come to different conclusions from evaluation the same data as you, that does not automatically mean that I am misunderstanding something. there could be different reasonable in…

[BB] Insults ignored
more...

>>> 	I can't shake the feeling, your are not actually trying to engage with my arguments, but rather try to swat them down with as little effort as possible...

[BB] Insults ignored.
more...

>>> 	But I understand you as saying that you categorically reject adding any DSCP requirement to the L4S drafts whether transiently for the duration of the experiment or permanently as part of the si…

[BB] Your proposal is not what the l4sops draft is about. Last May the 
chairs confirmed that the WG would move forward with L4S work on the 
basis of the currently documented L4S identifier. The l4sops draft has 
just been adopted with a clear statement of scope, concerning the L4S 
identifier and traffic that shares a queue where there is an RFC3168 AQM.

Given that WG decision, we shouldn't even be engaging in discussions of 
alternative identifiers on the list. But I recently decided to engage in 
the discussions about a guard DSCP because I couldn't see how the scheme 
could possibly solve the CE ambiguity problem, and I detected that some 
people thought it could.

I engaged first with Jonathan about his guard DSCP proposal, not because 
he's any more important than you. but because the description of even 
one scheme was confusing everyone, let alone different people chipping 
in with slight differences. I'll discuss yours once discussion on his 
has converged.

>>> 	[SM] IMHO at the expected main deployment points, at the DOCSIS-ISP<-> end-user edge tunneling will simply be a minor issue that could for the first iteration of L4S be ignored, and for deeper b…

[BB] The WG has already decided to take into account factors like 
tunnelling in its choice of identifier.

As a for instance, the last operator I worked for was an ISP itself, but 
it also wholesaled ISP services to retail ISPs. When I left 6 years ago, 
60% of end-customers reached their ISP via an L2TP tunnel over this 
wholesaler.

You will see a large number of encapsulation protocols in rfc7059. When 
I took on editorship of draft-ietf-tsvwg-rfc6040update-shim, in order to 
make the scope manageable, I asked the IETF intarea to tell me which 
tunnel protocols currently in active use or development involve a shim 
under an outer IP header. That resulted in this cut-down list:
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-rfc6040update-shim-13#section-6
That's doesn't include straightforward IP in IP.

I already knew tunnels and encapsulations are everywhere in the 
Internet. Both between 2 points within the network, and between 
end-systems. Having worked with the maintainers of each of those 
protocols, I discovered even more types of tunnel I didn't even know 
about, that are in widespread use.

>>>>> This is pretty much what you seem to argue for as being impractical, no?

[BB] You are talking about SLAs between networks. The reason I moved 
this offlist was because stuff like this has got nothing to do with the 
point at hand - deploying DSCP receiving logic in the transport 
protocols of receiving hosts all around the Internet.

>>> 	[SM] Well, these are about enabling NQB end to end, which would be e necessary condition for using DSCPs in the context I proposed, these quotes indicate that the sponsors of the NQB draft see t…

[BB] See my explanation about the ambiguity of end-to-end between L3 
people and L4 people.

>>> 	Honestly, Bob, these to questions indicate that you have not even started to look into how feasible a guard DSCP would be, but just trying to muddy the waters with under-researched objections, m…

[BB] Insults ignored.

This is not about whether the DSCP /can/ be read at the receiver.

ECN deployment was painstakingly slow mainly because it required new 
code to be deployed in 3 places: at senders, receivers and at the 
bottleneck. L4S requires updated code in 2 places (at the sender and in 
the network), which makes deployment difficult enough. Requiring 
receivers to have updated logic as well would knock back deployment by 
decades.

Example: an ISP with 20 million users, each with an average of 8 devices 
(160million devices). Imagine the ISP deploys L4S technology at its 
bottlenecks. You deploy L4S on your host. But, even once 10,000 hosts on 
that ISP have upgraded to L4S, your L4S device can still see 159,990,000 
non-L4S devices over that ISP, so the chances of you being able to use 
L4S with anyone else are tiny. In contrast, when there is no dependency 
on the receiver, when you choose that ISP and you choose to upgrade to 
L4S yourself, you get to use it all the time.

(AccECN requires changes at the receiver, and L4S depends on AccECN if 
it is to be used over TCP. But that doesn't hold back L4S deployment 
over other transports, such as RTP/RTCP or QUIC which have suitable ECN 
feedback logic at the receiver already.)

>>> 	[SM] Reading the relevant CableLabs DOCSIS standards and the Low Latency DOCSIS White paper (where you were one of the co-authors) clearly indicates, that NQB is very much intended as the canoni…

[BB] Insults ignored...

If you read the specs, I wouldn't have to spend all this 1-1 tutorial 
time with you telling you what the specs say.
Have you read 
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-11#section-5.4.1.1 
?
Have you read the first four paras of Section 7.7.5 of the DOCSIS MULPI 
spec? https://www.cablelabs.com/specifications/CM-SP-MULPIv4.0

And as well as looking for some evil conspiracy about a "missing" DSCP 
in L4S, have you considered that the words in the specs might just mean 
exactly what they say?

It is very relevant that NQB is not L4S. They are certainly 
complementary - for instance DOCSIS shows that they can share the same 
queue. But they are no more complementary than the Expedited Forwarding 
(EF) DSCP and L4S. And NQB certainly cannot be described as /being/ L4S.
* L4S traffic gets its low latency from its congestion control response, 
hence the use of the ECN field
* NQB traffic gets its low latency from light and non-bursty use of 
capacity, but it is unresponsive if congestion does occur, hence the use 
of a DSCP.

* L4S traffic carries ECT(1) in the ECN field, but typically the default 
(zero) DSCP (Best Efforts).
* NQB traffic carries the NQB DSCP and Not-ECT in the ECN field (given 
it is typically unresponsive).

Both identifiers on the same packet are not expected, but it is 
perfectly possible that traffic is both light and responsive. And in 
systems where they share the same queue (like DOCSIS) it anyway doesn't 
matter which classifier is applied first.

draft-ietf-tsvwg-ecn-l4s-id-11#section-5.4.1.1 gives some examples of 
other identifiers that could be classified into the same queue as L4S:
   o EF, Voice Admit, and other similar PHBs
   o DNS, ARP
But this doesn't mean that L4S /is/ EF.

At the risk of patronizing you, let me explain that industry-specific 
standards bodies (like DOCSIS, Broadband Forum, 3GPP, etc) pull together 
component standards (often from the IETF) into system standards. 
Operators of those network technologies can then expect to be able to 
purchase the interoperating parts of these systems from their equipment 
vendors, without each having to define their own systems and their own 
testing regimes.

DOCSIS /uses/ L4S and NQB, just as it uses Ethernet and DHCP. They are 
all items in the recipe that makes up DOCSIS.
Similarly a Salade Nicoise is not a conspiracy between tuna, french 
beans and eggs. They are just /used in/ the recipe.

* L4S was released 2 years before services like NQB were thought of.
* Greg White was working on a new low latency service for DOCSIS. It was 
intended for non-queue-building traffic (lower case, originally just his 
description of the type of traffic). When he read about L4S, he realized 
his traffic could share a queue with L4S to reduce complexity.
* Thomas Fossati and others had got significant interest from the 3GPP 
in a similar low latency service, originally called LoLa. Given its 
success in 3GPP, he (and others) described it in a draft for the IETF: 
draft-fossati-tsvwg-lola-00
* NQB and LoLa eventually came together in the NQB proposal.
* In future, it is possible the 3GPP might put some of its low latency 
services (identified by DSCPs) together with L4S in the same queue. This 
doesn't make these DSCPs part of L4S. Just as eggs are not part of tuna.

>>> I often wonder about the level of language lawyering happening in drafts and discussions, I would assume what matters more is what happens/is going to happen in reality than any amount of word sm…

[BB] L4S traffic doesn't have an NQB DSCP. And NQB traffic doesn't have 
ECT(1) in the ECN field. There is no missing DSCP on L4S packets. They 
are just two types of traffic that can coexist in the same queue. That's 
precisely what this text says. Please read what is in the lines of text. 
There is nothing written between the lines.

Bob

>>>>>> http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 1 Apr 2021 00:07:39 +0100

Sebastian,

Unfortunately, yet more correction is needed, just to clear my name of 
your accusations, which were off topic, which is why I took this 
conversation off list. But you've insisted it's onlist, so (sigh) pls 
see inline, tagged [BB]...

> 	[SM] A fact that you could have researched yourself before asking your broad question "is DSCP feed-back/reception actually possible". The fact that you did not, makes clear that your argument was…

[BB] Nothing in my emails was about whether one /can/ access the DSCP.
When you replied with "Since you asked," I hadn't asked.
I had said "[BB] None of these 3 quotes about NQB are anything to do 
with DSCP feedback, nor reading the DSCP at the receiver."

But you went on to tell me anyway, that one can access the DSCP in Linux.

Like most people on the list I already knew that you can (in FOSS OSs). 
Indeed, you've been able to access the DSCP in Linux since 1998 when 
Diffserv was first defined, and I'm sure FBSD was siimilar.

I took this off list 'cos it wasn't relevant.

What I said was nothing like it is not /possible/ to alter standards 
track protocols to feed back the DSCP. The point was that:
a) no-one is going to be motivated to alter all the standards track 
transport protocol specifications to feed back the DSCP just so that 
your DSCP scheme will work
b) it would give L4S deployment an additional dependency on L4S logic at 
the receiver, when that is currently not the case.

Please don't be so quick to blame me, when I think it was just you 
reading my email too fast.
In general, I think many on this list would appreciate you reading more, 
and writing less.

Bob

PS. Rod, your info about FBSD was understandable, given Sebastian had 
made it look like it would be relevant.