Subject: [tsvwg] Reasons for WGLC/RFC asap

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 18 Nov 2020 10:31:30 +0000

Hi all,

To continue on the discussions in the meeting, a recap and some extra thoughts. Did I miss some arguments?

Benefits to go for WGLC/RFC asap:

  *   There is NOW a big need for solutions that can support Low Latency for new Interactive applications
  *   The big L4S benefits were a good reason to justify the extra network effort to finally implement ECN in general and AQMs in network equipment
  *   Timing is optimal now: implementations in NW equipment are coming and deployment can start now
  *   Deployment of L4S support will include deployment of Classic ECN too! So even for the skeptics among us, that consider that the experiment can fail due to CCs not performing to expectations, we will fall back to having Classic ECN support
  *   Current drafts are about the network part, and are ready and stable for a very long time now.
  *   Only dependency to CCs in the drafts are the mandatory Prague requirements (only required input/review from future CC developers: are they feasible for you)
  *   We have a good baseline for a CC (upstreaming to Linux is blocked by the non-RFC status)
  *   Larger scale (outside the lab) experiments are blocked by non-RFCs status
  *   It will create the required traction within the CC community to come up with improvements (if needed at all for the applications that would benefit from it; applications that don't benefit from it yet, can/will not use it)
  *   NW operators have benefits now (classic ECN and good AQMs) and in the future can offer their customers better Low Latency experience for the popular interactive applications
  *   When more L4S CCs are developed, the real independent evaluation of those can start

Disadvantages to wait for WGLC/RFC:

  *   We'll gets stuck in an analysis paralysis (aren't we already?)
  *   Trust in L4S will vanish
  *   No signs that we can expect more traction in CC development; trust and expectations of continuous delays will not attract people working on it, as there will be plenty of time before deployments are materializing
  *   Product development of L4S will stall and die due to uncertainty on if L4S will finally materialize
  *   Product development of Classic ECN will stall and die due to uncertainty on how L4S will finally materialize

What are the advantages to wait? Do they overcome these disadvantages?

Regards,
Koen.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 18 Nov 2020 13:16:29 +0200

> On 18 Nov, 2020, at 12:31 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> • There is NOW a big need for solutions that can support Low Latency for new Interactive applications

I've heard this argument several times in recent WG sessions, usually phrased as "strong interest in L4S".  I think it is worth clarifying whether it is *actually* an interest in L4S specifically, or rather an interest in high-fidelity congestion control in general, of which L4S is the example that happens to be most active in the IETF.  The two might possibly be conflated in some minds.

I would hope that if there genuinely is interest in high-fidelity congestion control, with all that it can potentially enable, then the WG should look for a mechanism that is likely to succeed, and to be robust in the face of future developments.

So, I ask the following question:

Assume that L4S is shown to be unable to meet one or more of the Prague Requirements, and/or is unable to present a suitable safety case for approval as an Internet Experiment.  Should resources and support formerly allocated to L4S development be redirected to some alternative High Fidelity Congestion Control proposal which can rapidly meet those tests?  Yes or No.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 14:45:52 +0100

HI All,

as you might predict, I object. More verbosely below in-line pre-fixed with [SM]

> On Nov 18, 2020, at 11:31, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi all,
>  

	[SM] citation needed. That is simply an assertion and not a fact. Compared to the best of class AQM's ~5ms average queueing delay (with existing TCP mind you) L4S' 1ms really is only a minor improvement, so whoever is waiting now, is simply unaware of what is the state of the art.

> 	• The big L4S benefits were a good reason to justify the extra network effort to finally implement ECN in general and AQMs in network equipment

	[SM] Which big benefits again, certainly not introducing a robust and reliable bias for L4S over non-L4S flows also not introducing even more RTT-dependence than we have right now, nor the rfc3168 compatibility issues, or DualPI2's inherent misdesign...

> 	• Timing is optimal now: implementations in NW equipment are coming and deployment can start now

	[SM] That exact same argument will be exactly as true in any time point in the future. This is a rather thin argument for L4S, really.

> 	• Deployment of L4S support will include deployment of Classic ECN too! So even for the skeptics among us, that consider that the experiment can fail due to CCs not performing to expectations, we will fall back to having Classic ECN support

	[SM] Remind me again, how the outcome of the experiment is supposed to be measured, and how the experiment on failure is to be un-done?

> 	• Current drafts are about the network part, and are ready and stable for a very long time now.

	[SM] But assume all the hopes the network parts put in the protocols bear out in the end. Since these hopes include operational safety I object that the current network parts are ready and stable for a very long time. There is a standing list of issues with the network parts, like bias against non-L4S flows in general, unexplained magic variables all over the place, like the completely missing rationale for the 15ms delay-Target setting for the deep-queue side of the DualPI2. Non of this is new, but a documented short-coming will not automatically becom acceptable only because it has not been tackled for X years.

> 	• Only dependency to CCs in the drafts are the mandatory Prague requirements (only required input/review from future CC developers: are they feasible for you)

	[SM] This is bad engineering to outsource your safety measures to "stuff still to be designed", as it is bad engineering to assume that an optimal interplay between AQM and end-points can be designed by just thinking about the AQM conponent.

> 	• We have a good baseline for a CC (upstreaming to Linux is blocked by the non-RFC status)

	[SM] Please cite the mail from the net-dev list showing that objection, please. Or do you voluntarily refrain from asking for inclusion?

> 	• Larger scale (outside the lab) experiments are blocked by non-RFCs status

	[SM] Which would be stronger argument if all relevant lab experiments were already concluded with L4S coming out as ready for prime time. Recently posted data puts that very much in question.

> 	• It will create the required traction within the CC community to come up with improvements (if needed at all for the applications that would benefit from it; applications that don’t benefit from it yet, can/will not use it)
> 	• NW operators have benefits now (classic ECN and good AQMs) and in the future can offer their customers better Low Latency experience for the popular interactive applications

	[SM] Well, sure your bastardized version of PIE is better than no AQM (though I wonder about the consequences of you ripping-out the burst tolerance code, if you have data showing how this affects its operation, please post a link).

> 	• When more L4S CCs are developed, the real independent evaluation of those can start

	[SM] Which is not gated on L4S getting L4S status at all, only on convincing CC developers that L4S has enough merit to spend some time. What we heard here on the list is that even staunch L4S proponents like Ingemar, have refrained from creating an L4S requirements-compliant CC for their protocol, hardly a show of confidence.

> 	• We’ll gets stuck in an analysis paralysis (aren’t we already?)

	[SM] Nope, we are in, the issues are openly documented, but the developers try to outwait the critics instead of making the required real changes.

> 	• Trust in L4S will vanish

	[SM] After 7+ years, one or two more years are not going to catastrophically change the trust level, that is  not a logical argument.

> 	• No signs that we can expect more traction in CC development; trust and expectations of continuous delays will not attract people working on it, as there will be plenty of time before deployments are materializing

	[SM] Which the exact rationale, why the network bits need to be redesigned and changed to require much less cooperation from the CC side for L4S to reach its goals and promises (and maybe scaling down the promises)

> 	• Product development of L4S will stall and die due to uncertainty on if L4S will finally materialize

	[SM] Which might be the best outcome for customers until it has been demonstrated that L4S can robustly, reliably, and safely deliver its promised performance over the existing internet. Trust in L4S is going to vanish much quicker, if it turns out that it falls short of its promises and does not deliver.

> 	• Product development of Classic ECN will stall and die due to uncertainty on how L4S will finally materialize

	[SM] That is a non-sequitur, if at all the market opportunity for rfc3168 solutions will be larger if there is not a direct competitor. 

> What are the advantages to wait? Do they overcome these disadvantages?

	[SM] Waiting, like we did basically for the last two years is not giving any advantages. Using say, the next two years to actually test and modify the L4S network and CC bits to robustly, reliably and fairly deliver on their promises however would be well spent time. And if that dioes not happen in that time, it might be time to call the experiment off.

Best Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 18 Nov 2020 17:29:27 +0000

Hi Sebastian,

Can you revisit these arguments again from the non-Flow identifying point of view? If compared with an FQ, clearly a better flow rate can be enforced. L4S can also support an FQ solution, so the comparison should be fair here. So either compare FQ_L4S with FQ_Codel or compare DualPI2 with Single Queue CoDel.

Regards,
Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Wednesday, November 18, 2020 2:46 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] Reasons for WGLC/RFC asap

HI All,

Best Regards
	Sebastian

---

From: Steven Blake <[email-masked]>
Date: Wed, 18 Nov 2020 12:50:05 -0500

On Wed, 2020-11-18 at 10:31 +0000, De Schepper, Koen (Nokia -
BE/Antwerp) wrote:
> There is NOW a big need for solutions that can support Low Latency
> for new Interactive applicationsThe big L4S benefits were a good
> reason to justify the extra network effort to finally implement ECN
> in general and AQMs in network equipmentTiming is optimal now:
> implementations in NW equipment are coming and deployment can start
> nowDeployment of L4S support will include deployment of Classic ECN
> too! So even for the skeptics among us, that consider that the
> experiment can fail due to CCs not performing to expectations,
>  we will fall back to having Classic ECN supportCurrent drafts are
> about the network part, and are ready and stable for a very long time
> now.
> Only dependency to CCs in the drafts are the mandatory Prague
> requirements (only required input/review from future CC developers:
> are they feasible for you)We have a good baseline for a CC
> (upstreaming to Linux is blocked by the non-RFC status)

These are arguments to deploy a scheme that has been demonstrated to
work. Not arguments to conduct an *experiment* to determine whether a
scheme is capable of working (and under what conditions?).

Do you completely ignore the simulation results that show that L4S
currently does not live up to it's own hype? How are you going to
conduct the *experiment* without at least one CC that seems to work in
simulation?

> Larger scale (outside the lab) experiments are blocked by non-RFCs
> status

Why? What is preventing an interested network operator from conducting
L4S *experiments* now, using an experimental DSCP to signal L4S
packets? What is preventing two interested directly peering operators?
What is preventing two interested but non-peering operators from
running tunnels between themselves?

> It will create the required traction within the CC community to come
> up with improvements (if needed at all for the applications that
> would benefit from it; applications that don’t
>  benefit from it yet, can/will not use it)NW operators have benefits
> now (classic ECN and good AQMs) and in the future can offer their
> customers better Low Latency experience for the popular interactive
> applicationsWhen more L4S CCs are developed, the real independent
> evaluation of those can start
>  
> We’ll gets stuck in an analysis paralysis (aren’t we already?)Trust
> in L4S will vanishNo signs that we can expect more traction in CC
> development; trust and expectations of continuous delays will not
> attract people working on it, as there will be plenty of time before
>  deployments are materializingProduct development of L4S will stall
> and die due to uncertainty on if L4S will finally materializeProduct
> development of Classic ECN will stall and die due to uncertainty on
> how L4S will finally materialize
>  

As I have said before, the only thing blocking the L4S *experiement*
from getting IETF blessing IMHO is the wholly unnecessary insistence on
consuming an IP header bit *now* with no plan to relinquish it if the
*experiment* fails. Every objection raised to my statement has been an
argument predicated on ease of deployment. Ease of deployment as a
standard is something to be considered, yes. The allocation of an IP
header bit can be considered *after* evaluating the results of the
*experiment*, especially when there are practical work-arounds (e.g.,
experimental DSCP) to enable the *experiment* to proceed.
I see the same style of argument from particle physicists who want to
spend $40B to build a 150-km circumference particle collider, to find
what exactly? (Especially considering that all current beyond-Standard
Model theories have totally bombed).

Regards,

// Steve

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 20:27:35 +0100

Hi Koen,

> On Nov 18, 2020, at 18:29, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> Can you revisit these arguments again from the non-Flow identifying point of view?

	[SM] I think I already did, "sure your [...] version of PIE is better than no AQM", but that is pretty much where it ends. I am involved in supporting OpenWrt users to debloat their internet access links with SQM. BTW those should be pretty much your core audience, but are is not. 
	A number of yeas ago we came up with a pretty workable solution, by using (costly) traffic shapers to keep traffic away from the typically over-sized and under-managed buffers on both sides of typical internet access links. But that only moved the problem somewhere, where home users could tackle it. And it turned out for most users a competent modern AQM pretty much squashes most latency issues pretty well. Initially, sfq was the qdisc of choice, but that was quickly superseded by fq_codel, because that basically is the conceptual successor of sfq an notoceably better in a number of dimensions. 
	That worked amazingly well for most users. And those users unhappy with that solution ofen, we realized, did really require an explicitly unfair set-up, like reliably trying to send 8Mbps real-time video streams over a 10Mbps link. In that case the solution was to create dedicated shaper and filter hierarchies that allow for the desired targeted unfairness. 
	Cake then upped the ante by creating a two tier fairness system, which for example allows to share equitably between all active hosts in a network (and then still offer flow queueing inside of each host share), that pretty much solved the home access latency under load issue and the" please stop downloading when I am in a video conference" issue as well. 
	L4S as currently designed and implemented offers very little of that. It does not do host equitable sharing nor flow fairness, nor targeted unfairness, it offers some sort of "anything goes" with an admittedly lower average queueing delay (albeit only a few milliseconds lower, not multiple orders of magnitude). In short for the problem space where I sit, L4S solves nothing, but threatens to introduce additional sources of unfairness (because its bias against non-L4S flows, its increased RTT dependence, its unsuitable coupling core idea, ...).

> If compared with an FQ, clearly a better flow rate can be enforced. L4S can also support an FQ solution, so the comparison should be fair here.

	[SM] I tend to compare existing apples with existing apples, and there is no FQ L4S solution that I can compare here. Sure my existing FQ-AQM solution is probably inferior to your hypothetical FQ_L4S, but a) without supporting data I do not believe that (I am not that raid a FQ-fan boy, an FQ_L4S solution would still have to pass the same set of safety and sanity tests, I assume that it will be easier to pass them, but I want to see the data nevertheless) and b) that knowledge is pretty much useless in helping other's debloating their access link. I need existing field tested and "batle-hardened" solutions now. If at the begining of my involvement I had waited for L4S, I would still be waiting since like 7? years...

> So either compare FQ_L4S with FQ_Codel or compare DualPI2 with Single Queue CoDel.

	[SM] That is not how I think this comparison should be performed. The challenge is how to solve the latency under load problem on typical internet access links today, and the contenders are the existing testable deployable solutions, so e.g. SQM (a fancy name for traffic shaper plus competent modern AQM, mostly fq_codel, or as all in one solution cake) or DualPI2. 
	Just because you opted for a single queue solution, does not mean I and my users follow that insane stance. It turns out that the traffic shaper is the really expensive part, the fq components is comparatively cheap, so with L4S the CPU cycles gained from having a worse but computationally cheaper AQM will disappear in the noise as the shaper is the truly problematic part. So a lot of the rationale for a "single" queue solution (side-note: last time I checked two was a number larger than one) over an FQ solution goes away, and I do not subscribe to Bob's dogmatic rejection of FQ (side-note: I am more and more convinced that Bob does not actually understand FQ and what it brings to the table, but I digress).
	Yes for a core router or a router at an exchange point between two AS that equation will be different (there will be no shaper necessary, and anything not handled by the ASICs will probably be glacial in comparison). 
	BUT I do not believe for one minute, that L4S will end up anywhere close to core or transit routers, from its (intention or accidental) design it is a short RTT dragster race strip, and there typically are few core devices in the path, and the most relevant will be the ISP's access link AQMs (and maybe stuff on the data center side of the race track). 
	And on the end-user side I can tell you from direct experience, doing the FQ part is not a big issue (doing the shaping part is, at least if te shaper is supposed to not introduce unduly delay/jitter). 
	Any ISP is welcome today to copy what SQM does and move both the egress and ingress shaper to the CPE, where the computational cost can be easily distributed (each CPE only deal with its user's/owner's network traffic, much easier to get X devices that can shape @1Gbps, the on device doing that @XGbps, with X >> 1), without loss of (much) functionality. If the ISP is extra nice, they will put an extra AQM on the ingress side (and by all means as secondary back-stop AQM, use PIE if you must), but no need to restrict the solution by making accommodations for classes of machines that will never run L4S anyway, or to one designers decade long misinterpretation of a decent technical solution for a cult. And once you accept/realize that the natural position for such an latency reduction AQM is at the access link, the whole "is it safe with tunnels" argument for high-jacking CE goes out of the door, most endusers will rarely use tunnels, or use the proper network pluming to instantiate the shaper such that it will see the decapsulated traffic. This will not be perfect and cover 100% of tunnel cases, sure, but it will be in all likelihood good enough.

Okay, that was longer than intended, but I believe it was necessary to explain why I object to most of the L4S state and process. Too little (improvement over what interested users already do today), too late (SQM is in the field with real users since more than 5 years) at too high a cost (breaking backward compatiblity unnecessarily)

Regards
	Sebastian

---

From: Lars Eggert <[email-masked]>
Date: Thu, 19 Nov 2020 07:08:18 +0200

Hi,

just wanted to say that Steven's email below raises many of the points I tried to make - and expresses them better - so +1.

Thanks,
Lars

On 2020-11-18, at 19:50, Steven Blake <[email-masked]> wrote:
> 
> On Wed, 2020-11-18 at 10:31 +0000, De Schepper, Koen (Nokia - BE/Antwerp) wrote:
>> Hi all,
>> 
> These are arguments to deploy a scheme that has been demonstrated to work. Not arguments to conduct an *experiment* to determine whether a scheme is capable of working (and under what conditions?).
> 
> Do you completely ignore the simulation results that show that L4S currently does not live up to it's own hype? How are you going to conduct the *experiment* without at least one CC that seems to work in simulation?
> 
> Why? What is preventing an interested network operator from conducting L4S *experiments* now, using an experimental DSCP to signal L4S packets? What is preventing two interested directly peering operators? What is preventing two interested but non-peering operators from running tunnels between themselves?
> 
> 
> As I have said before, the only thing blocking the L4S *experiement* from getting IETF blessing IMHO is the wholly unnecessary insistence on consuming an IP header bit *now* with no plan to relinquish it if the *experiment* fails. Every objection raised to my statement has been an argument predicated on ease of deployment. Ease of deployment as a standard is something to be considered, yes. The allocation of an IP header bit can be considered *after* evaluating the results of the *experiment*, especially when there are practical work-arounds (e.g., experimental DSCP) to enable the *experiment* to proceed.
> 
> I see the same style of argument from particle physicists who want to spend $40B to build a 150-km circumference particle collider, to find what exactly? (Especially considering that all current beyond-Standard Model theories have totally bombed).
> 
> 
> Regards,
> 
> // Steve

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 19 Nov 2020 09:43:37 +0000

Lars, Steven

My opinion is that this work has stalled due to an endless discussion on the severity of the listed issues. Do you foresee that a year or so more of equally endless discussion will make anybody more wise ?, between meetings there are only a few people engaged in this debate that floods the TSVWG list, it is a safe bet that most outsiders hit the delete button.
 
I can understand the incentives to try and delay WGLC/RFC, perhaps some hopes that there will by some magic emerge a much better high-fidelity congestion marking in some form. In short, tear the whole thing apart and come up with something much better. 
The problem is that we are talking at least 2 years extra work to get into the state that the L4S drafts are today. 
The use of the ECT(1) code point tend to often lead to questions like "what do we do when L4S fails?". Well the nonce has been declared historic so the process for this is in place, but besides this it is perhaps good to consider that L4S can actually fly ?. 

I really think that it is high time to move this to WGLC, this has dragged on for too long. 

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Lars Eggert
> Sent: den 19 november 2020 06:08
> To: Steven Blake <[email-masked]>
> just wanted to say that Steven's email below raises many of the points I tried
> to make - and expresses them better - so +1.
> 
> Thanks,
> Lars
> 
> >> 	• The big L4S benefits were a good reason to justify the extra network
> effort to finally implement ECN in general and AQMs in network equipment
> >> 	• Deployment of L4S support will include deployment of Classic ECN
> too! So even for the skeptics among us, that consider that the experiment can
> fail due to CCs not performing to expectations, we will fall back to having
> Classic ECN support
> >> 	• Only dependency to CCs in the drafts are the mandatory Prague
> requirements (only required input/review from future CC developers: are
> they feasible for you)
> > Do you completely ignore the simulation results that show that L4S currently
> does not live up to it's own hype? How are you going to conduct the
> *experiment* without at least one CC that seems to work in simulation?
> >
> L4S *experiments* now, using an experimental DSCP to signal L4S packets?
> What is preventing two interested directly peering operators? What is
> preventing two interested but non-peering operators from running tunnels
> between themselves?
> >
> >
> >> 	• It will create the required traction within the CC community to come
> up with improvements (if needed at all for the applications that would benefit
> from it; applications that don’t benefit from it yet, can/will not use it)
> >> 	• NW operators have benefits now (classic ECN and good AQMs) and
> in the future can offer their customers better Low Latency experience for the
> popular interactive applications
> >> 	• No signs that we can expect more traction in CC development; trust
> and expectations of continuous delays will not attract people working on it, as
> there will be plenty of time before deployments are materializing
> >> 	• Product development of Classic ECN will stall and die due to
> uncertainty on how L4S will finally materialize
> >>
> > As I have said before, the only thing blocking the L4S *experiement* from
> getting IETF blessing IMHO is the wholly unnecessary insistence on consuming
> an IP header bit *now* with no plan to relinquish it if the *experiment* fails.
> Every objection raised to my statement has been an argument predicated on
> ease of deployment. Ease of deployment as a standard is something to be
> considered, yes. The allocation of an IP header bit can be considered *after*
> evaluating the results of the *experiment*, especially when there are
> practical work-arounds (e.g., experimental DSCP) to enable the *experiment*
> to proceed.
> >
> > I see the same style of argument from particle physicists who want to spend
> $40B to build a 150-km circumference particle collider, to find what exactly?
> (Especially considering that all current beyond-Standard Model theories have
> totally bombed).
> >
> >
> > Regards,
> >
> > // Steve

---

From: Lars Eggert <[email-masked]>
Date: Thu, 19 Nov 2020 12:13:29 +0200

Hi,

On 2020-11-19, at 11:43, Ingemar Johansson S <[email-masked]> wrote:
> My opinion is that this work has stalled due to an endless discussion on the severity of the listed issues. Do you foresee that a year or so more of equally endless discussion will make anybody more wise ?

I'd hope so. But maybe I am optimistic.

One problem is that there have been two camps that are mostly talking to (or shouting at) each other. That makes it quite unappealing to join the discussion for someone who isn't part of those camps. I agree that more of that kind of "discussion" is not going to be helpful. I think the two camps need to realize that they need to convince the *rest of the WG* of their respective views, not each other.

> , between meetings there are only a few people engaged in this debate that floods the TSVWG list, it is a safe bet that most outsiders hit the delete button.

Yes. Because mostly, as I wrote above, the discussion is very inwardly focused, highly detailed, very fast-paced, references email threads that go back years (without pointers) and so makes it very, very hard for someone who has not participated to engage and stay engaged.

> I can understand the incentives to try and delay WGLC/RFC, perhaps some hopes that there will by some magic emerge a much better high-fidelity congestion marking in some form. In short, tear the wh…

That's certainly not what I tried to say, and I didn't get that from Steven's email either.

My point (don't want to speak for Steven) was that without a TCP scheme that delivers some benefits when operating over a path with some new-style queuing while at the same time not suffering from serious regressions, it's kinda hard to argue that that new-style queueing is performing as desired. (And we can certainly debate what counts as a serious regression and what doesn't.) Basically, a strawman that satisfies (some of) the Prague requirements without regressions.

Maybe we have that TCP scheme already, but at least the recent results that Pete shared show that there are maybe still some question marks.

And we don't need the perfect scheme. But we need something that delivers enough benefit that the entities that are required to deploy this feel motivated to do so. Deploying this is a pretty big investment, after all.

> The problem is that we are talking at least 2 years extra work to get into the state that the L4S drafts are today.

I can't quite parse that sentence. Are you saying this would delay L4S for two years before we can LC it?

Like Steven, I don't understand where this extreme pressure comes from to LC something. Experimentation is not contingent on the LC, even large-scale experimentation.

> The use of the ECT(1) code point tend to often lead to questions like "what do we do when L4S fails?". Well the nonce has been declared historic so the process for this is in place, but besides thi…

I don't actually care at all that L4S uses ECT(1). I'm still unconvinced - at the moment - that L4S delivers on its promises. *That* is why I am questioning whether we're ready for a LC.

> I really think that it is high time to move this to WGLC, this has dragged on for too long.

"Time spent in WG" is not a useful criteria for determining whether something is ready for a LC.

Thanks,
Lars

---

From: Roland Bless <[email-masked]>
Date: Thu, 19 Nov 2020 11:35:29 +0100

Hi Lars and all,

I agree with every point that Lars made and want to emphasize that
I would also expect to have a working TCP scheme that fulfills at least
most of the promises that L4S made. It should'nt be necessary for others
to find the flaws within the current approach, I'd rather expect that
the proponents show what currently works and even what doesn't work so well.

I think it's then up to the WG to decide whether the remaining issues
are acceptable or if further improvements are necessary.

Regards,
 Roland

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 19 Nov 2020 12:15:39 +0100

Des\ar Lars,

thanks for your input, I will focus on one single point below.

> On Nov 19, 2020, at 11:13, Lars Eggert <[email-masked]> wrote:
> 
> Hi,
> 
> [...]

So personally I am at a loss what to think about the L4S RFC-ing process in the WG. Since years there is real testing data showing L4S failing to achieve its promises and goals and yet most of the WG members seem seems to ignore this fact during the meetings. Meetings in which mostly political arguments for RFC-ing the L4S drafts are brought forward like, "I share L4S's goals and hence I endorse its implementation and design" (I am confident not all holding that position have scrutinized the drafts or found time to thoroughly look at the data), or "it took so long already, let's get Brexit^W L4S done" (which is as non-technical as it can get, and not really appropriate for a proposal with demonstrated side-effects on the existing internet traffic). None of these repeated calls to a LC make any dent into the problem that the data shows L4S is not there yet.

	I apologize for not referencing old e-mail threads, but I do try to take care to always include links to data slides and I cite (and link) relevant text from existing RFCs and the L4S drafts. I try to put together the L4S promises (draft text) with data showing where promise and reality seem to diverge. I also try to highlight where I believe the text to be either confusing or outright wrong. What else can I do to engage others (like the impressive number of folks showing interest in progressing L3S in the April interim meeting) into a round of closer scrutiny?

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 19 Nov 2020 12:36:55 +0000

Hi Lars

Please see inline marked [IJ]
/Ingemar

> From: Lars Eggert <[email-masked]>
> Sent: den 19 november 2020 11:13
> To: Ingemar Johansson S <[email-masked]>
> Cc: Steven Blake <[email-masked]>; tsvwg IETF list
<[email-masked]>
> On 2020-11-19, at 11:43, Ingemar Johansson S
> <[email-masked]> wrote:
> > My opinion is that this work has stalled due to an endless discussion on
the
> severity of the listed issues. Do you foresee that a year or so more of
equally
> endless discussion will make anybody more wise ?
> 
> One problem is that there have been two camps that are mostly talking to
(or
> shouting at) each other. That makes it quite unappealing to join the
discussion
> for someone who isn't part of those camps. I agree that more of that kind
of
> "discussion" is not going to be helpful. I think the two camps need to
realize
> that they need to convince the *rest of the WG* of their respective views,
> not each other.
> 
> > , between meetings there are only a few people engaged in this debate
that
> floods the TSVWG list, it is a safe bet that most outsiders hit the delete
button.
> 
> Yes. Because mostly, as I wrote above, the discussion is very inwardly
focused,
> highly detailed, very fast-paced, references email threads that go back
years
> (without pointers) and so makes it very, very hard for someone who has not
> participated to engage and stay engaged.
[IJ] There was an interim (March / April ?) where L4S was described quite a
lot. Then I was happy to see that now we can move forward, unfortunately I
was wrong. As I see if (after following all the threads in this list) the
L4S drafts have move forward but the overall heated discussion on this
mailing list continues like before. The only way is see is that the L4S
drafts are moved to WGLC, then people will hopefully read the drafts and
come with requests for clarifications where needed. Until then you can only
expect more of the same long incomprehensible discussion threads until
March, when we will repeat the same process again.

> 
> > I can understand the incentives to try and delay WGLC/RFC, perhaps some
> hopes that there will by some magic emerge a much better high-fidelity
> congestion marking in some form. In short, tear the whole thing apart and
> come up with something much better.
> 
> That's certainly not what I tried to say, and I didn't get that from
Steven's email
> either.
[IJ] You are right, sorry for this, it was not my intention to put you and
Steve in that camp.

> 
> My point (don't want to speak for Steven) was that without a TCP scheme
that
> delivers some benefits when operating over a path with some new-style
> queuing while at the same time not suffering from serious regressions,
it's
> kinda hard to argue that that new-style queueing is performing as desired.
> (And we can certainly debate what counts as a serious regression and what
> doesn't.) Basically, a strawman that satisfies (some of) the Prague
> requirements without regressions.
> 
> Maybe we have that TCP scheme already, but at least the recent results
that
> Pete shared show that there are maybe still some question marks.
> 
> And we don't need the perfect scheme. But we need something that delivers
> enough benefit that the entities that are required to deploy this feel
> motivated to do so. Deploying this is a pretty big investment, after all.
[IJ] The only pain point I see now is the RTT bias in cases where long RTT
Prague flows compete with short RTT ditto. This is being addressed by the
developers and it is not only an L4S problem. Besides this, Prague will be
presented at ICCRG tomorrow as I understand it. 
Besides this there is discussion around all sorts of cases with RFC3168
style AQMs, additional discussion before a WGLC will definitely not make us
more wise.
As regards to investment, already today there is investment in this,
examples that are disclosed in the open are Broadcom and Nokia. I can
imagine that there is some expectation that L4S will materialize in RFCs 

> 
> > The problem is that we are talking at least 2 years extra work to get
into the
> state that the L4S drafts are today.
> 
> I can't quite parse that sentence. Are you saying this would delay L4S for
two
> years before we can LC it?
[IJ] This was in reference to the " try and delay WGLC/RFC " with the hope
that something better will emerge, it may as well be 5 years and we still
have only 2 bits. 

> 
> Like Steven, I don't understand where this extreme pressure comes from to
> LC something. Experimentation is not contingent on the LC, even
large-scale
> experimentation.
[IJ] If this would only be a IETF matter, then you are right. We however try
to address this also in 3GPP standards to make the whole thing work in
products, and that is of course hard to do if L4S is not even an RFC.

> 
> > The use of the ECT(1) code point tend to often lead to questions like
"what
> do we do when L4S fails?". Well the nonce has been declared historic so
the
> process for this is in place, but besides this it is perhaps good to
consider that
> L4S can actually fly ?.
> 
> I don't actually care at all that L4S uses ECT(1). I'm still unconvinced -
at the
> moment - that L4S delivers on its promises. *That* is why I am questioning
> whether we're ready for a LC.
> 
> > I really think that it is high time to move this to WGLC, this has
dragged on
> for too long.
> 
> "Time spent in WG" is not a useful criteria for determining whether
something
> is ready for a LC.

---

From: Lars Eggert <[email-masked]>
Date: Thu, 19 Nov 2020 15:10:02 +0200

Hi,

On 2020-11-19, at 14:36, Ingemar Johansson S <[email-masked]> wrote:
> The only way is see is that the L4S

to clarify: I don't have opinions about the L4S drafts. I haven't read them in a while, I agree that I should.

One point I am trying to make is that since the set of documents we are discussing seems incomplete, in that it doesn't seem to contain a TCP variant that intends to delivers benefits over L4S paths w/o regressions.

My main point though is that there seem to be questions raised about the performance and behavior of L4S with various TCP variants. This is not an issue with the content of the L4S drafts. It's a remaining uncertainty related to the experimental evaluation and analysis that the L4S mechanisms have seen so far. Going forward with a LC is not going to bring further clarity here.

> presented at ICCRG tomorrow as I understand it.

That is one point. I think interactions with tunnels was another.

I'm actually looking forward to the presentation on TCP Prague - is there a draft?

> style AQMs, additional discussion before a WGLC will definitely not make us

Discussion won't, but experimentation would.

> imagine that there is some expectation that L4S will materialize in RFCs

Sorry I was unclear. You are right that there is investment by vendors. But I think the key question if there will be an investment by operators, since they need to eventually buy L4S kit and deploy it. And that investment will only pay off if end systems actually have deployed a CC scheme that takes advantage of L4S. So the ready availability of such a scheme is IMO a key requirement.

> products, and that is of course hard to do if L4S is not even an RFC.

Is L4S currently a requirement for a future 3GPP release?

Thanks,
Lars

---

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 19 Nov 2020 14:24:18 +0000

Hi

Please see inline [IJ]

/Ingemar

> Sent: den 19 november 2020 14:10
<[email-masked]>
> On 2020-11-19, at 14:36, Ingemar Johansson S
> > drafts are moved to WGLC, then people will hopefully read the drafts
> > and come with requests for clarifications where needed. Until then you
> > can only expect more of the same long incomprehensible discussion
> > threads until March, when we will repeat the same process again.
> 
> to clarify: I don't have opinions about the L4S drafts. I haven't read
them in a
> while, I agree that I should.
> 
> One point I am trying to make is that since the set of documents we are
> discussing seems incomplete, in that it doesn't seem to contain a TCP
variant
> that intends to delivers benefits over L4S paths w/o regressions.
> 
> My main point though is that there seem to be questions raised about the
> performance and behavior of L4S with various TCP variants. This is not an
> issue with the content of the L4S drafts. It's a remaining uncertainty
related to
> the experimental evaluation and analysis that the L4S mechanisms have seen
> so far. Going forward with a LC is not going to bring further clarity
here.
> 
> > [IJ] The only pain point I see now is the RTT bias in cases where long
> > RTT Prague flows compete with short RTT ditto. This is being addressed
> > by the developers and it is not only an L4S problem. Besides this,
> > Prague will be presented at ICCRG tomorrow as I understand it.
> 
[IJ] My take is that this is something for L4S ops that I see as a product
of the L4S experiment. It is not something that needs to be answered on its
own as it is mainly the RFC3168 AQM issue in another shape, we don't know
how widely spread this problem is or to how large extent this equipment can
be updated, earlier discussions on fq-codel implementations in home gateways
was not conclusive I guess but it appears that many of these have automatic
upgrade features, I guess the reason here is to be able to upgrade to combat
security threats but this is admittedly speculation.

> 
> I'm actually looking forward to the presentation on TCP Prague - is there
a
> draft?
[IJ] Not that I know of, code is found at https://github.com/L4STeam/  
> 
> > Besides this there is discussion around all sorts of cases with
> > RFC3168 style AQMs, additional discussion before a WGLC will
> > definitely not make us more wise.
> 
[IJ] Yes, but we need to get past the discussion on all possible things that
can happen. Recall from the discussion yesterday that Stuart Cheshire and
his team has not need evidence of ECN marking AQMs. So I fear that we spend
a lot of time discussing problems that are very rare.

> Sorry I was unclear. You are right that there is investment by vendors.
But I
> think the key question if there will be an investment by operators, since
they
> need to eventually buy L4S kit and deploy it. And that investment will
only pay
> off if end systems actually have deployed a CC scheme that takes advantage
of
> L4S. So the ready availability of such a scheme is IMO a key requirement.
[IJ] I would say that we already have CC algos that can be used. Surely they
do not meet all the requirements today but I don't see why it will not be
possible. It is definitely the case that we will be discussing what typical
RTTs are like forever but that is not something that should delay the L4S
drafts I think.  

> 
> > [IJ] If this would only be a IETF matter, then you are right. We
> > however try to address this also in 3GPP standards to make the whole
> > thing work in products, and that is of course hard to do if L4S is not
even an
> RFC.
[IJ] L4S is not currently pushed for in 3GPP, there is however work on the
support for extended reality that requires low latency. L4S will definitely
fit there. I would expect that L4S support can be proposed ~ mid next year.

---

From: "Scharf, Michael" <[email-masked]>
Date: Thu, 19 Nov 2020 14:52:12 +0000

I won't comment on the question discussed here. This is just a small reminder not to forget pieces outside TSVWG...

> discussing seems incomplete, in that it doesn't seem to contain a TCP variant
> issue with the content of the L4S drafts. It's a remaining uncertainty related
> to the experimental evaluation and analysis that the L4S mechanisms have
> seen so far. Going forward with a LC is not going to bring further clarity here.

As per draft-ietf-tsvwg-l4s-arch-08, "[...] the implementation of TCP receivers will have to be upgraded [RFC7560].  Work to standardize and implement more accurate ECN feedback for TCP (AccECN) is in progress [I-D.ietf-tcpm-accurate-ecn]".

draft-ietf-tcpm-accurate-ecn is a WG item in TCPM targeting standards track. In order to finish draft-ietf-tcpm-accurate-ecn, TCPM needs reviews and feedback. For instance, recently the encoding of the TCP option changed. IMHO would be important to understand whether the solution now addresses all known needs.

Has everybody who plans to run or runs TCP-based experiments with L4S already reviewed draft-ietf-tcpm-accurate-ecn? If not, why not?

Please feel free to post comments regarding draft-ietf-tcpm-accurate-ecn on [email-masked].

Michael

---

From: Pete Heist <[email-masked]>
Date: Thu, 19 Nov 2020 17:22:16 +0100

Hi Koen,

Rather than thinking of this as advantages and disadvantages to
waiting, I see it as an engineering process. It was decided earlier
this year that the L4S proposal has enough support to continue, so
we're on that path now. Part of that decision, as I understood it, also
recognized that there are valid safety concerns around compatibility
with existing AQMs, and some solution needs to be devised.

RFC3168 bottleneck detection was added to TCP Prague, which appears to
be difficult to do reliably when there is jitter or cross-flow traffic,
and it has since been disabled in the reference implementation. The
l4s-ops draft was started, but isn't complete yet and may need WG
adoption as part of a LC. We can then decide how effective the proposed
mitigations are against the risks and prevalence.

To start a WGLC now would circumvent that earlier recognition that a
safety case needs to be made. Meanwhile, since testing showed that
tunnels through RFC3168 FQ AQMs are a straightforward path to unsafe
flow interaction, along with other issues relative to the goals, it
doesn't seem like the engineering process is done just yet.

Regards,
Pete

BE/Antwerp) wrote:
> * The big L4S benefits were a good reason to justify the extra
> network
> * Timing is optimal now: implementations in NW equipment are coming
> and
> deployment can start now
> * Deployment of L4S support will include deployment of Classic ECN
> too!
> So even for the skeptics among us, that consider that the experiment
> can fail due to CCs not performing to expectations, we will fall back
> to having Classic ECN support
> * Only dependency to CCs in the drafts are the mandatory Prague
> are they feasible for you)
> * We have a good baseline for a CC (upstreaming to Linux is blocked
> by
> the non-RFC status)
> * It will create the required traction within the CC community to
> come
> would benefit from it; applications that don’t benefit from it yet,
> can/will not use it)
> * NW operators have benefits now (classic ECN and good AQMs) and in
> the
> future can offer their customers better Low Latency experience for
> the popular interactive applications
> * We’ll gets stuck in an analysis paralysis (aren’t we already?)
> * No signs that we can expect more traction in CC development; trust
> and expectations of continuous delays will not attract people working
> on it, as there will be plenty of time before deployments are
> materializing
> * Product development of Classic ECN will stall and die due to

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 19 Nov 2020 16:34:58 +0000

On 19/11/2020 16:22, Pete Heist wrote:
> we're on that path now. Part of that decision, as I understood it, 
> also recognized that there are valid safety concerns around 
> compatibility with existing AQMs, and some solution needs to be devised.
>
> be difficult to do reliably when there is jitter or cross-flow 
> traffic, and it has since been disabled in the reference 
> implementation. The l4s-ops draft was started, but isn't complete yet 
> and may need WG adoption as part of a LC. We can then decide how 
> effective the proposed mitigations are against the risks and prevalence.
>
By the way, I liked your data - and it helped me a lot to look at this, 
thanks very much for doing this.

It would help me if you clarify what you mean by  "unsafe" - to me 
"safety" relates to traffic unresponsive to drop, as in CBR traffic, 
etc. I've not understood how CE-marked traffic can erode safety, but 
maybe I missed something?

I'm not sure why "tunnels have crept in here. There have always been 
side-effects with classification (and hence scheduling), but I don't see 
new issues relating to "tunnels" with ECN.

I'm not commenting on when the Chairs think a WGLC will provide useful 
information, we'll say that in due course.

Best wishes,

Gorry

>>     network effort to finally implement ECN in general and AQMs in
>>     network equipment
>>     and deployment can start now
>>     experiment can fail due to CCs not performing to expectations, we
>>     will fall back to having Classic ECN support
>>     requirements (only required input/review from future CC
>>     developers: are they feasible for you)
>>     by the non-RFC status)
>>     come up with improvements (if needed at all for the applications
>>     that would benefit from it; applications that don’t benefit from
>>     it yet, can/will not use it)
>>     the future can offer their customers better Low Latency
>>     experience for the popular interactive applications
>>   * No signs that we can expect more traction in CC development;
>>     trust and expectations of continuous delays will not attract
>>     people working on it, as there will be plenty of time before
>>     deployments are materializing

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 19 Nov 2020 18:46:15 +0200

> On 19 Nov, 2020, at 6:34 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
> It would help me if you clarify what you mean by  "unsafe" - to me "safety" relates to traffic unresponsive to drop, as in CBR traffic, etc. I've not understood how CE-marked traffic can erode safety, but maybe I missed something?

There is the potential for traffic between existing, RFC-compliant endpoints over an existing, RFC-compliant network to suffer a Denial of Service due to the presence of L4S traffic on that same network.  In some runs, we see CUBIC and Reno forced down to the minimum cwnd of 2 segments, and would be forced down further if that were possible.  That is the safety concern.

There is separate concern that the reference implementation of L4S doesn't meet its own specifications.  That's less of a safety concern, but we think L4S proponents should be more concerned about it than appears to be the case.  These results can (and should) be confirmed in the lab before attempting any kind of deployment.

> I'm not sure why "tunnels have crept in here. There have always been side-effects with classification (and hence scheduling), but I don't see new issues relating to "tunnels" with ECN.

An L4S middlebox can cope with tunnels containing a mixture of conventional and L4S traffic.  An RFC-compliant middlebox cannot, even if it implements FQ, which is otherwise held up as being "sufficient" to isolate L4S traffic from conventional traffic.  It is the fact that tunnels can defeat FQ that is causing renewed concern here.

The l4s-ops draft appears to say some things about this which should be a relevant point of discussion.  However, L4S proponents appear to be keen to proceed with L4S WGLC without waiting for l4s-ops.  That is a problem.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 19 Nov 2020 20:07:33 +0200

> On 19 Nov, 2020, at 4:24 pm, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Recall from the discussion yesterday that Stuart Cheshire and

Okay, I'm going to interject on this.  I don't know precisely how Apple is collecting their data, but you can see the data collected by a MacOS X machine by entering the following in Terminal:

	sudo netstat -sp tcp

About two-thirds of the way down the output, you should find a group of statistics related to ECN.  Here's mine:

	917 client connections attempted to negotiate ECN
		789 client connections successfully negotiated ECN
		97 times graceful fallback to Non-ECN connection
		1 time lost ECN negotiating SYN, followed by retransmission
		0 server connection attempted to negotiate ECN
		0 server connection successfully negotiated ECN
		0 time lost ECN negotiating SYN-ACK, followed by retransmission
		162 times received congestion experienced (CE) notification
		0 time CWR was sent in response to ECE
		1986 times sent ECE notification
		32 connections received CE atleast once
		0 connection received ECE atleast once
		298 connections using ECN have seen packet loss but no CE
		17 connections using ECN have seen packet loss and CE
		15 connections using ECN received CE but no packet loss
		0 connection fell back to non-ECN due to SYN-loss
		14 connections fell back to non-ECN due to reordering
		0 connection fell back to non-ECN due to excessive CE-markings

Of the connections where ECN negotiation was attempted, most (almost 90%) succeeded, and of these some did show ECN activity in practice - noting that many connections probably did not reach the shaper limit set by my ECN-enabled router, or reached the capacity of my rural LTE connection at a time when it was less than the shaper limit.

But to put this into context, further up is:

	117038 connection requests

This is a machine with ECN support forced on by appropriate entries in /etc/sysctl.conf.  Even so, less than 1% of TCP connections initiated even *attempted* to negotiate ECN.  This led me to discover that MacOS X had unexpectedly and erroneously designated my network as an ECN blackhole, a condition that was temporarily resolved by turning wifi off and on again, but soon recurred for unknown reasons.  (This effect does not occur with my Linux-based machines.)

If this erroneous ECN blackholing phenomenon is common in Apple devices - not only desktops and laptops but also iPhones and iPads - then that would certainly explain why Stuart Cheshire sees so little ECN activity in the data available to him.  I would encourage him to double-check that with appropriate technical employees at Apple.

I note also that Jake Holland's data from Akamai depends on client endpoints initiating ECN negotiation, which is by no means universal thus far, and may therefore understate the deployment of RFC-3168 compliant AQMs (which exert congestion signalling via packet drops for Not-ECT traffic) by a large factor, especially in the small number of ASNs that already show significant ECN activity in that data.

In short, I believe there's a lot more ECN deployment in practice than most people realise.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Thu, 19 Nov 2020 13:40:53 -0500

On Thu, 2020-11-19 at 09:43 +0000, Ingemar Johansson S wrote:
> Lars, Steven
> 
> My opinion is that this work has stalled due to an endless discussion
> on the severity of the listed issues. Do you foresee that a year or
> so more of equally endless discussion will make anybody more wise ?,
> between meetings there are only a few people engaged in this debate
> that floods the TSVWG list, it is a safe bet that most outsiders hit
> the delete button.
>  
> I can understand the incentives to try and delay WGLC/RFC, perhaps
> some hopes that there will by some magic emerge a much better high-
> fidelity congestion marking in some form. In short, tear the whole
> thing apart and come up with something much better. 
> into the state that the L4S drafts are today. 
> "what do we do when L4S fails?". Well the nonce has been declared
> historic so the process for this is in place, but besides this it is
> perhaps good to consider that L4S can actually fly ?. 
> 
> dragged on for too long. 

The objective should not be to produce documents. The objective should
be to produce *functioning technology*.

Are all of the component pieces of L4S functioning technology that can
be beneficially deployed in the Internet? Who knows? The simulation
results are inconclusive, at best.

What is the bare minimum that IETF needs to agree on to enable
interested parties to safely conduct and coordinate L4S *experiments*
in real networks? Focus on an answer to that instead of repeating sunk-
cost fallacies.

Regards,

// Steve

---

From: Pete Heist <[email-masked]>
Date: Thu, 19 Nov 2020 21:02:39 +0100

On Thu, 2020-11-19 at 16:34 +0000, Gorry Fairhurst wrote:
> > RFC3168 bottleneck detection was added to TCP Prague, which appears
> > to be difficult to do reliably when there is jitter or cross-flow
> > implementation. The l4s-ops draft was started, but isn't complete
> > yet and may need WG adoption as part of a LC. We can then decide
> > how effective the proposed mitigations are against the risks and
> > prevalence.
> > 
> > 
> > 
> > To start a WGLC now would circumvent that earlier recognition that
> > a safety case needs to be made. Meanwhile, since testing showed
> > that tunnels through RFC3168 FQ AQMs are a straightforward path to
> > unsafe flow interaction, along with other issues relative to the
> > goals, it doesn't seem like the engineering process is done just
> > yet.
> > 
> > 
> > 
> 
> By the way, I liked your data - and it helped me a lot to look at
> this, thanks very much for doing this.
I'm glad, as I think we're at our best when we're doing engineering and
producing data. I wish it were easier to do!
> maybe I missed something?
Sure, so the existing RFC3168 CE signal in use on the Internet today
indicates an MD (multiplicative decrease), whereas the redefined CE
signal in L4S indicates an AD (additive decrease). Two congestion
controls responding to CE in a different way, or one that responds to
CE with an AD and one that responds only to drop (i.e. all standard
congestion controls that advertise Not-ECT), will not interact safely
in the same RFC3168 signaling queue. We're probably on the same page
here already, but I'll refer to section 5 of RFC8257.

That is one of the reasons why ECT(1) is used in L4S to place L4S flows
in the L queue- to keep them separate from conventional flows in the C
queue. As long as flows have advertised their capability correctly,
that works.

However, existing RFC3168 queues do not have knowledge of L4S,
therefore will not know that ECT(1) means that traffic needs to be
segregated and signaled in a different way. They will signal a Prague
flow, which sets ECT(1), with CE, expecting the flow to respond with an
MD, rather than AD. Meanwhile they'll signal an RFC3168 or non-ECN flow
with either CE or drop, and in either case the flow will respond with
an MD, causing conventional flows to yield to Prague flows to varying
degrees depending on the AQM in use.

Here's an example of CUBIC and Prague when they end up in the same
fq_codel queue:
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg

Here's a more extreme example of Reno and Prague sharing a single PIE
queue with ECN enabled (less common):
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-reno-pie-50Mbit-20ms_tcp_delivery_with_rtt.svg

In the example with PIE, Reno appears to be driven at or close to
minimum cwnd. In the fq_codel example, the steady state throughput of
Prague:CUBIC is around 19:1. We've seen a range in the Codel case from
around 12:1 to 20:1. In my opinion, we could use the word "unsafe" here
in both cases.
> side-effects with classification (and hence scheduling), but I don't
> see new issues relating to "tunnels" with ECN.
Tunnels are relevant because they provide an easy practical path to the
unsafe flow interaction described above. The widely used fq_codel qdisc
has ECN enabled by default. Fortunately, because it has flow-fair
queueing, Prague flows and conventional flows are usually placed in a
separate queue (hash collisions aside), causing Prague to only affect
itself with additional delay (TCP RTT). However, a tunnel's
encapsulated packets all share the same fq_codel queue because they all
have the same 5-tuple, so there is unsafe interaction between the
tunnel's flows. Here we use Wireguard through fq_codel:

http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s5-tunnel/l4s-s5-tunnel-phys-wireguard-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg

I'll leave it to the WG to come up with examples of what types of
tunnels and traffic scenarios could lead to this, but one example is a
user who has a privacy VPN on their PC, and fq_codel on their home
gateway. Let's say one flow connects to an L4S capable server, and
another flow to a non-L4S, conventional server. The L4S flow will
dominate the non-L4S one (whether it's ECN capable or not), probably
causing some level of poor service, perhaps for a video stream,
download, or whatever.
> I'm not commenting on when the Chairs think a WGLC will provide
> useful information, we'll say that in due course.
Ok, I trust that we'll engage enough disinterested people into
congestion control who will add their input.

Thanks Gorry for looking this over. :)
> > > To continue on the discussions in the meeting, a recap and some
> > > extra thoughts. Did I miss some arguments?
> > >  
> > > * When more L4S CCs are developed, the real independent
> > > evaluation
> > > of those can start
> > >  
> > > * Product development of L4S will stall and die due to
> > > uncertainty
> > > on if L4S will finally materialize

---

From: Greg White <[email-masked]>
Date: Fri, 20 Nov 2020 00:42:06 +0000

Hi Pete,

Thanks for summarizing your concerns.

As others have pointed out, it is believed that the vast majority of RFC3168 implementations are fq_codel/CAKE instances, so this is an important situation to understand.

I think that it has also been pointed out that these implementations are all software upgradable, and many of them are run on systems that automatically (or readily at least) receive updates that address security vulnerabilities, fix bugs, make performance or feature enhancements, etc.

Is it possible that in these software implementations, the LSB of the ECN field can be added to the FQ flow identifier, thus allowing the FQ to segregate the L4S traffic in the tunnel from the Classic traffic?  If so, would that solve this problem?  Is that an experiment that you’d be willing to try?

It would be good also to test this with the low CE threshold enabled for queues that contain ECN==*1 as well.  These seem like two really simple changes that could propagate into the fq_codel/CAKE deployed base along the timelines that it will take for L4S experimentation to ramp up.

-Greg

From: tsvwg <[email-masked]> on behalf of Pete Heist <[email-masked]>
Date: Thursday, November 19, 2020 at 1:03 PM
To: Gorry Fairhurst <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] Reasons for WGLC/RFC asap

Hi Koen,

Rather than thinking of this as advantages and disadvantages to waiting, I see it as an engineering process. It was decided earlier this year that the L4S proposal has enough support to continue, so we're on that path now. Part of that decision, as I understood it, also recognized that there are valid safety concerns around compatibility with existing AQMs, and some solution needs to be devised.

RFC3168 bottleneck detection was added to TCP Prague, which appears to be difficult to do reliably when there is jitter or cross-flow traffic, and it has since been disabled in the reference implementation. The l4s-ops draft was started, but isn't complete yet and may need WG adoption as part of a LC. We can then decide how effective the proposed mitigations are against the risks and prevalence.

To start a WGLC now would circumvent that earlier recognition that a safety case needs to be made. Meanwhile, since testing showed that tunnels through RFC3168 FQ AQMs are a straightforward path to unsafe flow interaction, along with other issues relative to the goals, it doesn't seem like the engineering process is done just yet.

By the way, I liked your data - and it helped me a lot to look at this, thanks very much for doing this.
I'm glad, as I think we're at our best when we're doing engineering and producing data. I wish it were easier to do!

It would help me if you clarify what you mean by  "unsafe" - to me "safety" relates to traffic unresponsive to drop, as in CBR traffic, etc. I've not understood how CE-marked traffic can erode safety, but maybe I missed something?
Sure, so the existing RFC3168 CE signal in use on the Internet today indicates an MD (multiplicative decrease), whereas the redefined CE signal in L4S indicates an AD (additive decrease). Two congestion controls responding to CE in a different way, or one that responds to CE with an AD and one that responds only to drop (i.e. all standard congestion controls that advertise Not-ECT), will not interact safely in the same RFC3168 signaling queue. We're probably on the same page here already, but I'll refer to section 5 of RFC8257.

That is one of the reasons why ECT(1) is used in L4S to place L4S flows in the L queue- to keep them separate from conventional flows in the C queue. As long as flows have advertised their capability correctly, that works.

However, existing RFC3168 queues do not have knowledge of L4S, therefore will not know that ECT(1) means that traffic needs to be segregated and signaled in a different way. They will signal a Prague flow, which sets ECT(1), with CE, expecting the flow to respond with an MD, rather than AD. Meanwhile they'll signal an RFC3168 or non-ECN flow with either CE or drop, and in either case the flow will respond with an MD, causing conventional flows to yield to Prague flows to varying degrees depending on the AQM in use.

Here's an example of CUBIC and Prague when they end up in the same fq_codel queue:
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg

Here's a more extreme example of Reno and Prague sharing a single PIE queue with ECN enabled (less common):
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-reno-pie-50Mbit-20ms_tcp_delivery_with_rtt.svg

In the example with PIE, Reno appears to be driven at or close to minimum cwnd. In the fq_codel example, the steady state throughput of Prague:CUBIC is around 19:1. We've seen a range in the Codel case from around 12:1 to 20:1. In my opinion, we could use the word "unsafe" here in both cases.

I'm not sure why "tunnels have crept in here. There have always been side-effects with classification (and hence scheduling), but I don't see new issues relating to "tunnels" with ECN.
Tunnels are relevant because they provide an easy practical path to the unsafe flow interaction described above. The widely used fq_codel qdisc has ECN enabled by default. Fortunately, because it has flow-fair queueing, Prague flows and conventional flows are usually placed in a separate queue (hash collisions aside), causing Prague to only affect itself with additional delay (TCP RTT). However, a tunnel's encapsulated packets all share the same fq_codel queue because they all have the same 5-tuple, so there is unsafe interaction between the tunnel's flows. Here we use Wireguard through fq_codel:

I'll leave it to the WG to come up with examples of what types of tunnels and traffic scenarios could lead to this, but one example is a user who has a privacy VPN on their PC, and fq_codel on their home gateway. Let's say one flow connects to an L4S capable server, and another flow to a non-L4S, conventional server. The L4S flow will dominate the non-L4S one (whether it's ECN capable or not), probably causing some level of poor service, perhaps for a video stream, download, or whatever.

I'm not commenting on when the Chairs think a WGLC will provide useful information, we'll say that in due course.
Ok, I trust that we'll engage enough disinterested people into congestion control who will add their input.

Best wishes,

Gorry

Regards,
Pete

Hi all,

·         There is NOW a big need for solutions that can support Low Latency for new Interactive applications

·         The big L4S benefits were a good reason to justify the extra network effort to finally implement ECN in general and AQMs in network equipment

·         Timing is optimal now: implementations in NW equipment are coming and deployment can start now

·         Deployment of L4S support will include deployment of Classic ECN too! So even for the skeptics among us, that consider that the experiment can fail due to CCs not performing to expectations, we will fall back to having Classic ECN support

·         Current drafts are about the network part, and are ready and stable for a very long time now.

·         Only dependency to CCs in the drafts are the mandatory Prague requirements (only required input/review from future CC developers: are they feasible for you)

·         We have a good baseline for a CC (upstreaming to Linux is blocked by the non-RFC status)

·         Larger scale (outside the lab) experiments are blocked by non-RFCs status

·         It will create the required traction within the CC community to come up with improvements (if needed at all for the applications that would benefit from it; applications that don’t benefit from it yet, can/will not use it)

·         NW operators have benefits now (classic ECN and good AQMs) and in the future can offer their customers better Low Latency experience for the popular interactive applications

·         When more L4S CCs are developed, the real independent evaluation of those can start

·         We’ll gets stuck in an analysis paralysis (aren’t we already?)

·         Trust in L4S will vanish

·         No signs that we can expect more traction in CC development; trust and expectations of continuous delays will not attract people working on it, as there will be plenty of time before deployments are materializing

·         Product development of L4S will stall and die due to uncertainty on if L4S will finally materialize

·         Product development of Classic ECN will stall and die due to uncertainty on how L4S will finally materialize

Regards,
Koen.

-->

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 20 Nov 2020 03:38:36 +0200

> On 20 Nov, 2020, at 2:42 am, Greg White <[email-masked]> wrote:
> 

It's technically possible.  But a problem that immediately comes to mind is that incoming CE-marked packets would then get sorted into a separate queue from ECT(0) packets.  They would thus likely be reordered relative to the rest of the flow, and it increases total occupancy of the flow hash table which increases the risk of hash collisions.  Overall, it just seems a bit inelegant.

And then there's the possibility of tunnels which don't properly handle propagating ECN to the outer header.  I don't know how prevalent these are, but that has the potential to disable this workaround, putting you back in the same position.

Finally, I really think you overestimate the chances of getting software updates out to a lot of these devices.  In many cases, we consider ourselves lucky to get software updates past the first year after hitting the shelves, and we have to manually check and click through the wizard to make the update actually happen - it is *not* automatic for most CPE devices (imagine if the user unwittingly cut the power during one, corrupting the firmware).  So fq_codel as it exists today will be a feature of the Internet for the foreseeable future.

In general, you should consider me skeptical of any proposed solution to L4S' coexistence problems that relies heavily on changing existing features of the internet.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Fri, 20 Nov 2020 07:51:38 +0100

One add below...

On Fri, 2020-11-20 at 03:38 +0200, Jonathan Morton wrote:
> > Is it possible that in these software implementations, the LSB of
> > the ECN field can be added to the FQ flow identifier, thus allowing
> > the FQ to segregate the L4S traffic in the tunnel from the Classic
> > traffic?  If so, would that solve this problem?  Is that an
> > experiment that you’d be willing to try?  
> 
> It's technically possible.  But a problem that immediately comes to
> mind is that incoming CE-marked packets would then get sorted into a
> separate queue from ECT(0) packets.  They would thus likely be
> reordered relative to the rest of the flow, and it increases total
> occupancy of the flow hash table which increases the risk of hash
> collisions.  Overall, it just seems a bit inelegant.
> 
> And then there's the possibility of tunnels which don't properly
> handle propagating ECN to the outer header.  I don't know how
> prevalent these are, but that has the potential to disable this
> workaround, putting you back in the same position.
> 
> Finally, I really think you overestimate the chances of getting
> software updates out to a lot of these devices.  In many cases, we
> consider ourselves lucky to get software updates past the first year
> after hitting the shelves, and we have to manually check and click
> through the wizard to make the update actually happen - it is *not*
> automatic for most CPE devices (imagine if the user unwittingly cut
> the power during one, corrupting the firmware).  So fq_codel as it
> exists today will be a feature of the Internet for the foreseeable
> future.

I have some experience with Ubiquiti's EdgeRouter products, and its
Smart Queueing feature with fq_codel. Their mainline code is still on a
3.10 series kernel, apparently because the offloads support for the SoC
platform was a licensed kit that's hard to get upgraded. The upgrade
cycles on these products are very slow relative to desktop and server
Linux distributions.

> In general, you should consider me skeptical of any proposed solution
> to L4S' coexistence problems that relies heavily on changing existing
> features of the internet.
> 
>  - Jonathan Morton
>

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 20 Nov 2020 08:52:24 +0100

Hi Ingemar,

a correction below.

> On Nov 19, 2020, at 15:24, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi
> 
> Please see inline [IJ]
> 
> /Ingemar
> 
> upgrade features,

	[SM] This is simply untrue, most home router will only see an update if the user actively monitor's the provider's or manufacturer's website and most manufacturer's will only create and distribute updates around the time a device is still marketed/sold. There is pretty little home gear out these which gets automatic updates or only notifications of possible updates. I agree that that would be a great situation to be in, but realistically the home router update and security situation is only mildly above the update and security situation of IoT devices. Remember it is the "s" in IoT that stands for security/safety...
	So, if your stance is, automatic updates are robust, reliable and wide-spread, please post some references supporting that hypothesis.

> I guess the reason here is to be able to upgrade to combat

	[SM] For someone putting their operational safety in that basket I would have expected more robust and reliable numbers for how many devices might be amenable to automated updates. Like hard numbers supporting the hypothesis that automatic updates are wide spread enough to allow distribution of such functionality updates. Also think about who is supposed to make back-ports of the required Linux kernel changes to the often ancient kernel versions SoC-SDKs are based on? To be honest that level of engineering by wishful thinking frightens me a bit.

> [IJ] Not that I know of, code is found at https://github.com/L4STeam/  

	[SM] Is it just me that sees a problem in that the reference protocol implementation just exists in a repository, without even an attempt of writing an internet dratf describing that protocol? There are zero guarantees the tomorrows TCP Prague still is L4S compliant (today's is not, the rfc3168 detection and fall back are disabled by default, see https://github.com/L4STeam/linux/commit/b256daedc7672b2188f19e8b6f71ef5db7afc720).

> a lot of time discussing problems that are very rare.

	[SM] Ingemar, I and other OpenWrt users have an rfc3168 AQM on my internet access link, that by default uses ECN for the downlink. I do not have the view of the world as apple has, but I can guarantee the number is grater than zero. And as stated before people doing this today are exactly the latency conscious users L4S should aim at winning over... 

> possible.

	[SM] This tells more about your confidence in your vision than in what might be achiebale in those CCs. IMHO ithe onus is on team L4S to demonstrate that possibility by demonstrating a CC that ticks all the check marks (does not need to be perfect, but should show significantly better performance in some dimension, while doing at least no harm in others, currently TCP Prague fails on both accounts).

> It is definitely the case that we will be discussing what typical

	[SM] Sorry, this is getting absurd, I point out both Bob completely missing how PIE works (and what it's target variable actually describes ) and an inconsistency between two of the L4S drafts only to be ridiculed here. Ingemar, publishing RFC with inconsistent terms and definition is not a sign of a professional authoring process. Reviews are supposed to help finding such issues, and the response of the authors should not be something like your offense above, but a simple "thanks for pointing out the inconsistency, we are going to fix it, here is our proposed change". The ambiguity between the DualQ and the id drafts I pointed out are real, and "typical RTT" is not a well known term of art that every IETF member or operator is going to understand intuitively.
	Also, if my worst fear is correct, protocol and AQM "typical RTT"  values need to be selected in lock-step to be effective, and such a requirement needs to be made explicit in an RFC. And if I am wrong that the two uses of the term have different definitions that needs to be disambiguated in the internet drafts, no?

> fit there. I would expect that L4S support can be proposed ~ mid next year.

	[SM] So you are willing to push an under-tested solution into 3GPP, but you insist upon that un-tested solution having IETF blessing? That seems rather odd, I would have expected that the primary requirement should be "works robustly and reliably without (unacceptable) side-effects" and not only "has RFC status". I have a hard time believing 3GPP works like that...

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 20 Nov 2020 08:22:54 +0000

Hi

My take is that network equipment need to be upgraded for other reasons than
AQM functionality. And then we have the case that network gear becomes old
or breaks down (my home WiFi routers have been fragged by lightning twice).
No I don't have hard numbers but I don't see that lack of knowledge about
how for instance edge routers are equipped and how upgradeable they are,
should to be used as an argument against WGLC for the L4S drafts (that
target experimental status). 
This applies also to the VPN tunnels discussion in a separate thread.  
These potential issues can be a part on the L4S experiment and solutions can
be documented as a result of this, but it is not necessarily so that L4S
need to adapt to all possible issues. 

/Ingemar

> Sent: den 20 november 2020 08:52
> Cc: Lars Eggert <[email-masked]>; tsvwg IETF list <[email-masked]>
> > On Nov 19, 2020, at 15:24, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> > Hi
> >
> > Please see inline [IJ]
> >
> > /Ingemar
> >
> >>> and come with requests for clarifications where needed. Until then
> >>> you can only expect more of the same long incomprehensible
> >>> discussion threads until March, when we will repeat the same process
> again.
> >>
> >> One point I am trying to make is that since the set of documents we
> >> are discussing seems incomplete, in that it doesn't seem to contain a
> >> TCP
> > variant
> >> My main point though is that there seem to be questions raised about
> >> the performance and behavior of L4S with various TCP variants. This
> >> is not an issue with the content of the L4S drafts. It's a remaining
> >> uncertainty
> > related to
> >> the experimental evaluation and analysis that the L4S mechanisms have
> >> seen so far. Going forward with a LC is not going to bring further
> >> clarity
> > here.
> >>
> >>> [IJ] The only pain point I see now is the RTT bias in cases where
> >>> long RTT Prague flows compete with short RTT ditto. This is being
> >>> addressed by the developers and it is not only an L4S problem.
> >>> Besides this, Prague will be presented at ICCRG tomorrow as I
understand
> it.
> > [IJ] My take is that this is something for L4S ops that I see as a
> > product of the L4S experiment. It is not something that needs to be
> > answered on its own as it is mainly the RFC3168 AQM issue in another
> > shape, we don't know how widely spread this problem is or to how large
> > extent this equipment can be updated, earlier discussions on fq-codel
> > implementations in home gateways was not conclusive I guess but it
> > appears that many of these have automatic upgrade features,
> 
> 	[SM] This is simply untrue, most home router will only see an update
if
> the user actively monitor's the provider's or manufacturer's website and
most
> manufacturer's will only create and distribute updates around the time a
> device is still marketed/sold. There is pretty little home gear out these
which
> gets automatic updates or only notifications of possible updates. I agree
that
> that would be a great situation to be in, but realistically the home
router
> update and security situation is only mildly above the update and security
> situation of IoT devices. Remember it is the "s" in IoT that stands for
> security/safety...
> 	So, if your stance is, automatic updates are robust, reliable and
wide-
> spread, please post some references supporting that hypothesis.
> 
> 
> 
> 
> > I guess the reason here is to be able to upgrade to combat security
> > threats but this is admittedly speculation.
> 
> 	[SM] For someone putting their operational safety in that basket I
> would have expected more robust and reliable numbers for how many
> devices might be amenable to automated updates. Like hard numbers
> supporting the hypothesis that automatic updates are wide spread enough to
> allow distribution of such functionality updates. Also think about who is
> supposed to make back-ports of the required Linux kernel changes to the
> often ancient kernel versions SoC-SDKs are based on? To be honest that
level
> of engineering by wishful thinking frightens me a bit.
> 
> 
> >
> >>
> > [IJ] Not that I know of, code is found at
> > https://protect2.fireeye.com/v1/url?k=3f509ea0-60cba7e3-3f50de3b-
> 861fc
> > b972bfc-3fb038567b51ac5d&q=1&e=0ad66ee5-96b6-405d-84d7-
> e83c5a03bcba&u=
> > https%3A%2F%2Fgithub.com%2FL4STeam%2F
> 
> 	[SM] Is it just me that sees a problem in that the reference
protocol
> implementation just exists in a repository, without even an attempt of
writing
> an internet dratf describing that protocol? There are zero guarantees the
> tomorrows TCP Prague still is L4S compliant (today's is not, the rfc3168
> detection and fall back are disabled by default, see
> https://protect2.fireeye.com/v1/url?k=c7557841-98ce4102-c75538da-
> 861fcb972bfc-22e4ec3d607132d2&q=1&e=0ad66ee5-96b6-405d-84d7-
> e83c5a03bcba&u=https%3A%2F%2Fgithub.com%2FL4STeam%2Flinux%2Fcom
> mit%2Fb256daedc7672b2188f19e8b6f71ef5db7afc720).
> 
> 
> >>
> > [IJ] Yes, but we need to get past the discussion on all possible
> > things that can happen. Recall from the discussion yesterday that
> > Stuart Cheshire and his team has not need evidence of ECN marking
> > AQMs. So I fear that we spend a lot of time discussing problems that are
> very rare.
> 
> 	[SM] Ingemar, I and other OpenWrt users have an rfc3168 AQM on
> my internet access link, that by default uses ECN for the downlink. I do
not
> have the view of the world as apple has, but I can guarantee the number is
> grater than zero. And as stated before people doing this today are exactly
the
> latency conscious users L4S should aim at winning over...
> 
> >
> >>
> >>> examples that are disclosed in the open are Broadcom and Nokia. I
> >>> can imagine that there is some expectation that L4S will materialize
> >>> in RFCs
> >>
> > [IJ] I would say that we already have CC algos that can be used.
> > Surely they do not meet all the requirements today but I don't see why
> > it will not be possible.
> 
> 	[SM] This tells more about your confidence in your vision than in
what
> might be achiebale in those CCs. IMHO ithe onus is on team L4S to
> demonstrate that possibility by demonstrating a CC that ticks all the
check
> marks (does not need to be perfect, but should show significantly better
> performance in some dimension, while doing at least no harm in others,
> currently TCP Prague fails on both accounts).
> 
> 
> > It is definitely the case that we will be discussing what typical RTTs
> > are like forever but that is not something that should delay the L4S
> > drafts I think.
> 
> 	[SM] Sorry, this is getting absurd, I point out both Bob completely
> missing how PIE works (and what it's target variable actually describes )
and an
> inconsistency between two of the L4S drafts only to be ridiculed here.
> Ingemar, publishing RFC with inconsistent terms and definition is not a
sign of
> a professional authoring process. Reviews are supposed to help finding
such
> issues, and the response of the authors should not be something like your
> offense above, but a simple "thanks for pointing out the inconsistency, we
are
> going to fix it, here is our proposed change". The ambiguity between the
> DualQ and the id drafts I pointed out are real, and "typical RTT" is not a
well
> known term of art that every IETF member or operator is going to
understand
> intuitively.
> 	Also, if my worst fear is correct, protocol and AQM "typical RTT"
> values need to be selected in lock-step to be effective, and such a
> requirement needs to be made explicit in an RFC. And if I am wrong that
the
> two uses of the term have different definitions that needs to be
> disambiguated in the internet drafts, no?
> 
> 
> >
> >>
> > [IJ] L4S is not currently pushed for in 3GPP, there is however work on
> > the support for extended reality that requires low latency. L4S will
> > definitely fit there. I would expect that L4S support can be proposed ~
mid
> next year.
> 
> 	[SM] So you are willing to push an under-tested solution into 3GPP,
> but you insist upon that un-tested solution having IETF blessing? That
seems
> rather odd, I would have expected that the primary requirement should be
> "works robustly and reliably without (unacceptable) side-effects" and not
only
> "has RFC status". I have a hard time believing 3GPP works like that...
> 
> Best Regards
> 	Sebastian
> 
> 
> >
> >>
> >> Thanks,
> >> Lars

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 20 Nov 2020 09:34:12 +0100

Hi Ingemar,

> On Nov 20, 2020, at 09:22, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 
> AQM functionality.

	[SM] In the home router space that time point often only comes when either:
1) the old device ceases to work/operate at all (block smoke)
2) the old device is not compatible with the access tech anymore (ADSL router on VDSL2 plant)

These have time frames on the order of 5-15 years, if you are in for the long haul, you might be fine, but banking on that for operational safety is a gamble you undertake with somebody else's quality of internet access...

> And then we have the case that network gear becomes old

	[SM] Exactly, but that does not have reliable time constants to predict when the fleet currently in service is going to be replaced, no?

> No I don't have hard numbers

	[SM] Which all of us knew...

> but I don't see that lack of knowledge about

	[SM] But universal deployment. Sorry, as long as L4S has side-effects in the real internet these need to be taken into consideration, independent whether standards or experimental track RFCs are sought. The critical issue here is not word in an RFC or whether there exists an RFC at al, but what happens in the real world with non-ETC(1) data flows.

> need to adapt to all possible issues. 

	[SM] Again, L4S wants something special, the exclusive right to one rate IP header code point, and the permit to run rough-shed over existing CE users, which exist in the real world. The least to be expected from L4S is to demonstrate that its un-doubted negative side-effects are sufficiently controlled to "do no harm" to the existing internet. The way to do that is not to argue ad infinitum in a mailing list why you BELIEVE that these concerns don't matter, but to run experiments and show empirical data demonstrating these concerns can be safely put to rest.
	As S. Blake explained (again) to ru these experiments over the existing internet, no RFC is required and using carefully selected DSCPs and SLA's the participating partners can make sure to not affect unsuspecting parties during those experiments.

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 20 Nov 2020 09:10:24 +0000

Hi Sebastian. 

I appears to me that we have had this discussion for 6 months of more and it
is quite obvious that we can agree to disagree. 
I don't see that the potential audience in this WG will become much wiser if
we continue the discussion for yet another 6 months or more. 
My standpoint remains that these issues are to be addressed during the L4S
experiment. 

And.. to help this experiment gain traction also outside IETF it is
necessary that the L4S draft have RFC status.

/Ingemar

> Sent: den 20 november 2020 09:34
> > On Nov 20, 2020, at 09:22, Ingemar Johansson S
> > My take is that network equipment need to be upgraded for other
> > reasons than AQM functionality.
> 
> These have time frames on the order of 5-15 years, if you are in for the
long
> haul, you might be fine, but banking on that for operational safety is a
gamble
> you undertake with somebody else's quality of internet access...
> 
> 
> > And then we have the case that network gear becomes old or breaks down
> > (my home WiFi routers have been fragged by lightning twice).
> 
> 	[SM] Exactly, but that does not have reliable time constants to
predict
> when the fleet currently in service is going to be replaced, no?
> 
> 
> > but I don't see that lack of knowledge about how for instance edge
> > routers are equipped and how upgradeable they are, should to be used
> > as an argument against WGLC for the L4S drafts (that target
> > experimental status).
> 
> 	[SM] But universal deployment. Sorry, as long as L4S has
side-effects
> in the real internet these need to be taken into consideration,
independent
> whether standards or experimental track RFCs are sought. The critical
issue
> here is not word in an RFC or whether there exists an RFC at al, but what
> happens in the real world with non-ETC(1) data flows.
> 
> 
> > These potential issues can be a part on the L4S experiment and
> > solutions can be documented as a result of this, but it is not
> > necessarily so that L4S need to adapt to all possible issues.
> 
> 	[SM] Again, L4S wants something special, the exclusive right to one
> rate IP header code point, and the permit to run rough-shed over existing
CE
> users, which exist in the real world. The least to be expected from L4S is
to
> demonstrate that its un-doubted negative side-effects are sufficiently
> controlled to "do no harm" to the existing internet. The way to do that is
not
> to argue ad infinitum in a mailing list why you BELIEVE that these
concerns
> don't matter, but to run experiments and show empirical data demonstrating
> these concerns can be safely put to rest.
> 	As S. Blake explained (again) to ru these experiments over the
existing
> internet, no RFC is required and using carefully selected DSCPs and SLA's
the
> participating partners can make sure to not affect unsuspecting parties
during
> those experiments.
> >>>>> The only way is see is that the L4S drafts are moved to WGLC, then
> >>>>> people will hopefully read the drafts and come with requests for
> >>>>> clarifications where needed. Until then you can only expect more
> >>>>> of the same long incomprehensible discussion threads until March,
> >>>>> when we will repeat the same process
> >> again.
> >>>>
> >>>> are discussing seems incomplete, in that it doesn't seem to contain
> >>>> a TCP
> >>> variant
> >>>> My main point though is that there seem to be questions raised
> >>>> about the performance and behavior of L4S with various TCP
> >>>> variants. This is not an issue with the content of the L4S drafts.
> >>>> It's a remaining uncertainty
> >>> related to
> >>>> the experimental evaluation and analysis that the L4S mechanisms
> >>>> have seen so far. Going forward with a LC is not going to bring
> >>>> further clarity
> >>> here.
> >>>>
> >>> shape, we don't know how widely spread this problem is or to how
> >>> large extent this equipment can be updated, earlier discussions on
> >>> fq-codel implementations in home gateways was not conclusive I guess
> >>> but it appears that many of these have automatic upgrade features,
> >>
> >> manufacturer's will only create and distribute updates around the
> >> time a device is still marketed/sold. There is pretty little home
> >> gear out these
> > which
> >> update and security situation is only mildly above the update and
> >> security situation of IoT devices. Remember it is the "s" in IoT that
> >> stands for security/safety...
> >> supporting the hypothesis that automatic updates are wide spread
> >> enough to allow distribution of such functionality updates. Also
> >> think about who is supposed to make back-ports of the required Linux
> >> kernel changes to the often ancient kernel versions SoC-SDKs are
> >> based on? To be honest that
> > level
> >> an internet dratf describing that protocol? There are zero guarantees
> >> the tomorrows TCP Prague still is L4S compliant (today's is not, the
> >> rfc3168 detection and fall back are disabled by default, see
> my
> >> internet access link, that by default uses ECN for the downlink. I do
> > not
> >> have the view of the world as apple has, but I can guarantee the
> >> number is grater than zero. And as stated before people doing this
> >> today are exactly
> > the
> >>>>> As regards to investment, already today there is investment in
> >>>>> this, examples that are disclosed in the open are Broadcom and
> >>>>> Nokia. I can imagine that there is some expectation that L4S will
> >>>>> materialize in RFCs
> >>>>
> >>> Surely they do not meet all the requirements today but I don't see
> >>> why it will not be possible.
> >>
> >> marks (does not need to be perfect, but should show significantly
> >> better performance in some dimension, while doing at least no harm in
> >> others, currently TCP Prague fails on both accounts).
> >>
> >>
> >>> RTTs are like forever but that is not something that should delay
> >>> the L4S drafts I think.
> >>
> >> issues, and the response of the authors should not be something like
> >> your offense above, but a simple "thanks for pointing out the
> >> inconsistency, we
> > are
> >> going to fix it, here is our proposed change". The ambiguity between
> >> the DualQ and the id drafts I pointed out are real, and "typical RTT"
> >> is not a
> > well
> >>>>> however try to address this also in 3GPP standards to make the
> >>>>> whole thing work in products, and that is of course hard to do if
> >>>>> L4S is not
> >>> even an
> >>>> RFC.
> >>>>
> >>> [IJ] L4S is not currently pushed for in 3GPP, there is however work
> >>> on the support for extended reality that requires low latency. L4S
> >>> will definitely fit there. I would expect that L4S support can be
> >>> proposed ~
> > mid
> >> next year.
> >>
> >> rather odd, I would have expected that the primary requirement should
> >> be "works robustly and reliably without (unacceptable) side-effects"
> >> and not
> > only

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 20 Nov 2020 10:26:57 +0100

Hi Ingemar,

more below in-line, prefixed [SM].

Tl;dr: We are discussing whether L4S operational safety can be achieved by requiring/expecting that all existing network nodes will be updated in time to fix up the safety gaps the current L4S design and implementation introduced. 

> On Nov 20, 2020, at 10:10, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Sebastian. 
> 

	[SM] Ingemar, we are talking about hard facts here, that will not change whether you accept them or not. There are no "alternative" facts in reality. Either you are right, and all relevant network nodes (including all CPE/home-routers) can be updated to accommodate L4S' new demands on the network layers in sufficient time, or they can not. The conservative prediction is that they can and will not (unless we are talking about a time frame for starting the L4S roll-out of another decade). You say, that is wromg and it can be done mich quicker, and hence L4S can skip safety steps. Fine, now proof that your prediction is likely and we can talk. But continuously claiming other peoples updates will safeL4S' posterior without giving data that supports that prediction is simply the same old engineerng by wishful thinking that I have complained about in the past.

> we continue the discussion for yet another 6 months or more. 

	[SM] This is why you need not to simply repeat your assertions and conviction, but present data that convince others that your view holds merit and is not sinply taken because it is convenient to push L4S through this WG's process.

> My standpoint remains that these issues are to be addressed during the L4S

	[SM] Tat is a tall order to add to the L4S experiment, solving the existing critical update and security issue that afflicts home-routers (and similarly IoT)... 

> necessary that the L4S draft have RFC status.

	[SM] That is conjecture, sorry. You are trying to make the WG "ratify" the current L4S drafts by implying that 3GPP is currentky up-help by lack of RFC status, which seems not exactly true. IIUC 3GPP has in no way committed to L4S, it is just you (and maybe others) that want to make it that way. No doubt, that for this endeavor RFC status could be helpful, but please do not paint that as an already sorted and settled 3GPP requirement.

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 20 Nov 2020 09:44:08 +0000

Hi

Snip-snip
"IIUC 3GPP has in no way committed to L4S, it is just you (and maybe others)
that want to make it that way. No doubt, that for this endeavor RFC status
could be helpful, but please do not paint that as an already sorted and
settled 3GPP requirement."
Well.., I don't...

/Ingemar 

> Sent: den 20 november 2020 10:27
> Tl;dr: We are discussing whether L4S operational safety can be achieved by
> requiring/expecting that all existing network nodes will be updated in
time to
> fix up the safety gaps the current L4S design and implementation
introduced.
> 
> 
> > On Nov 20, 2020, at 10:10, Ingemar Johansson S
> > I appears to me that we have had this discussion for 6 months of more
> > and it is quite obvious that we can agree to disagree.
> 
> 	[SM] Ingemar, we are talking about hard facts here, that will not
> change whether you accept them or not. There are no "alternative" facts in
> reality. Either you are right, and all relevant network nodes (including
all
> CPE/home-routers) can be updated to accommodate L4S' new demands on
> the network layers in sufficient time, or they can not. The conservative
> prediction is that they can and will not (unless we are talking about a
time
> frame for starting the L4S roll-out of another decade). You say, that is
wromg
> and it can be done mich quicker, and hence L4S can skip safety steps.
Fine,
> now proof that your prediction is likely and we can talk. But continuously
> claiming other peoples updates will safeL4S' posterior without giving data
that
> supports that prediction is simply the same old engineerng by wishful
thinking
> that I have complained about in the past.
> 
> 
> > I don't see that the potential audience in this WG will become much
> > wiser if we continue the discussion for yet another 6 months or more.
> 
> 	[SM] This is why you need not to simply repeat your assertions and
> conviction, but present data that convince others that your view holds
merit
> and is not sinply taken because it is convenient to push L4S through this
WG's
> process.
> 
> 
> > My standpoint remains that these issues are to be addressed during the
> > L4S experiment.
> 
> 	[SM] Tat is a tall order to add to the L4S experiment, solving the
> existing critical update and security issue that afflicts home-routers
(and
> similarly IoT)...
> 	[SM] That is conjecture, sorry. You are trying to make the WG
"ratify"
> the current L4S drafts by implying that 3GPP is currentky up-help by lack
of
> RFC status, which seems not exactly true. IIUC 3GPP has in no way
committed
> to L4S, it is just you (and maybe others) that want to make it that way.
No
> doubt, that for this endeavor RFC status could be helpful, but please do
not
> paint that as an already sorted and settled 3GPP requirement.
> 
> Best Regards
> 	Sebastian
> 
> 
> 
> 
> >
> > /Ingemar
> >
> >
> >>> And then we have the case that network gear becomes old or breaks
> >>> down (my home WiFi routers have been fragged by lightning twice).
> >>
> >> here is not word in an RFC or whether there exists an RFC at al, but
> >> what happens in the real world with non-ETC(1) data flows.
> >>
> >>
> >> demonstrate that its un-doubted negative side-effects are
> >> sufficiently controlled to "do no harm" to the existing internet. The
> >> way to do that is
> > not
> >> don't matter, but to run experiments and show empirical data
> >> demonstrating these concerns can be safely put to rest.
> >>>>>>> The only way is see is that the L4S drafts are moved to WGLC,
> >>>>>>> then people will hopefully read the drafts and come with
> >>>>>>> requests for clarifications where needed. Until then you can
> >>>>>>> only expect more of the same long incomprehensible discussion
> >>>>>> One point I am trying to make is that since the set of documents
> >>>>>> we are discussing seems incomplete, in that it doesn't seem to
> >>>>>> contain a TCP
> >>>>> variant
> >>>>>>> [IJ] The only pain point I see now is the RTT bias in cases
> >>>>>>> where long RTT Prague flows compete with short RTT ditto. This
> >>>>>>> is being addressed by the developers and it is not only an L4S
> problem.
> >>>>> product of the L4S experiment. It is not something that needs to
> >>>>> be answered on its own as it is mainly the RFC3168 AQM issue in
> >>>>> another shape, we don't know how widely spread this problem is or
> >>>>> to how large extent this equipment can be updated, earlier
> >>>>> discussions on fq-codel implementations in home gateways was not
> >>>>> conclusive I guess but it appears that many of these have
> >>>>> automatic upgrade features,
> >>>>
> >>>> security situation of IoT devices. Remember it is the "s" in IoT
> >>>> that stands for security/safety...
> >>>> think about who is supposed to make back-ports of the required
> >>>> Linux kernel changes to the often ancient kernel versions SoC-SDKs
> >>>> are based on? To be honest that
> >>> level
> >>>> an internet dratf describing that protocol? There are zero
> >>>> guarantees the tomorrows TCP Prague still is L4S compliant (today's
> >>>> is not, the
> >>>>>>> Nokia. I can imagine that there is some expectation that L4S
> >>>>>>> will materialize in RFCs
> >>>>>>
> >>>> better performance in some dimension, while doing at least no harm
> >>>> in others, currently TCP Prague fails on both accounts).
> >>>>
> >>>>
> >>>> 	[SM] Sorry, this is getting absurd, I point out both Bob
> >>>> completely missing how PIE works (and what it's target variable
> >>>> actually describes )
> >>> and an
> >>>> issues, and the response of the authors should not be something
> >>>> like your offense above, but a simple "thanks for pointing out the
> >>>> inconsistency, we
> >>> are
> >>>> going to fix it, here is our proposed change". The ambiguity
> >>>> between the DualQ and the id drafts I pointed out are real, and
"typical
> RTT"
> >>>>>>> whole thing work in products, and that is of course hard to do
> >>>>>>> if L4S is not
> >>>>> even an
> >>>>>> RFC.
> >>>>>>
> >>>>> [IJ] L4S is not currently pushed for in 3GPP, there is however
> >>>>> work on the support for extended reality that requires low
> >>>>> latency. L4S will definitely fit there. I would expect that L4S
> >>>>> support can be proposed ~
> >>> mid
> >>>> next year.
> >>>>
> >>>> 	[SM] So you are willing to push an under-tested solution into
> >>>> 3GPP, but you insist upon that un-tested solution having IETF
blessing?
> >>>> That
> >>> seems
> >>>> rather odd, I would have expected that the primary requirement
> >>>> should be "works robustly and reliably without (unacceptable) side-
> effects"
> >>>> and not
> >>> only

---

From: Greg White <[email-masked]>
Date: Fri, 20 Nov 2020 22:14:40 +0000

This is great news!  Those users will undoubtedly be asking for (or implementing and submitting PRs to enable) L4S support in OpenWrt (it’s a simple software change after all, no kernel updates or new hardware needed).  They will be some of the first to see the benefits of it!  I would hope, for the sake of the Internet and the IETF, that people like you will do what you can to ensure that the L4S experiment has the best chance of success.

I don’t think I’ve heard you say that you are opposed to high-fidelity congestion control, or that you believe that it is impossible or inherently dangerous.  I seem to remember that you were (are?) of the belief that it would be a good thing in the context of the SCE codepoint definition, you just aren’t happy with the L4S codepoint decision.  Actually, I don’t remember hearing anyone claim that high-fidelity congestion signaling makes congestion control worse, or is inherently dangerous.

So, I think the discussion now centers on the following points?

  *   TCP Prague is unfinished, and some would like to see it finished before WGLC on the L4S drafts.
     *   Current TCP Prague performance does not show the benefits of high-fidelity CC in a number of cases.  Many believe that this is not an indication that high-fidelity CC is impossible, but others still want to be convinced.
  *   The DualQ AQM does not provide as precise per-flow fairness as FQ (or possibly VDQ-CSAQM) with the current TCP Prague
  *   Coexistence in light of single queue RFC3168 bottlenecks is not well understood (their prevalence, the frequency with which they are saturated with mixed-type long-running flows of moderate BDP, etc.) so we need to have safeguards in case it is a real issue
  *   Tunnels carrying mixed traffic can exhibit intra-tunnel flow unfairness in current FQ RFC3168 ECN bottlenecks, and non-RFC6040 compliant tunnels can still exhibit that issue in L4S-aware FQ bottlenecks

Also, there seems to be some difference of opinion in the community on the fundamental question of whether it is considered acceptable for a new congestion control to cause some level of harm (in the Ware sense*) to existing CCs, even if it otherwise provides substantial benefits that would be desirable to its users and encourage its adoption (and thus contribute to the decline in deployment of the status quo CCs).  In my personal opinion, strict adherence to zero-harm is too constraining, since it doesn’t provide any way to assess the upside potential of a new algorithm.

* https://www.cs.cmu.edu/~rware/assets/pdf/ware-hotnets19.pdf

-Greg

From: tsvwg <[email-masked]> on behalf of Sebastian Moeller <[email-masked]>
Date: Friday, November 20, 2020 at 12:52 AM
To: Ingemar Johansson S <ingemar.s.johansson=[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] Reasons for WGLC/RFC asap

Hi Ingemar,

a correction below.

> On Nov 19, 2020, at 15:24, Ingemar Johansson S  wrote:

>> From: Lars Eggert

>> To: Ingemar Johansson S

>> Cc: Steven Blake ; tsvwg IETF list

>>  wrote:

> [IJ] Not that I know of, code is found at https://protection.greathorn.com/services/v2/lookupUrl/e306a141-abad-430f-a4f2-02ad90bdc40e/327/a503b21fc948618a92ec662ee8b3671098594e6d?domain=github.com&path=/L4STeam/

  [SM] Is it just me that sees a problem in that the reference protocol implementation just exists in a repository, without even an attempt of writing an internet dratf describing that protocol? There are zero guarantees the tomorrows TCP Prague still is L4S compliant (today's is not, the rfc3168 detection and fall back are disabled by default, see https://protection.greathorn.com/services/v2/lookupUrl/81c6e38a-dede-4789-8d8c-692e8b1cf6f9/327/a503b21fc948618a92ec662ee8b3671098594e6d?domain=github.com&path=/L4STeam/linux/commit/b256daedc7672b2188f19e8b6f71ef5db7afc720).

>>> imagine that there is some expectation that L4S will materialize in

>>> RFCs

Best Regards

  Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 21 Nov 2020 09:25:41 +0100

Hi Greg,

more below, prefixed [SM].

> On Nov 20, 2020, at 23:14, Greg White <[email-masked]> wrote:
> 
> This is great news!  

	[SM] I appreciate your positive tone, but IMHO tone isn't to only or even the biggest issue L4S faces. The fact that team L4S continuously ignores/talks down my inputs, even though I have relevant in the field experience? Not sure I would call that "great". 

> Those users will undoubtedly be asking for (or implementing and submitting PRs to enable) L4S support in OpenWrt (it’s a simple software change after all, no kernel updates or new hardware needed).  

	[SM] What are you talking about, here? Qdiscs in Linux come as part of the kernel, so a kernel update would be very much needed tp\o get an L4S compatible AQM out to the users. But first such an AQM would need to be included in the upstream kernel, OpenWrt tries hard  to stay close to the kernel and carry as little non-"mainline-able code as possible.

> They will be some of the first to see the benefits of it!  

	[SM] Let's not get carries away here, please. The question in the WG still is can L4S actually demonstrate its promised performance (without introducing additional regressions) over a number of realistic conditions. And the answer to that challenge so far is a "meh" at best. 

> I would hope, for the sake of the Internet and the IETF, that people like you will do what you can to ensure that the L4S experiment has the best chance of success.

	[SM] Well, to get on board, L4S first needs to convince me. Interactions with Koen and Bob leave me under the impression that L4S is not exceptionally well engineered. Both offer quite shall we say "interesting" theories about how RTT-dependence happens, or how PIE and CoDel work that are so far removed from reality that I have a hard time accepting any of their assessments at face value without first looking at supporting data. And that gets to the next problem, many of my requests for data have been responded to with "go measure yourself if you want to see that data". I have very little good-will left, abd I have a considerably less rosy view on current L4S chance to succeed in the real internet than you. And with the current state of affairs I am not going to hang my name and the little reputation I have on the L4S train, sorry.

>  
> I don’t think I’ve heard you say that you are opposed to high-fidelity congestion control, or that you believe that it is impossible or inherently dangerous.

	[SM] I consider HFCC an interesting idea in which I see promise. I now want to see hard-boiled evaluation and testing to see how well this idea stands up to the harsh reality of the existing internet, yes. And sure I HOPE it will prove itself, but let's get the data first and let's look at the eventual results accepting the possibility that HFCC might not actually work well in reality.

>  I seem to remember that you were (are?) of the belief that it would be a good thing in the context of the SCE codepoint definition, you just aren’t happy with the L4S codepoint decision.  

	[SM] Yes, I still consider the re-definition of CE as a completely gratuitous change that causes dangerous safety issues with existing users. And I do not like the  way team L4S treats the resulting safety issues. Merely trying to argue issues away solves nothing. Actively resisting calls to generate relevant data does not fill me with confidence. Behaving as if year old data measured with DCTCP would extrapolate to TCP Prague without even a shred of data showing that; in spite of the fact that TCP Prague introduced significant changes to its DCTCP ancestor, and hence is expected to behave differently. Not sure I would call any of the great or even sufficient diligent engineering.

> Actually, I don’t remember hearing anyone claim that high-fidelity congestion signaling makes congestion control worse, or is inherently dangerous.

	[SM] This is pretty much the reason IMHO, why the L4S drafts are still under consideration, but a one point L4S will need to deliver on its promises. Unfortunately it is not there yet (and after 7? years of really slow progress, I am not convinced that it ever will get there).

> 	• TCP Prague is unfinished, and some would like to see it finished before WGLC on the L4S drafts.

	[SM] Not to put too fine a point, if the project is to introduce HFCC to the internet, that HFCC should be one on the core products of the L4S project, but it is treated as a side show and preferable somebody else's problem. Under a conservative assumption that signaling mechanism and reaction to the signaling are not completely orthogonal issues, it seems obvious that they should be developed and evaluated in lock-step. But that is almost the opposite of the L4S approach. I happen to predict that as a consequence of that approach, the network bits are far from optimal or even good enough, as their interaction with the HFCC in the endpoints is understudied.

> 		• Current TCP Prague performance does not show the benefits of high-fidelity CC in a number of cases.  

	[SM] +1; unfortunately that includes quite a number of realistic and common conditions, like multiple flows as different RTTs, where L4S TCP Prague fall way behind the status quo.

> Many believe that this is not an indication that high-fidelity CC is impossible, but others still want to be convinced.

	[SM] Not sure anybody claimed from L4S/TCP Pragues failures that HFCC is impossible, but rather that the L4S approach is simply lackluster and not good enough.

> 	• The DualQ AQM does not provide as precise per-flow fairness as FQ (or possibly VDQ-CSAQM) with the current TCP Prague

	[SM] That is too mild a description, current DualQ will introduce a robust and reliable RATE advantage for TCP Prague at any RTT combination. That is not "not provide as precise per-flow fairness" that is actual predictable UNFAIRNESS. Let's call a spade a spade here, no need for euphemisms or window dressing, this is an engineering, not a marketing organisation.

> 	• Coexistence in light of single queue RFC3168 bottlenecks is not well understood (their prevalence, the frequency with which they are saturated with mixed-type long-running flows of moderate BDP, etc.) so we need to have safeguards in case it is a real issue

	[SM] +1; unfortunately the currentt approach seems to be tailored to pass a few conditions that were used in the tests demonstrating that issue in the first place and constitute no robust or general solution to the issue. As seen how classification performance tanks when a bit more jitter is introduced than the algorithm designer expected.

> 	• Tunnels carrying mixed traffic can exhibit intra-tunnel flow unfairness in current FQ RFC3168 ECN bottlenecks, and non-RFC6040 compliant tunnels can still exhibit that issue in L4S-aware FQ bottlenecks 

	[SM] Yepp, mostly a consequence of re-defining CE, and quite ironic given the stated rationale for that re-definition was to better deal with tunnels... (Which IMHO is not a good argument, as initial deployment will be mostly at the ISP end-user edge, where tunneling is still possible but certainly not the majority use case and the end-user might be able to adjust the tunnel behavior).

>  
> Also, there seems to be some difference of opinion in the community on the fundamental question of whether it is considered acceptable for a new congestion control to cause some level of harm (in the Ware sense*) to existing CCs,

	[SM] This is not a qualitative, but a quantitative question.

> even if it otherwise provides substantial benefits

	[SM] Yes, a realistic honest ledger showing benefits and costs, without the usual marketing and hype would be a good document to offer the the WG to help in that decision.

> that would be desirable to its users and encourage its adoption (and thus contribute to the decline in deployment of the status quo CCs).

	[SM] This is gated on the open question whether L4S as currentky designed and implemented works well enough to begin with. I see a considerable mismatch between actual delivered performance and promised performance and I would at the very least expect that these are brought in line, hopefully by toning down the overly excited promises in the drafts and by fixing the implementation to actually work under broader conditions and without an L4S over classic bias.

>  In my personal opinion, strict adherence to zero-harm is too constraining, since it doesn’t provide any way to assess the upside potential of a new algorithm.

	[SM] +1; but current L4S' issue is not that it is not given leeway to do a bit more harm, but that ATM its harm on rate in some conditions realy seems bound by TCPs unwillingness to reduce its congestion widows below to segments...

> * https://www.cs.cmu.edu/~rware/assets/pdf/ware-hotnets19.pdf

	[SM] Yes, great link from Wes, but I am sure we will end in weird food-fight immediately, as team L4S will try to trade the harm done on the rates of non-L4S flows with the benefit to the latency under load for the L4S flows. Ignoring the fact that in both cases the harm-benefits are pro L4S and contra non-L4S. 

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 21 Nov 2020 16:32:15 +0200

> On 21 Nov, 2020, at 10:25 am, Sebastian Moeller <[email-masked]> wrote:
> 

As the Ware paper already points out, the "harm" metric deliberately *ignores* the benefit to the newcomer CC's traffic, and focuses on the harm done to status-quo CCs' traffic.  In this respect, merely glancing at the test data posted by myself and Pete Heist over the past few weeks (and indeed the past year) shows that L4S and TCP Prague are capable of inflicting substantial harm on existing CUBIC traffic, which has been established as being quite prevalent in today's Internet.

The acceptable threshold of "harm" to permit deployment is not zero, but some function of the harm done by the same workload added using the same status-quo CC.  The precise form of that threshold function is not yet fully established, but it is possible to obtain objective numbers describing "harm" and begin to analyse them.  I predict that when such analysis is published, L4S and TCP Prague will not come out of it looking good.  But it may take some time to rejig our test suites to generate accurate harm metrics.

 - Jonathan Morton