Subject: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

From: Pete Heist <[email-masked]>
Date: Thu, 12 Nov 2020 21:31:12 +0100

Hi,

We have posted some new tests of L4S here:

https://github.com/heistp/l4s-tests

These tests cover two-flow fairness in several qdiscs with different
path RTTs per-flow, and transient behavior in fq_codel queues.

The results raise some concerns about a bias in favor of L4S flows in
DualPI2 queues, throughput imbalances between flows with different path
RTTs, and intra-flow latency spikes upon rate reductions in fq_codel.
The repo above contains a walk-through of the key findings, and links
to more results in the Appendix.

Regards,
Pete

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 13 Nov 2020 00:11:15 +0100

Hi Pete,

great data. IMHO Especially the fact that short-RTT Prague will severely outcompete long-RTT Prague way more than the traditional dumb FIFO will, strongly supports  the following hypotheses I have been posting before:

a) Dualpi2/TCP Prague are no way ready for deployment, but are basically still in the toy stage

b) all that L4S will effectively do, be it by intent or simply by mis-design, is built a "fast lane" for short RTT low hop count traffic. Given all the hype (still!) in the L4S internet drafts about how this is the future of the internet... Also a fast lane that requires active cooperation from the leaf ISP (if they keep their bottlenecks at FIFO, I see no compelling reason for TCP Prague at all), which will require SLAs between CDNs and ISPs.

c) too little, too late. Really only modest gains in RTT (compared to best of class AQMs like fq_codel or cake) and noticeable regressions in sharing behavior both between different CCs and flows of different RTTs (and that compared to dumb queue "management" with a simple FIFO).

Also quite sobering, that it AGAIN was not team L4S bringing real data to the table to support their claims and promises. Then again looking at the RTT fairness for Prague versus Prague under DualQ, I can understand why team L4S stayed mumm, and rather argued for allowing unfettered self-mutilation of L4S compliant transport protocols in regards to RTT fairness:

"So there is no need  to mandate that an L4S implementer does no harm to themselves, which window-based congestion controls tend to do at higher RTT. Of course, this doesn't preclude implementers reducing or eliminating RTT bias for larger than typical RTTs, but it removes any requirement to do so."

After years of advertising on increased RTT independence (in spite of the data showing that the proposed combination of DualQ and TCP Prague actually increases RTT bias), team L4S in the last minute no less, decides to do a 180 and just change the requirements to allow for the rather unhealthy behavior demonstrated for L4S... 
With the current enhanced RTT bias, who in their right mind is going to use TCP Prague; which currently is the only transport, that at least attempted to tackle all the "requirements" L4S poses for transports that want to use the ECT(1) express way? Then again, since none of the network elements are designed and required to actually check and enforce these requirements, calling these "requirements" is a bit of stretch anyway, maybe they should be changed from MUST requirements to COULD musings.... 

Best Regards
	Sebastian

> On Nov 12, 2020, at 21:31, Pete Heist <[email-masked]> wrote:
> 
> Hi,
>

---

From: "Black, David" <[email-masked]>
Date: Sun, 15 Nov 2020 01:26:44 +0000

Hi Sebastian,

Some comments as an individual, not a WG chair.

First, I think Pete has pretty clearly established that TCP Prague is research, and hence (IMHO) TCP Prague ought to be headed for ICCRG as its primary forum rather than TSVWG.

To date, the 'Prague L4S Requirements' (Appendix A of draft-ietf-tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague.  That association ought to be teased apart so that the resulting L4S scalable congestion control requirements provide a reasonable design space that can include a number of other congestion control designs - in addition to what's been discussed, e.g., SCReAM, it would be useful to better understand what it would take for implementations of protocols such as DCTCP and BBR (e.g., for QUIC) to meet those requirements.

My overall take on the requirements is that in 20/20 hindsight, some of them were overly optimistic, and hence need to be backed off/toned down/broadened to encompass what is reasonable in "running code" well beyond TCP Prague. That sort of collision between interesting ideas and network realities is not an unheard-of scenario in IETF, so I hesitate to view the need for changes to these requirements as evidence that the original ideas were inherently defective, as I've seen far more dramatic changes, e.g., some number of years ago, the first design of iSCSI login was elegant ... and resulted in implementations that did not interoperate, resulting in a complete redesign.

Thanks, --David

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
> Sent: Thursday, November 12, 2020 6:11 PM
> To: Pete Heist
> Cc: tsvwg IETF list
> Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency
> 
> 
> [EXTERNAL EMAIL]
> 
> Hi Pete,
> 
> great data. IMHO Especially the fact that short-RTT Prague will severely
> outcompete long-RTT Prague way more than the traditional dumb FIFO will,
> strongly supports  the following hypotheses I have been posting before:
> 
> b) all that L4S will effectively do, be it by intent or simply by mis-design, is built a
> "fast lane" for short RTT low hop count traffic. Given all the hype (still!) in the L4S
> internet drafts about how this is the future of the internet... Also a fast lane that
> requires active cooperation from the leaf ISP (if they keep their bottlenecks at
> FIFO, I see no compelling reason for TCP Prague at all), which will require SLAs
> between CDNs and ISPs.
> 
> c) too little, too late. Really only modest gains in RTT (compared to best of class
> AQMs like fq_codel or cake) and noticeable regressions in sharing behavior both
> between different CCs and flows of different RTTs (and that compared to dumb
> queue "management" with a simple FIFO).
> 
> 
> 
> Also quite sobering, that it AGAIN was not team L4S bringing real data to the table
> to support their claims and promises. Then again looking at the RTT fairness for
> Prague versus Prague under DualQ, I can understand why team L4S stayed
> mumm, and rather argued for allowing unfettered self-mutilation of L4S
> compliant transport protocols in regards to RTT fairness:
> 
> "So there is no need  to mandate that an L4S implementer does no harm to
> themselves, which window-based congestion controls tend to do at higher RTT.
> Of course, this doesn't preclude implementers reducing or eliminating RTT bias
> for larger than typical RTTs, but it removes any requirement to do so."
> 
> After years of advertising on increased RTT independence (in spite of the data
> showing that the proposed combination of DualQ and TCP Prague actually
> increases RTT bias), team L4S in the last minute no less, decides to do a 180 and
> just change the requirements to allow for the rather unhealthy behavior
> demonstrated for L4S...
> With the current enhanced RTT bias, who in their right mind is going to use TCP
> Prague; which currently is the only transport, that at least attempted to tackle all
> the "requirements" L4S poses for transports that want to use the ECT(1) express
> way? Then again, since none of the network elements are designed and required
> to actually check and enforce these requirements, calling these "requirements" is
> a bit of stretch anyway, maybe they should be changed from MUST requirements
> to COULD musings....
> 
> 
> Best Regards
> 	Sebastian
> 
> 
> > DualPI2 queues, throughput imbalances between flows with different
> > path RTTs, and intra-flow latency spikes upon rate reductions in fq_codel.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 15 Nov 2020 11:59:08 +0100

Hi David,

thanks for your thoughtful response. Some comments/questions below, prefixed with [SM]

> On Nov 15, 2020, at 02:26, Black, David <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] I fully agree, TCP Prague would be helped considerably by being discussed at the proper WG. To me it is clear that the safety of the L4S elements to be "ratified" by this WG can hardly made dependent on a protocol under active research and development, since at the current time it is still an open question if the TCP Prague claims and aims can realistically be reached; the current TCP Prague falls clearly short of its high aspirations.

> 
> To date, the 'Prague L4S Requirements' (Appendix A of draft-ietf-tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague.  

	[SM] Yes, because so far no other protocol has been aiming for fulfilling the "Prague requirements". I asked both the SCReAM developer and the QUIC maintainer explicitly about that, and Christian ignored the question completely (indicating to me that he takes neither me and/or the Prague requirements seriously), Ingemar (rightly) argued to look at that after L4S is ratified and SCReAM gains official L4S support. 
	That is somewhat a sad state of affairs, because the Prague requirements read a bit like protocol fan-fiction where quite a number of theoretically desirable properties have been listed with little consideration, whether the magnitude and combination of wished for properties is actually realistically implementable. And a bit more discussion of those requirements with actual real-life protocol designers and implementers might have helped to keep the scope realistic, no?

> That association ought to be teased apart so that the resulting L4S scalable congestion control requirements provide a reasonable design space that can include a number of other congestion control designs - in addition to what's been discussed, e.g., SCReAM, it would be useful to better understand what it would take for implementations of protocols such as DCTCP and BBR (e.g., for QUIC) to meet those requirements.

	[SM] +1; yes. Now, how realistic is such a teasing apart and potential redefinition of the protocol parts of L4S after the network parts as currently designed an implemented have been deployed, based on an experimental track RFC? I remember Koen admitting that team L4S's interests in protocols is only marginal, which does not fill me with confidence that their scheme is going to ever work well. I base this on the theoretical observation that communication typically has senders, receivers and signals, and trying to design a meaningful new communication channel by focussing on the signal part alone while mostly ignoring the sender and receiver part seems overly optimistic.

> 
> My overall take on the requirements is that in 20/20 hindsight, some of them were overly optimistic, and hence need to be backed off/toned down/broadened to encompass what is reasonable in "running code" well beyond TCP Prague.

	+1; as I have argued (too) forcefully in the past, especially the RTT de-bias requirement as originally written seems to be impossible to tackle, as long as time flows unidirectionally. But the last proposal for changing that aims considerably too low and does violate the spirit and intention of the original requirement.

> That sort of collision between interesting ideas and network realities is not an unheard-of scenario in IETF, so I hesitate to view the need for changes to these requirements as evidence that the original ideas were inherently defective, as I've seen far more dramatic changes, e.g., some number of years ago, the first design of iSCSI login was elegant ... and resulted in implementations that did not interoperate, resulting in a complete redesign.

	+1; this seems quite realistic. I see two direct consequences from that line of though:

A) If L4S ratification is to happen now, the safety requirements one expects from new network features with to be expected side-effects on other traffic need to be managed and controlled within the new network features and should not be out-sourced onto to-be-done-sometime-in-the-future protocol work (especially since that might look quite different than envisioned right now). IMHO especially dualQ is not finished in that regard and should not be considered "running code", sure it runs, but it does not do what the drafts promise nor what seems necessary under your observation that the Prague requirements are still quite fluid.

B) This WG should come up with an actionable plan on how to assess L4S safety and success and a pre-determined procedure how to undo the L4S experiment entirely (including re-gaining the ECT(1) codepoint for other uses) in case that the current design and implementation of the network features proves to not yield the desired outcome once actually used protocols actually attempt to meet the Prague requirements.

	Now, what I believe is that this WG seems to actually go for option C) of not requiring the network features to be safe by design and not requiring a clear exit-strategy in case of failure. I guess the best I can hope for is that in the course of the rfc4858 process my objections will be reported as extreme discontent under clause 3.1. (1.f).

Best Regards
        Sebastian

P.S.: To head off that argument before it is being repeated by team L4S, if the current state is the best under the self-imposed design constraints for the L4S AQM, then it clearly is time to change those constraints. If I can not participate safely in normal traffic with my hand tied behind my back, maybe I need to untie my hands instead of arguing for laxer requirements to account for my self-imposed handicap?

---

From: Pete Heist <[email-masked]>
Date: Sun, 15 Nov 2020 12:42:27 +0100

Hi guys, and thanks for the replies.

Firstly, there was a new test result added for tunnels yesterday, but
I'll put that in a separate thread so it isn't buried.

Beyond that, I think it would be productive to try to figure out what
might be done to solve the reported issues, limiting that exploration
for now to the confines of the L4S design. We'd like to support forward
progress while facing the issues head on if possible. I'll take a shot
here as tersely as I can:

1) Starting with an easier one, can the reported intolerance to bursts
(https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by
starting marking in the L queue later? Doing so may also improve RTT
fairness for Prague flows.

2) For the reported network bias, can this be fixed in the DualPI2 and
Prague implementations? One chart which we didn't reference in our
writeup illustrates the consistent bias for Prague vs CUBIC.
Essentially, Prague wins until there is an RTT imbalance of 20ms/80ms:
 
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/s1-charts/l4s_network_bias.svg

3) The issues that arise due to the redefinition of CE may be the
hardest. This includes the domination of L4S over non-L4S flows in the
same RFC3168 queue (see the issue with tunnels reported above for a
common path to that), and the intra-flow latency spikes for L4S flows
(https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes). 

Afaik, the proposed solutions in the L4S architecture are:

* RFC3168 bottleneck detection in the endpoints. However, we went
through a round of testing earlier this year that showed accurate
bottleneck detection is likely to be difficult with jitter and cross-
flow traffic. It's disabled at present in the reference implementation.

* L4S operational guidance for network operators. So far I haven't had
time to review this, but I may have some feedback at least in the area
of testing. I suspect the effectiveness of guidance will be influenced
by human factors.

* To not consider a 16:1 throughput imbalance between L4S and non-L4S
flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
Although we're not sure of the worst case, what we're seeing now is
outside of my comfort zone, personally.

For me, the class of problems in #3 are my area of greatest concern, as
there are many more RFC3168 bottlenecks deployed today than just a few
years ago
(https://github.com/heistp/l4s-tests/#deployments-of-fq_codel). #2 is
also important to me, as I trust we're not trying to introduce a so-
called "fast lane" for certain traffic.

Although I won't be able to support a WGLC until I see tested code that
addresses the issues, I do want to support the WG along its present
path to a conclusion...

Pete

On Sun, 2020-11-15 at 01:26 +0000, Black, David wrote:
> First, I think Pete has pretty clearly established that TCP Prague is
> research, and hence (IMHO) TCP Prague ought to be headed for ICCRG as
> its primary forum rather than TSVWG.
> 
> To date, the 'Prague L4S Requirements' (Appendix A of draft-ietf-
> tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague. 
> That association ought to be teased apart so that the resulting L4S
> scalable congestion control requirements provide a reasonable design
> space that can include a number of other congestion control designs -
> in addition to what's been discussed, e.g., SCReAM, it would be
> useful to better understand what it would take for implementations of
> protocols such as DCTCP and BBR (e.g., for QUIC) to meet those
> requirements.
> 
> My overall take on the requirements is that in 20/20 hindsight, some
> of them were overly optimistic, and hence need to be backed off/toned
> down/broadened to encompass what is reasonable in "running code" well
> beyond TCP Prague. That sort of collision between interesting ideas
> and network realities is not an unheard-of scenario in IETF, so I
> hesitate to view the need for changes to these requirements as
> evidence that the original ideas were inherently defective, as I've
> seen far more dramatic changes, e.g., some number of years ago, the
> first design of iSCSI login was elegant ... and resulted in
> implementations that did not interoperate, resulting in a complete
> redesign.
> 
> Thanks, --David
> 
> > a) Dualpi2/TCP Prague are no way ready for deployment, but are
> > basically still in
> > the toy stage
> >

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 15 Nov 2020 13:06:40 +0000 (UTC)

Hi Pete,
 Interesting data. 
Maybe I've missed it, but I don't see whether these tests were run against a version of TCP-Prague having the RTT-independence workaround mentioned during interim-2020-tsvwg-01 [1], and if so what its mode was. Can you clarify that?

Thanks,
Alex

[1] Slide 16 of https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/materials/slides-interim-2020-tsvwg-01-sessa-l4s-tcp-prague-update-00.pdf

 On Thursday, November 12, 2020, 9:03:17 PM GMT, Pete Heist <[email-masked]> wrote: 

Hi,

Regards,
Pete

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 15 Nov 2020 15:23:32 +0200

> On 15 Nov, 2020, at 3:06 pm, [email-masked] wrote:
> 

We're using the latest version of the L4S kernel, with any TCP Prague-specific parameters left at the default setting.  Looking at the code, the default appears to disable the 15ms RTT bias embedded in TCP Prague, which as you note was proposed to counter the bias imposed by DualPI2.

But even if that were enabled, it's a nasty workaround which addresses only one aspect of the unfairness we show in our results.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 15 Nov 2020 14:29:07 +0100

Hi Alex,

more below in-line, pre-fixed [SM].

> On Nov 15, 2020, at 14:06, [email-masked] wrote:
> 
> Hi Pete,
>  Interesting data. 
> Maybe I've missed it, but I don't see whether these tests were run against a version of TCP-Prague having the RTT-independence workaround

	[SM] Which "RTT-independence workaround" are you talking about here? All that is implemented in TCP Prague is a crutch around the issue that DualPI2 catastrophically fails at its core job, with short RTT. That is a L4S/DualPi2-unique problem that is orthogonal to reduction of RTT-bias as mentioned in the Prague requirements. Just because Koen and Bob, framed this as addressing "RTT bias" does not make that assertion a fact...

> mentioned during interim-2020-tsvwg-01 [1], and if so what its mode was. Can you clarify that?

	The tests are well documented and use the latest Linux kernel from team L4S' repository https://github.com/L4STeam/linux/tree/testing (which happens to be the most recently changed branch as well as the default branch). This therefore refelects team L4S recommended TCP Prague configuration.

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 15 Nov 2020 19:13:41 +0200

> On 15 Nov, 2020, at 3:26 am, Black, David <[email-masked]> wrote:
> 

Along these lines, let's look at the components of L4S:

1: The "architecture" involves a qualitative, revolutionary alteration to the meaning of a CE mark (relative to that established in RFC-3168 and revised quantitatively in RFC-8511).  In essence, it introduces AIAD semantics where AIMD was previously expected.

2: A classifier codepoint - ECT(1) - asserts to the network that this alternative CE semantic is in force on a given flow.  Important to note that existing networks do not expect this signal, and are thus incapable of correctly interpreting it; they assume that an ECT(1) packet should be treated the same as an ECT(0) packet, because that is what RFC-3168 specifies.

3: AccECN is a TCP extension that provides precise, reliable feedback of the density of CE marks on TCP flows.  This is deemed necessary because the AIAD steady-state has multiple CE marks per RTT, rather than multiple RTTs per CE mark in the AIMD steady-state.

4: TCP Prague, as the reference congestion-control algorithm, implements the classifier codepoint at origin when AccECN is negotiated, and exhibits a DCTCP-like (AIAD) response to CE feedback, and a NewReno-like (AIMD) response to packet loss.

5: DualPI2, as the reference L4S-aware AQM and qdisc, uses the classifier codepoint to implement what amounts to a PHB, differentiating L4S traffic from "classic" conventional traffic.  This PHB both implements the required change in AQM behaviour and provides an effective prioritisation of L4S traffic.  According to the designers, it explicitly avoids maintaining per-flow state.

6: Operational guidelines, which - as far as we can tell - boil down to "run L4S only across specially prepared networks and functionally isolate it from the conventional Internet".

The collection of components above, in the forms thus far demonstrated, fail to achieve both #4 and #5 of the "Prague requirements", namely "Fall back to Reno-friendly congestion control on classic ECN bottlenecks" and "Reduce RTT dependence".  We demonstrated that through test results some months ago, and again in the test results released just now.  We have not explicitly tested for achievement of Requirement #6, since we believe that to be less vital to interoperability with the existing Internet.

If we're going to get specific, I believe the failure to achieve Requirement #4 is the fault of Component #1 in the above list, while the failure to achieve Requirement #5 is the fault of Component #5.  Meanwhile, Component #2 is blocking progress on an alternative design which has a better chance of success.  I think these conclusions are supported by the failure of attempts to address these shortcomings by modifying Component #4, and by the introduction of Component #6 after the discussion in March rather than proceeding with technical refinements to the other components.

> My overall take on the requirements is that in 20/20 hindsight, some of them were overly optimistic, and hence need to be backed off/toned down/broadened to encompass what is reasonable in "running…

From a systems engineering perspective, it is possible to both see how the design of L4S was developed iteratively starting from DCTCP, and to recognise that this was an inherently flawed approach.  It was well known from the start that DCTCP did not interoperate with AIMD transports at common bottlenecks, but after the dust has cleared, the basic mechanism of DCTCP is the one thing that has been held constant in L4S' design - almost dogmatically, in fact.

The result is a system that, like DCTCP, can only be deployed into networks that have been specially prepared for it, and fails to interoperate with AIMD traffic without specific assistance from L4S-aware in-network components.  That is not really the sort of progress that justifies so much of the WG's time and attention, regardless of how desirable a move towards high-fidelity congestion control is.

So I fear that a substantial redesign of L4S is indeed called for, if it is to actually succeed.  This has to start with component #1 in the above list - the redefinition of the semantics of a CE mark.  Restore CE to RFC-8511 semantics, and most of the severe interoperability problems we have ben talking about for the past two years simply disappear; only networks trying to take advantage of the new behaviour need even be aware that it exists.

I need hardly remind the WG that I and my colleagues have been presenting an alternative design which does exactly that.  We'll happy to discuss it further as and when appropriate.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 16 Nov 2020 20:43:47 +0000

Hi Pete,

Thanks for the tests. They do confirm our test results from many years ago now, and are the reason why related Prague requirements are in the L4S-ID draft. I guess they are a good introduction for people recently joining the discussions and a reminder for the rest that it is important to take those conditions into account when designing and configuring both the end-host and the network mechanisms. 

May I also remind that the Linux Prague code is research driven, and focused only on de-risking the bigger safety challenges. Most of the improvements (except low latency) were not prioritized up to now, as we are sure some are low hanging fruit and others with the right incentives will want to spend time on the harder ones. Most of the commercial parties with an expected interest and benefit of an L4S CC have been reasonable quite, I hope because they are confident that the network drafts and the mandatory Prague requirements and proposed improvements are feasible. I guess some are also less keen on sharing their efforts with competition and others might be waiting for real deployments before getting really into action. If there are concerns from their side, of course we would like to hear those too. The only concern I heard up to now is that all this takes too long before they can start the experiment. They want asap real world deployment...

>> (https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by starting marking in the L queue later? Doing so may also improve RTT fairness for Prague flows.

Indeed, the solution is to make sure that your L4S threshold configuration is in line with your network capabilities. If you know that certain network elements aggregate and burst packets in 4ms chunks, then it makes no sense at all to mark packets in that node at 1ms, nor in subsequent network nodes, with similar bottleneck capacities. We tried to address this by explaining this in section 6.3 of the L4S Architecture draft: https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-08#page-20. Is this sufficient to cover this issue?

>> 2) For the reported network bias, can this be fixed in the DualPI2 and Prague implementations? One chart which we didn't reference in our writeup illustrates the consistent bias for Prague vs CUBIC.

Since an important design goal of L4S is that it also supports network nodes that don't identify individual flows. To compensate individual flow's RTT you need to know them first. To solve this anyways, it would need a per packet indication of the RTT (ref past proposals like RCP and XCP). If we would have space to put this info in the header, we wouldn't be discussing about this single available ECN bit, right? Therefor the End-System is the right place to handle this, and why the RTT-independent requirement is important. The current implementation has a non-Prague-compliant default (not RTT-independent), which we need to fix clearly. It would be good to agree on a reasonable compromise of a default f(RTT) correction function. We also implemented a gradual correction over time, so a new (short) flow would not be delayed, and will be responsive, only if it lasts for a while it will gradually converge to the RTT-independent rate. This could be in the range of the convergence time of the typical-RTT flows.

Feel free to experiment with the available options.

>> * To not consider a 16:1 throughput imbalance between L4S and non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
>> Although we're not sure of the worst case, what we're seeing now is outside of my comfort zone, personally.

Actually, I'm also not happy with this strong RTT dependency. You probably will see similar unfairness cases if you use under these conditions Reno on a CoDel AQM with a target of 5ms. Cubic compensates performance for the longer RTT flows, Prague (with the right settings) compensates mandatory for the smaller RTT. Clearly a production congestion control is likely to do both, or fall back to Classic Cubic behavior whenever the RTT is too long for interactive services.

So the exact out-of-our-confort-zone exists today already, but there are solutions available, just to be integrated.

>> 3) The issues that arise due to the redefinition of CE may be the hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), 
>> and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes).

This is fortunately showing the flaws of CoDel and on top of that the lack of the CoDel implementation to respond with drop when ECN flows get into overload. Even more problematic for Cubic on CoDel (deployed today), your results show it is also bad (more than 2 seconds latency for longer than 10 seconds when throughput reduces from 50Mbps to 1Mbps on an 80ms base RTT path). When compared under the same conditions with Cubic on PI2 you can definitely see how it can be improved using an appropriate AQM (a very short peak of I guess less than 100ms extra latency). Hopefully these CoDel instances can be fixed and can support the ECN threshold for L4S instead (and maybe use PI2 instead of CoDel for Classic). On the other hand these extremes would be also easy cases for both Classic and L4S CCs to detect and fall back on a delay based CC when latencies are that bad (falling back to Classic ECN or Classic drop would even not help to avoid the CoDel problem). But I doubt the Linux Cubic developers would be happy to extend Cubic to avoid CoDel flaws??

>> Although I won't be able to support a WGLC until I see tested code that addresses the issues, I do want to support the WG along its present path to a conclusion...

Try switching on the RTT-independent settings of Prague, and I assume it is a matter of just coding a fallback to Cubic if your detected RTT is too big. So as far as I'm concerned, we do have running code! 

I'll try to get the defaults right. I assume f(RTT)=max(25ms, RTT) and a gradual EWMA convergence reaching the 25ms behavior after about 10 seconds would be good defaults? If people have suggestions or preferences for other values, then let me know. If people have time and want to implement the Cubic fallback (or even better longer RTT-independence), then let me know too.

Regards,
Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Pete Heist
Sent: Sunday, November 15, 2020 12:42 PM
To: Black, David <[email-masked]>; Sebastian Moeller <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

Firstly, there was a new test result added for tunnels yesterday, but I'll put that in a separate thread so it isn't buried.

Beyond that, I think it would be productive to try to figure out what might be done to solve the reported issues, limiting that exploration for now to the confines of the L4S design. We'd like to support forward progress while facing the issues head on if possible. I'll take a shot here as tersely as I can:

1) Starting with an easier one, can the reported intolerance to bursts
(https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed by starting marking in the L queue later? Doing so may also improve RTT fairness for Prague flows.

2) For the reported network bias, can this be fixed in the DualPI2 and Prague implementations? One chart which we didn't reference in our writeup illustrates the consistent bias for Prague vs CUBIC.
Essentially, Prague wins until there is an RTT imbalance of 20ms/80ms:

3) The issues that arise due to the redefinition of CE may be the hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes). 

* RFC3168 bottleneck detection in the endpoints. However, we went through a round of testing earlier this year that showed accurate bottleneck detection is likely to be difficult with jitter and cross- flow traffic. It's disabled at present in the reference implementation.

* L4S operational guidance for network operators. So far I haven't had time to review this, but I may have some feedback at least in the area of testing. I suspect the effectiveness of guidance will be influenced by human factors.

* To not consider a 16:1 throughput imbalance between L4S and non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent tests.
Although we're not sure of the worst case, what we're seeing now is outside of my comfort zone, personally.

For me, the class of problems in #3 are my area of greatest concern, as there are many more RFC3168 bottlenecks deployed today than just a few years ago (https://github.com/heistp/l4s-tests/#deployments-of-fq_codel). #2 is also important to me, as I trust we're not trying to introduce a so- called "fast lane" for certain traffic.

Although I won't be able to support a WGLC until I see tested code that addresses the issues, I do want to support the WG along its present path to a conclusion...

Pete

On Sun, 2020-11-15 at 01:26 +0000, Black, David wrote:
> tsvwg-ecn-l4s-id) have been strongly associated with TCP Prague. That 
> association ought to be teased apart so that the resulting L4S 
> in addition to what's been discussed, e.g., SCReAM, it would be useful 
> to better understand what it would take for implementations of 
> > great data. IMHO Especially the fact that short-RTT Prague will 
> > severely outcompete long-RTT Prague way more than the traditional 
> > dumb FIFO will, strongly supports  the following hypotheses I have 
> > been posting
> > before:
> > 
> > basically still in the toy stage
> > 
> > b) all that L4S will effectively do, be it by intent or simply by 
> > mis-design, is built a "fast lane" for short RTT low hop count 
> > traffic. Given all the hype
> > (still!) in the L4S
> > requires active cooperation from the leaf ISP (if they keep their 
> > bottlenecks at FIFO, I see no compelling reason for TCP Prague at 
> > all), which will require SLAs between CDNs and ISPs.
> > 
> > c) too little, too late. Really only modest gains in RTT (compared 
> > to best of class AQMs like fq_codel or cake) and noticeable 
> > regressions in sharing behavior both between different CCs and flows 
> > of different RTTs (and that compared to dumb queue "management" with 
> > a simple FIFO).
> > 
> > 
> > 
> > Also quite sobering, that it AGAIN was not team L4S bringing real 
> > data to the table to support their claims and promises. Then again 
> > looking at the RTT fairness for Prague versus Prague under DualQ, I 
> > can understand why team L4S stayed mumm, and rather argued for 
> > allowing unfettered self-mutilation of L4S compliant transport 
> > protocols in regards to RTT fairness:
> > 
> > "So there is no need  to mandate that an L4S implementer does no 
> > harm to themselves, which window-based congestion controls tend to 
> > do at higher RTT.
> > Of course, this doesn't preclude implementers reducing or 
> > eliminating RTT bias for larger than typical RTTs, but it removes 
> > any requirement to do so."
> > 
> > After years of advertising on increased RTT independence (in spite 
> > of the data showing that the proposed combination of DualQ and TCP 
> > Prague actually increases RTT bias), team L4S in the last minute no 
> > less, decides to do a 180 and just change the requirements to allow 
> > for the rather unhealthy behavior demonstrated for L4S...
> > With the current enhanced RTT bias, who in their right mind is going 
> > to use TCP Prague; which currently is the only transport, that at 
> > least attempted to tackle all the "requirements" L4S poses for 
> > transports that want to use the
> > ECT(1) express
> > way? Then again, since none of the network elements are designed and 
> > required to actually check and enforce these requirements, calling 
> > these "requirements" is a bit of stretch anyway, maybe they should 
> > be changed from MUST requirements to COULD musings....
> > 
> > 
> > Best Regards
> >         Sebastian
> > 
> > 
> > > These tests cover two-flow fairness in several qdiscs with 
> > > different path RTTs per-flow, and transient behavior in fq_codel 
> > > queues.
> > > 
> > > The repo above contains a walk-through of the key findings, and 
> > > links to more results in the Appendix.
> > > 
> > > Regards,
> > > Pete
> > > 
> > > 
>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 16 Nov 2020 23:13:59 +0200

> On 16 Nov, 2020, at 10:43 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

It's not running code until it actually *is* implemented, and we can run tests showing whether it actually works or not.  We should not need to change default settings in order to show safety in common network topologies.  It must also be documented in such a way that future implementors are likely to produce a similarly high quality result.

Frankly, it is not a very high bar we're holding up here.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 16 Nov 2020 22:37:20 +0100

Hi Koen,

> On Nov 16, 2020, at 21:43, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Pete,
> 

	[SM] Please supply a link to your results from many years ago showing how the current state of TCP Prague behaves? 

> 
> May I also remind that the Linux Prague code is research driven, and focused only on de-risking the bigger safety challenges.

	[SM] Papering over obvious flaws in the core L4S design at a place that can be easily abandoned later would be a much less friendly interpretation.

> Most of the improvements (except low latency) were not prioritized up to now,

	[SM] Mmmh, did you not just argue you focussed on safety challenges first?

> as we are sure some are low hanging fruit and others with the right incentives will want to spend time on the harder ones.

	[SM] The fact that TCP Prague has terrible performance over FIFO-bottleneck long haul paths will make it very unlikely that someone will be incentivized to continue TCP Prague development after you abandon it.

> Most of the commercial parties with an expected interest and benefit of an L4S CC have been reasonable quite,

	[SM] Specifically none responded for my repeated begging for presenting data to demonstrate that my fears are wrong. My conclusion from that lack of response is that either nobody cares for L4S or worse nobody bothered to independently confirm its suitability and safety.

> I hope because they are confident that the network drafts and the mandatory Prague requirements and proposed improvements are feasible. I guess some are also less keen on sharing their efforts with competition and others might be waiting for real deployments before getting really into action. If there are concerns from their side, of course we would like to hear those too. The only concern I heard up to now is that all this takes too long before they can start the experiment. They want asap real world deployment...

	[SM] There is zero rationale why a controlled experiment could not have already been performed over the existing internet. An experimental RFC is not a requirement to conduct an experiment. What you call an experiment, really is an exercise in deployment.

> Indeed, the solution is to make sure that your L4S threshold configuration is in line with your network capabilities. If you know that certain network elements aggregate and burst packets in 4ms ch…

	[SM] What happened to the disire for AQM's to be auto-tuning to avoid the biggest stumbling block for their deployment? Think https://tools.ietf.org/html/rfc7567#page-18 "4.3.  AQM Algorithm Deployment SHOULD NOT Require Operational Tuning".

> Since an important design goal of L4S is that it also supports network nodes that don't identify individual flows.

	[SM] Well, it seems more and more obvious that that design decision puts L4S at odds with achieving its goals....

> To compensate individual flow's RTT you need to know them first. To solve this anyways, it would need a per packet indication of the RTT (ref past proposals like RCP and XCP). If we would have space to put this info in the header, we wouldn't be discussing about this single available ECN bit, right? Therefor the End-System is the right place to handle this, and why the RTT-independent requirement is important.

	[SM] Please keep your story straight, you just proposed to considerably water down the RTT de-bias requirement such, that it is not really applicable to arbitrary paths anymore, only those that the AQM deployer cares for, like a fast-lane for short RTTs.

> The current implementation has a non-Prague-compliant default (not RTT-independent), which we need to fix clearly. It would be good to agree on a reasonable compromise of a default f(RTT) correction function. We also implemented a gradual correction over time, so a new (short) flow would not be delayed, and will be responsive, only if it lasts for a while it will gradually converge to the RTT-independent rate. This could be in the range of the convergence time of the typical-RTT flows.

	[SM] As I have shown with Pete's data in my response to Bob your problem is not the to be expected RTT dpendence of the thoughput of competing TCP flows, but the combination of DualPI2 and TCP Prague actually makes the situation much worse, see (repeated from my other mail):

Let's look at some data, shall we?
This is a plot showing the rates achieved by two CUBIC flows (80ms pRTT and 20 ms pRTT) in a bottleneck with a 200 packet FIFO at an egress rate of 50 Mbps
http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-pfifo_200_-10Mbit-80ms-20ms_tcp_delivery_with_rtt.svg
Both flows actually get more or less the same thoughput.
	Now adding an AQM we expect things to get worse, especially single queue AQMs are known to do that, so here are "same" two CUBIC flows under the same conditions only the bottleneck now operates DualPI2 but since the flows are not ECT(1) they are handled by the PIE instance responsible for the shallow queue:
http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-dualpi2-50Mbit-80ms-20ms_tcp_delivery_with_rtt.svg
Here eye-balling it tells us that the 80ms pRTT flow only gets around half the thoughput of the 20ms one, not beautiful, but certainly acceptable by the concurrent decrease of the flows' queueing delay (roughly a reduction of 230ms per flow).
	But now, if we look at the same test with two TCP Prague flows running through the shallow-queue's modified? PIE instance we see a throughput imbalance of around 45:5 Mbps or 9:1.
	So from rough parity we went down to 9:1, that IMHO is a clear design issue in L4S because who in their right mind is going to use TCP Prague if that basically means such steep sacrifices in throughput through longer paths? Really, it is not that 80ms is an extreme RTT that is irrelevant, think trans Atlantic links.

	Trying to frame this as somehow reducing RTT bias seems simply counterfactual, L4S is making the situation worse (while using wording that makes the naive reader think it helps solving the issue).

> Actually, I'm also not happy with this strong RTT dependency. You probably will see similar unfairness cases if you use under these conditions Reno on a CoDel AQM with a target of 5ms.

	[SM] Have a look at the fq_codel:

Here is how the dual CUBIC test looks in fq_codel:
http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-fq_codel-50Mbit-80ms-20ms_tcp_delivery_with_rtt.svg
Way less than the factor two throughput difference as seen in your PIE example. Judging from Figure 6 in Høiland-Jørgensen, Toke, Per Hurtig, and Anna Brunstrom. ‘The Good, the Bad and the WiFi: Modern AQMs in a Residential Setting’. Computer Networks 89 (4 October 2015): 90–106. https://doi.org/10.1016/j.comnet.2015.07.014. single-queue CoDel will behave quite close to songle-queue PIE.

Your intuition seems to be wrong.

> Cubic compensates performance for the longer RTT flows, Prague (with the right settings) compensates mandatory for the smaller RTT. Clearly a production congestion control is likely to do both, or fall back to Classic Cubic behavior whenever the RTT is too long for interactive services.

	[SM] Well, TCP Prague is your vehicle to demonstrate to the world that your claims are achievable, so go and achieve them. Until that point all siuch claims are really just predictions and promises

> This is fortunately showing the flaws of CoDel and on top of that the lack of the CoDel implementation to respond with drop when ECN flows get into overload. Even more problematic for Cubic on CoDe…

	[SM] Too late, codel is out in the wild, no amount of whining is going to change that, if you design a protocol for the existing internet you need to accept that fact and adapt to it.

> Try switching on the RTT-independent settings of Prague, and I assume it is a matter of just coding a fallback to Cubic if your detected RTT is too big. So as far as I'm concerned, we do have runni…

	[SM] The idea behind this requirement is to have code that can be executed (runs) but also solves the problem at hand. TCP Prage as from the default repository might do the former, but as Pete showed fails at the latter. 

> I'll try to get the defaults right. I assume f(RTT)=max(25ms, RTT) and a gradual EWMA convergence reaching the 25ms behavior after about 10 seconds would be good defaults? If people have suggestion…

	[SM] Think auto-tuning, what you do is window dressing, by modifying your toggles to fit a specific individal benchmark you look bad at, but unless TCP Prague does that adaptively roughly correct, this is no viable solution, that can be used as argument that the other deeply flawed L4S components are safe to deploy.

Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 16 Nov 2020 22:03:20 +0000

Hi Sebastian,

>	[SM] Have a look at the fq_codel:

I really meant "CoDel", not "FQ_CoDel".

> But I doubt the Linux Cubic developers would be happy to extend Cubic to avoid CoDel flaws??
>

So because CoDel it is deployed and you get these bad results today with existing Cubic deployments already, it is not a safety issue, but whining is appropriate???? It clearly shows that this "bar" is kept very high for some and is laying at the ground for others 😉.

Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Monday, November 16, 2020 10:37 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Pete Heist <[email-masked]>; Black, David <[email-masked]>; tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

Hi Koen,

This is a plot showing the rates achieved by two CUBIC flows (80ms pRTT and 20 ms pRTT) in a bottleneck with a 200 packet FIFO at an egress rate of 50 Mbps http://sce.dnsmgr.net/_archive/l4s-2020-11-07T203721-s1-s4/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-cubic-pfifo_200_-10Mbit-80ms-20ms_tcp_delivery_with_rtt.svg

>>> hardest. This includes the domination of L4S over non-L4S flows in the same RFC3168 queue (see the issue with tunnels reported above for a common path to that), and the intra-flow latency spikes for L4S flows (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes).
> 

Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Nov 2020 07:30:37 +0100

Hi Koen.

More below in-line, prefixed [SM]. But first I take your lack of response to my description of the causes of RTT bias of a lack of meaningful counterarguments.

On 16 November 2020 23:03:20 CET, "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]> wrote:
>Hi Sebastian,
>
>>> Actually, I'm also not happy with this strong RTT dependency. You
>probably will see similar unfairness cases if you use under these
>conditions Reno on a CoDel AQM with a target of 5ms.
>>

        [SM] If you read my full section you will find a prediction for CoDel based on a published paper with real data. Why do you bother to wrote a response but not to actually read what you are responding to?

>>	[SM] Too late, codel is out in the wild, no amount of whining is
>going to change that, if you design a protocol for the existing
>internet you need to accept that fact and adapt to it.
>
>So because CoDel it is deployed and you get these bad results today
>with existing Cubic deployments already, it is not a safety issue, but
>whining is appropriate???? It clearly shows that this "bar" is kept
>very high for some and is laying at the ground for others 😉.

        [SM] I am confident that the bulk of deployed CoDel instances come as sub-AQMs inside of FQ_CoDel or cake. And these are pretty good at offering order of magnitude lower queuing delay than FIFOs, without irritating fairness issues. Something L4S still has to demonstrate. But let's assume for a moment your sentiment would be correct and CoDel would be a broken abomination that should never have received RFC status; how is that an argument for repeating that mistake with L4S and giving it a pass even though it fails to meet its own promises, instead of learning from the CoDel mistake and making sure things work properly before blessing them with RFC status? This is a discussion I had several times with my kids and it is a bit puzzling that I need to repeat it here....

Best Regards
        Sebastian

>> On Nov 16, 2020, at 21:43, De Schepper, Koen (Nokia - BE/Antwerp)
><[email-masked]> wrote:
>> 
>> Hi Pete,
>> 
>> Thanks for the tests. They do confirm our test results from many
>years ago now, and are the reason why related Prague requirements are
>in the L4S-ID draft. I guess they are a good introduction for people
>recently joining the discussions and a reminder for the rest that it is
>important to take those conditions into account when designing and
>configuring both the end-host and the network mechanisms. 
>
>	[SM] Please supply a link to your results from many years ago showing
>how the current state of TCP Prague behaves? 
>
>
>> 
>> May I also remind that the Linux Prague code is research driven, and
>focused only on de-risking the bigger safety challenges.
>
>
>	[SM] Papering over obvious flaws in the core L4S design at a place
>that can be easily abandoned later would be a much less friendly
>interpretation.
>
>
>> Most of the improvements (except low latency) were not prioritized up
>
>> to now,
>
>> as we are sure some are low hanging fruit and others with the right
>incentives will want to spend time on the harder ones.
>
>	[SM] The fact that TCP Prague has terrible performance over
>FIFO-bottleneck long haul paths will make it very unlikely that someone
>will be incentivized to continue TCP Prague development after you
>abandon it.
>
>	[SM] Specifically none responded for my repeated begging for
>presenting data to demonstrate that my fears are wrong. My conclusion
>from that lack of response is that either nobody cares for L4S or worse
>nobody bothered to independently confirm its suitability and safety.
>
>
>> I hope because they are confident that the network drafts and the
>mandatory Prague requirements and proposed improvements are feasible. I
>guess some are also less keen on sharing their efforts with competition
>and others might be waiting for real deployments before getting really
>into action. If there are concerns from their side, of course we would
>like to hear those too. The only concern I heard up to now is that all
>this takes too long before they can start the experiment. They want
>asap real world deployment...
>
>	[SM] There is zero rationale why a controlled experiment could not
>have already been performed over the existing internet. An experimental
>RFC is not a requirement to conduct an experiment. What you call an
>experiment, really is an exercise in deployment.
>
>
>> 
>>>> (https://github.com/heistp/l4s-tests/#burst-intolerance) be fixed
>by starting marking in the L queue later? Doing so may also improve RTT
>> Indeed, the solution is to make sure that your L4S threshold
>configuration is in line with your network capabilities. If you know
>that certain network elements aggregate and burst packets in 4ms
>chunks, then it makes no sense at all to mark packets in that node at
>1ms, nor in subsequent network nodes, with similar bottleneck
>capacities. We tried to address this by explaining this in section 6.3
>of the L4S Architecture draft:
>https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-08#page-20. Is
>this sufficient to cover this issue?
>
>	[SM] What happened to the disire for AQM's to be auto-tuning to avoid
>the biggest stumbling block for their deployment? Think
>https://tools.ietf.org/html/rfc7567#page-18 "4.3.  AQM Algorithm
>Deployment SHOULD NOT Require Operational Tuning".
>
>
>
>> 
>>>> 2) For the reported network bias, can this be fixed in the DualPI2
>and Prague implementations? One chart which we didn't reference in our
>> Since an important design goal of L4S is that it also supports
>network nodes that don't identify individual flows.
>
>	[SM] Well, it seems more and more obvious that that design decision
>puts L4S at odds with achieving its goals....
>
>
>> To compensate individual flow's RTT you need to know them first. To
>solve this anyways, it would need a per packet indication of the RTT
>(ref past proposals like RCP and XCP). If we would have space to put
>this info in the header, we wouldn't be discussing about this single
>available ECN bit, right? Therefor the End-System is the right place to
>handle this, and why the RTT-independent requirement is important.
>
>	[SM] Please keep your story straight, you just proposed to
>considerably water down the RTT de-bias requirement such, that it is
>not really applicable to arbitrary paths anymore, only those that the
>AQM deployer cares for, like a fast-lane for short RTTs.
>
>
>
>> The current implementation has a non-Prague-compliant default (not
>RTT-independent), which we need to fix clearly. It would be good to
>agree on a reasonable compromise of a default f(RTT) correction
>function. We also implemented a gradual correction over time, so a new
>(short) flow would not be delayed, and will be responsive, only if it
>lasts for a while it will gradually converge to the RTT-independent
>rate. This could be in the range of the convergence time of the
>typical-RTT flows.
>
>	[SM] As I have shown with Pete's data in my response to Bob your
>problem is not the to be expected RTT dpendence of the thoughput of
>competing TCP flows, but the combination of DualPI2 and TCP Prague
>actually makes the situation much worse, see (repeated from my other
>mail):
>
>This is a plot showing the rates achieved by two CUBIC flows (80ms pRTT
>and 20 ms pRTT) in a bottleneck with a 200 packet FIFO at an egress
>rate of 50 Mbps
>	Now adding an AQM we expect things to get worse, especially single
>queue AQMs are known to do that, so here are "same" two CUBIC flows
>under the same conditions only the bottleneck now operates DualPI2 but
>since the flows are not ECT(1) they are handled by the PIE instance
>responsible for the shallow queue:
>Here eye-balling it tells us that the 80ms pRTT flow only gets around
>half the thoughput of the 20ms one, not beautiful, but certainly
>acceptable by the concurrent decrease of the flows' queueing delay
>(roughly a reduction of 230ms per flow).
>	But now, if we look at the same test with two TCP Prague flows running
>through the shallow-queue's modified? PIE instance we see a throughput
>imbalance of around 45:5 Mbps or 9:1.
>	So from rough parity we went down to 9:1, that IMHO is a clear design
>issue in L4S because who in their right mind is going to use TCP Prague
>if that basically means such steep sacrifices in throughput through
>longer paths? Really, it is not that 80ms is an extreme RTT that is
>irrelevant, think trans Atlantic links.
>
>
>
>	Trying to frame this as somehow reducing RTT bias seems simply
>counterfactual, L4S is making the situation worse (while using wording
>that makes the naive reader think it helps solving the issue).
>
>
>> 
>>>> * To not consider a 16:1 throughput imbalance between L4S and
>non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our recent
>tests.
>> Actually, I'm also not happy with this strong RTT dependency. You
>probably will see similar unfairness cases if you use under these
>conditions Reno on a CoDel AQM with a target of 5ms.
>
>Way less than the factor two throughput difference as seen in your PIE
>example. Judging from Figure 6 in Høiland-Jørgensen, Toke, Per Hurtig,
>and Anna Brunstrom. ‘The Good, the Bad and the WiFi: Modern AQMs in a
>Residential Setting’. Computer Networks 89 (4 October 2015): 90–106.
>https://doi.org/10.1016/j.comnet.2015.07.014. single-queue CoDel will
>behave quite close to songle-queue PIE.
>
>
>> Cubic compensates performance for the longer RTT flows, Prague (with
>the right settings) compensates mandatory for the smaller RTT. Clearly
>a production congestion control is likely to do both, or fall back to
>Classic Cubic behavior whenever the RTT is too long for interactive
>services.
>
>	[SM] Well, TCP Prague is your vehicle to demonstrate to the world that
>your claims are achievable, so go and achieve them. Until that point
>all siuch claims are really just predictions and promises
>
>> 
>> So the exact out-of-our-confort-zone exists today already, but there
>are solutions available, just to be integrated.
>> 
>>>> hardest. This includes the domination of L4S over non-L4S flows in
>the same RFC3168 queue (see the issue with tunnels reported above for a
>> This is fortunately showing the flaws of CoDel and on top of that the
>lack of the CoDel implementation to respond with drop when ECN flows
>get into overload. Even more problematic for Cubic on CoDel (deployed
>today), your results show it is also bad (more than 2 seconds latency
>for longer than 10 seconds when throughput reduces from 50Mbps to 1Mbps
>on an 80ms base RTT path). When compared under the same conditions with
>Cubic on PI2 you can definitely see how it can be improved using an
>appropriate AQM (a very short peak of I guess less than 100ms extra
>latency). Hopefully these CoDel instances can be fixed and can support
>the ECN threshold for L4S instead (and maybe use PI2 instead of CoDel
>for Classic). On the other hand these extremes would be also easy cases
>for both Classic and L4S CCs to detect and fall back on a delay based
>CC when latencies are that bad (falling back to Classic ECN or Classic
>drop would even not help to avoid the CoDel problem). But I doubt the
>Linux Cubic developers would be happy to extend Cubic to avoid CoDel
>flaws??
>
>	[SM] Too late, codel is out in the wild, no amount of whining is going
>to change that, if you design a protocol for the existing internet you
>need to accept that fact and adapt to it.
>
>
>> 
>>>> Although I won't be able to support a WGLC until I see tested code
>that addresses the issues, I do want to support the WG along its
>present path to a conclusion...
>> 
>> Try switching on the RTT-independent settings of Prague, and I assume
>it is a matter of just coding a fallback to Cubic if your detected RTT
>is too big. So as far as I'm concerned, we do have running code! 
>
>	[SM] The idea behind this requirement is to have code that can be
>executed (runs) but also solves the problem at hand. TCP Prage as from
>the default repository might do the former, but as Pete showed fails at
>the latter. 
>
>
>> 
>> I'll try to get the defaults right. I assume f(RTT)=max(25ms, RTT)
>and a gradual EWMA convergence reaching the 25ms behavior after about
>10 seconds would be good defaults? If people have suggestions or
>preferences for other values, then let me know. If people have time and
>want to implement the Cubic fallback (or even better longer
>RTT-independence), then let me know too.
>
>	[SM] Think auto-tuning, what you do is window dressing, by modifying
>your toggles to fit a specific individal benchmark you look bad at, but
>unless TCP Prague does that adaptively roughly correct, this is no
>viable solution, that can be used as argument that the other deeply
>flawed L4S components are safe to deploy.
>
>
>Regards
>	Sebastian
>
>
>> 
>> Regards,
>> Koen.
>> 
>> 
>> 2) For the reported network bias, can this be fixed in the DualPI2
>and Prague implementations? One chart which we didn't reference in our
>> * L4S operational guidance for network operators. So far I haven't
>had time to review this, but I may have some feedback at least in the
>area of testing. I suspect the effectiveness of guidance will be
>influenced by human factors.
>> 
>> For me, the class of problems in #3 are my area of greatest concern,
>as there are many more RFC3168 bottlenecks deployed today than just a
>few years ago
>> Although I won't be able to support a WGLC until I see tested code
>that addresses the issues, I do want to support the WG along its
>present path to a conclusion...
>> 
>> Pete
>> 
>That 
>>>>> The results raise some concerns about a bias in favor of L4S flows
>
>>>>> in

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Lars Eggert <[email-masked]>
Date: Tue, 17 Nov 2020 10:15:30 +0200

Hi,

> Frankly, it is not a very high bar we're holding up here.

FWIW, I agree with Jonathan here.

Independent verification of experimental results is an essential part of the scientific process in general. And for TCP in particular, it was used to great effect during the "highspeed TCP wars" of the early 2000s, when the proponents of BIC/CUBIC, Compound TCP, HSTCP, etc. all tried to recreate each others experiments and results. Many bugs and unstated assumptions were identified as a result, and the individual algorithms all benefitted immensely. I'm hoping that we can use this approach to make progress in this (apparently endless) debate, as well.

A reference implementation for a standards proposal therefore really needs to implement that proposal accurately and in its latest instantiation. And if the code requires parameterization (esp. scenario-specific parameterization) beyond what the spec describes, then either the code or the spec are incomplete.

Thanks,
Lars

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Nov 2020 11:46:36 +0000

Hi. 
This is the RTT bias question that has been beating this email list for more
than a year by now.

I imagine that, as Prague is work in progress, then the original authors of
this mail thread could have considered to try out the different options that
address the RTT bias. Or at least consider to re-run the experiment instead
of digging down in long discussions on what is default and not. Prague is
work in progress after all, I don't think that this is unreasonable to ask
and it could have saved us a few roundtrips in in this mailthread ?

Besides this, I was of the impression that the congestion control discussion
would fit better in ICCRG?. 
And as RTT bias is not just an L4S problem then I guess it would me more
fruitful to contribute to work that tries to address it.
/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Lars Eggert
> Sent: den 17 november 2020 09:16
> To: Jonathan Morton <[email-masked]>
> > It's not running code until it actually *is* implemented, and we can run
tests
> showing whether it actually works or not.  We should not need to change
> default settings in order to show safety in common network topologies.  It
> must also be documented in such a way that future implementors are likely
to
> produce a similarly high quality result.
> >
> Independent verification of experimental results is an essential part of
the
> scientific process in general. And for TCP in particular, it was used to
great
> effect during the "highspeed TCP wars" of the early 2000s, when the
> proponents of BIC/CUBIC, Compound TCP, HSTCP, etc. all tried to recreate
> each others experiments and results. Many bugs and unstated assumptions
> were identified as a result, and the individual algorithms all benefitted
> immensely. I'm hoping that we can use this approach to make progress in
this
> (apparently endless) debate, as well.
> 
> A reference implementation for a standards proposal therefore really needs
> to implement that proposal accurately and in its latest instantiation. And
if the
> code requires parameterization (esp. scenario-specific parameterization)
> beyond what the spec describes, then either the code or the spec are
> incomplete.
> 
> Thanks,
> Lars

---

From: Lars Eggert <[email-masked]>
Date: Tue, 17 Nov 2020 14:32:32 +0200

Hi,

> and it could have saved us a few roundtrips in in this mailthread ?

I've long ago given up trying to follow the blow-by-blow of this overall discussion. And maybe that means I've disqualified myself from having an opinion. But I don't quite know what to make of your statement.

If "Prague" (I guess you mean a TCP variant taking advantage of L4S?) is a "work in progress", why are we then pushing towards finalizing the L4S part of the overall scheme?

Maybe I'm naive here, but I always thought the queuing scheme and a TCP variant for it would necessarily go together. That TCP variant might be an initial design that maybe doesn't take full advantage of all the benefits the queueing scheme might offer, but surely it should demonstrate some significant benefit without significant downsides?

And we can of course have a discussion on whether RTT unfairness (or any other property or lack of one) is a significant downside or not, but I don't think we've had that discussion?

Basically, I'm trying to re-engage here for a bit to see if there is *any* hope for this entire area of work, if I should just ignore it again or even start arguing we should abandon it all, given how wedged the discussion between the camps has been for a long time.

Thanks,
Lars

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Nov 2020 13:25:14 +0000

Hi

I see Prague as work on a congestion control algorithm that can exploit L4S.
Other work is done on e.g. BBRv2 that can use L4S as well, and then we have
SCReAM that is likely to be updated to make better use of L4S. Like with
other congestion control I don't foresee a one size fits all here, for
instance realtime media has different dynamics than bulk transfers
I see all these algorithms as work in progress and I don't see it explicit
in the charter that there must be a fully functional L4S capable congestion
control available. 
I see that the Prague work addresses many of the issues, among then the RTT
unfairness issue. It is still not perfect I guess but all this is a matter
of brain cycles in --> results out. Prague is an initial design and I
imagine that it will be improved over time. Also I see it likely that L4S
capable CCs can draw attention among academics in the future but for that to
happen we need to move forward to give the signal that we move forward, to
make it worthwhile to engage in the work. 

The RTT unfainess.. I believe that it has been discussed to death by now. To
be honest I don't believe that additional discussion will do anything more
than consume energy. Feel free to disagree.

/Ingemar

> From: Lars Eggert <[email-masked]>
> Sent: den 17 november 2020 13:33
> To: Ingemar Johansson S <[email-masked]>
> Cc: Jonathan Morton <[email-masked]>; tsvwg IETF list
> <[email-masked]>
> On 2020-11-17, at 13:46, Ingemar Johansson S
> <[email-masked]> wrote:
> > I imagine that, as Prague is work in progress, then the original
> > authors of this mail thread could have considered to try out the
> > different options that address the RTT bias. Or at least consider to
> > re-run the experiment instead of digging down in long discussions on
> > what is default and not. Prague is work in progress after all, I don't
> > think that this is unreasonable to ask and it could have saved us a few
> roundtrips in in this mailthread ?
> 
> I've long ago given up trying to follow the blow-by-blow of this overall
> discussion. And maybe that means I've disqualified myself from having an
> opinion. But I don't quite know what to make of your statement.
> 
> If "Prague" (I guess you mean a TCP variant taking advantage of L4S?) is a
> "work in progress", why are we then pushing towards finalizing the L4S
part of
> the overall scheme?
> 
> Maybe I'm naive here, but I always thought the queuing scheme and a TCP
> variant for it would necessarily go together. That TCP variant might be an
initial
> design that maybe doesn't take full advantage of all the benefits the
queueing
> scheme might offer, but surely it should demonstrate some significant
benefit
> without significant downsides?
> 
> And we can of course have a discussion on whether RTT unfairness (or any
> other property or lack of one) is a significant downside or not, but I
don't think
> we've had that discussion?
> 
> Basically, I'm trying to re-engage here for a bit to see if there is *any*
hope
> for this entire area of work, if I should just ignore it again or even
start arguing
> we should abandon it all, given how wedged the discussion between the
> camps has been for a long time.
> 
> Thanks,
> Lars

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Nov 2020 14:34:06 +0100

Hi Lars,

I appreciate your considerate and balanced position.

I would like to ask a question, you say:

"If "Prague" (I guess you mean a TCP variant taking advantage of L4S?) is a "work in progress", why are we then pushing towards finalizing the L4S part of the overall scheme?"

And I agree that is worth discussing. The problem is, that the network bits of L4S, basically the AQM are sold on a set of promises that require a compliant protocol to realize themselves.

Honestly, I wonder how one would go and test, characterize the network bits without having a fully working protocol to test them against/with.

All of this indicates that L4S simply is not fully baked, or baked enough to consider turning its components into RFCs. And the rationale for not doing that is simply, that L4S is not with-out side effects, and spending scarce resources like an IPv4/6 codepoint should be well weighted against the risk that the code-point might be wasted. If L4S wants to use that codepoint exclusively, I think it is not too much to ask, that L4S first demonstrates that it actually can deliver on its promises in real and realistic internet conditions and that at the least it does not make things (considerably) worse than they are right now.
	IMHO L4S has still a way ahead of itself to demonstrate safety and promised functionality before it should be even considered to be turned into an RFC.

Best Regards
	Sebastian

> On Nov 17, 2020, at 13:32, Lars Eggert <[email-masked]> wrote:
> 
> Hi,
>

---

From: Lars Eggert <[email-masked]>
Date: Tue, 17 Nov 2020 15:38:04 +0200

Hi,

> instance realtime media has different dynamics than bulk transfers

thanks for explaining! I agree.

> in the charter that there must be a fully functional L4S capable congestion

Here, I disagree. IMO an existence proof of a CC scheme that derives some benefits from L4S in at least some realistic/common scenarios without suffering from issues is *key* - why else would we do even do L4S (or any other queueing scheme)? How would we even know that we got the queuing part right?

> make it worthwhile to engage in the work.

The future may bring advanced CC schemes that optimize their use of L4S, delivering large benefits without suffering from any issues. But, it also may not. An existence proof of a CC scheme that managed to deliver some benefits for some key scenarios *now* without regressions in others would go a long way to convince me that such advanced schemes are on the horizon.

> than consume energy. Feel free to disagree.

Maybe it's been too long, but I can't recall the WG having anything like agreement on the question whether less RTT fairness compared to "standard" TCP is OK or not. And I don't have a strong opinion here. What we do need though is agreement in the WG what kind of regressions/changes in behavior from current TCP we're OK with for this new variant of TCP, and which ones we're not. Have we had that discussion?

Thanks,
Lars

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Nov 2020 14:49:01 +0100

Ingemar,

> On Nov 17, 2020, at 14:25, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi
> 

	[SM] Fair enough, it really does not matter which protocol is used to demonstrate that L4S' promises can be reached in a way that does not endanger the existing internet. So far TCP Prague is the only contender though, so it will have to carry the lift to demonstrate that L4S is safe to deploy.

> SCReAM that is likely to be updated to make better use of L4S.

	[SM] None of these are done and can be tested today, and none of these actually try to implement the Prague requirements that are essential to L4S' safety (IMHO a harebrained decision, protocols gain nothing by implementing the Prague requirements, and hence will not do so, the AQM does not enforce much, so we end up with L4S as a drag race track for CDN to eyeballs, where the only hurdle o admission is going to be to set ECT(1)).

> Like with

	Well, if L4S would retract its proposal to a) exclusively use the ECT(1) codepoint, and b) to redefine how a flow should respond to CE markings, all of that would be reasonable arguments. But L4S is not free of nasty side-effects that need to be considered when weighing its cost and benefits. And without a properly working protocol neither functionality not safety can be fully assessed, Case in point, the fact that protocols are, under the misnamed moniker reduction of RTT-bias, expected to make up for DualPI2's inability to equitably share between the two queues. If protocols do not do this, L4S' operational safety goes to hell, and we are back at ECT(1) flows getting 16 times the rate that non-ECT(1) flows get. If the AQM would be properly designed without such a glaring failure mode the onus on protocols would be reduced.

> unfairness issue.

	[SM] It does not, as I have posted multiple times, TCP Pragues RTT fairness is considerably worse the e.g. Cubic's. The only thing that is potentially solved (if the repositoeries defaults are adjusted) is the 15ms fairness hole, but labeling that an RTT fairness issue has always been an attempt at smoke and mirrors to deflect from the root cause of the issue by the misdesigned AQM.

> It is still not perfect I guess but all this is a matter
> imagine that it will be improved over time.

	[SM] Sure, by all means take your time and iterate until you get it right, but please pause the L4S internet draft process until that point has been reached.

> Also I see it likely that L4S

	[SM] Well, what is missing is that team L4S offers a theory how RTT bias actually realizes itself and how it can be meaningfully and generally addressed by the end-points. Trying to eliminate RTT-bias without first understanding its root causes seems like a sub-optimal use of everybody's time.

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 17 Nov 2020 14:48:02 +0000

Hi

I see in a parallel thread that Koen mentioned the default RTT bias
compensation is changed in Prague so I understand that it is now possible to
rerun the tests that spawned this thread. I guess this should be able to
show if the RTT bias issue to solved to an acceptable degree (whatever that
may be).  

Also I recall that Bob presented material that outlined the RTT unfairness
many moons ago (pre-Corona), don't recall that it caused too much commotion.
So I guess more involvement is from others (not just pro/against L4S folks)
to sort this out otherwise we will just have another lengthy mail thread
that leads nowhere.

/Ingemar  

> Sent: den 17 november 2020 14:38
> On 2020-11-17, at 15:25, Ingemar Johansson S
> > Other work is done on e.g. BBRv2 that can use L4S as well, and then we
> > have SCReAM that is likely to be updated to make better use of L4S.
> > Like with other congestion control I don't foresee a one size fits all
> > here, for instance realtime media has different dynamics than bulk
> > transfers
> 
> > I see all these algorithms as work in progress and I don't see it
> > explicit in the charter that there must be a fully functional L4S
> > capable congestion control available.
> 
> Here, I disagree. IMO an existence proof of a CC scheme that derives some
> benefits from L4S in at least some realistic/common scenarios without
> suffering from issues is *key* - why else would we do even do L4S (or any
> other queueing scheme)? How would we even know that we got the queuing
> part right?
> 
> > I see that the Prague work addresses many of the issues, among then
> > the RTT unfairness issue. It is still not perfect I guess but all this
> > is a matter of brain cycles in --> results out. Prague is an initial
> > design and I imagine that it will be improved over time. Also I see it
> > likely that L4S capable CCs can draw attention among academics in the
> > future but for that to happen we need to move forward to give the
> > signal that we move forward, to make it worthwhile to engage in the
work.
> 
> The future may bring advanced CC schemes that optimize their use of L4S,
> delivering large benefits without suffering from any issues. But, it also
may
> not. An existence proof of a CC scheme that managed to deliver some
> benefits for some key scenarios *now* without regressions in others would
> go a long way to convince me that such advanced schemes are on the
horizon.
> 
> > The RTT unfainess.. I believe that it has been discussed to death by
> > now. To be honest I don't believe that additional discussion will do
> > anything more than consume energy. Feel free to disagree.
> 
> 
> Maybe it's been too long, but I can't recall the WG having anything like
> agreement on the question whether less RTT fairness compared to "standard"
> TCP is OK or not. And I don't have a strong opinion here. What we do need
> though is agreement in the WG what kind of regressions/changes in behavior
> from current TCP we're OK with for this new variant of TCP, and which ones
> we're not. Have we had that discussion?
> 
> Thanks,
> Lars

---

From: Pete Heist <[email-masked]>
Date: Tue, 17 Nov 2020 20:13:55 +0100

Hi Koen, some responses inline...

On Mon, 2020-11-16 at 20:43 +0000, De Schepper, Koen (Nokia -
BE/Antwerp) wrote:
> recently joining the discussions and a reminder for the rest that it
> is important to take those conditions into account when designing and
> focused only on de-risking the bigger safety challenges. Most of the
> improvements (except low latency) were not prioritized up to now, as
> we are sure some are low hanging fruit and others with the right
> incentives will want to spend time on the harder ones. Most of the
> commercial parties with an expected interest and benefit of an L4S CC
> have been reasonable quite, I hope because they are confident that
> the network drafts and the mandatory Prague requirements and proposed
> improvements are feasible. I guess some are also less keen on sharing
> their efforts with competition and others might be waiting for real
> deployments before getting really into action. If there are concerns
> from their side, of course we would like to hear those too. The only
> concern I heard up to now is that all this takes too long before they
> can start the experiment. They want asap real world deployment...
> 
> > > by starting marking in the L queue later? Doing so may also
> > > improve RTT fairness for Prague flows.
> 
> capacities. We tried to address this by explaining this in section
> 6.3 of the L4S Architecture draft: 

It's good that it was covered in the draft. One source of burstiness
unrelated to the link layer that could be added or addressed is cross-
flow traffic. I'm not an expert on this topic, but I noticed a mention
of it in the description of Figure 3 in this paper:

http://buffer-workshop.stanford.edu/papers/paper34.pdf

It would be good if we knew how much this source of burstiness amounts
to. To me however, it seems like a minority of Internet applications
would need marking near 1ms, and in the cases where that is needed, the
application could use DSCP to supply a hint that it's willing to accept
a potentially large throughput reduction for such a low delay.

> > > 2) For the reported network bias, can this be fixed in the
> > > DualPI2 and Prague implementations? One chart which we didn't
> > > reference in our writeup illustrates the consistent bias for
> > > Prague vs CUBIC.
> 
> network nodes that don't identify individual flows. To compensate
> individual flow's RTT you need to know them first. To solve this
> anyways, it would need a per packet indication of the RTT (ref past
> proposals like RCP and XCP). If we would have space to put this info
> in the header, we wouldn't be discussing about this single available
> ECN bit, right? Therefor the End-System is the right place to handle
> this, and why the RTT-independent requirement is important. The
> current implementation has a non-Prague-compliant default (not RTT-
> independent), which we need to fix clearly. It would be good to agree
> on a reasonable compromise of a default f(RTT) correction function.
> We also implemented a gradual correction over time, so a new (short)
> flow would not be delayed, and will be responsive, only if it lasts
> for a while it will gradually converge to the RTT-independent rate.
> This could be in the range of the convergence time of the typical-RTT
> flows.
> 

Although I won't have time myself for more experiments with this, our
test source is published, so it would be possible for someone else to
try to add more parameter variations to the batch.

Importantly, the behavior using the default parameters should work well
across a wide range of RTTs without tuning, which I think others
pointed out as well.

> > > non-L4S flows a safety problem. We've seen 11:1 to 18:1 in our
> > > recent tests.
> > > Although we're not sure of the worst case, what we're seeing now
> > > is outside of my comfort zone, personally.
> 
> conditions Reno on a CoDel AQM with a target of 5ms. Cubic
> compensates performance for the longer RTT flows, Prague (with the
> right settings) compensates mandatory for the smaller RTT. Clearly a
> production congestion control is likely to do both, or fall back to

Actually the 16:1 referred to in this case wasn't from RTT dependence,
but when L4S and non-L4S flows share the same RFC3168 queue. That ratio
can be significantly higher in some cases (for example with PIE using
the default parameters plus ECN enabled), but it's around this ratio
for Codel queues with the default parameters.

I haven't seen much discussion around it yet (admitting that I'm a
little behind!), but we did post a newer result showing how tunneled
traffic causes L4S and non-L4S flows to wind up in the same queue:

https://github.com/heistp/l4s-tests/#unsafety-in-tunnels-through-rfc3168-bottlenecks

> > > hardest. This includes the domination of L4S over non-L4S flows
> > > in the same RFC3168 queue (see the issue with tunnels reported
> > > above for a common path to that), 
> > > and the intra-flow latency spikes for L4S flows (   
> > > https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes).
> 
> for longer than 10 seconds when throughput reduces from 50Mbps to
> 1Mbps on an 80ms base RTT path). When compared under the same
> conditions with Cubic on PI2 you can definitely see how it can be
> improved using an appropriate AQM (a very short peak of I guess less
> than 100ms extra latency). Hopefully these CoDel instances can be
> fixed and can support the ECN threshold for L4S instead (and maybe
> use PI2 instead of CoDel for Classic). On the other hand these
> extremes would be also easy cases for both Classic and L4S CCs to
> detect and fall back on a delay based CC when latencies are that bad
> (falling back to Classic ECN or Classic drop would even not help to
> avoid the CoDel problem). But I doubt the Linux Cubic developers
> would be happy to extend Cubic to avoid CoDel flaws??

Yes, we're aware that the transient behavior of Codel sometimes leaves
something left to be desired. The magnitude of spikes is related to the
starting bandwidth, ending bandwidth, and RTT. Although CUBIC's spikes
can also be long in extreme cases, for a more common 50Mbps to 10Mbps
reduction, the response is much more reasonable for CUBIC (maybe 400ms
spike that decays sharply and ends in 1.5s):

http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s2-codel-rate-step/l4s-s2-codel-rate-step-ns-clean-cubic-fq_codel-50Mbit-10mbit-80ms_tcp_delivery_with_rtt.svg

than for Prague (400ms spike with much slower decay, lasting 8s):

http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s2-codel-rate-step/l4s-s2-codel-rate-step-ns-clean-prague-fq_codel-50Mbit-10mbit-80ms_tcp_delivery_with_rtt.svg

The source of this, mainly for other readers, is the difference between
the MD signal of today's RFC3168 CE, and the AD signal expected by L4S
transports after having redefined the CE codepoint, with no reliable
way of differentiating between the two. The present situation is that
there are many deployed fq_codel instances, all signaling by default
with RFC3168 ECN
(https://github.com/heistp/l4s-tests/#deployments-of-fq_codel), and
it's likely to remain that way for a long time. So, we do have to
consider these cases.

As an aside, overload protection was added to the COBALT AQM using the
BLUE algorithm in part because of this aspect of Codel. That starts at
around 400ms, which may have improved the above cases. However, that
was designed for unresponsive flows and extreme situations, rather than
as ordinary congestion control for well-behaved flows.

> > > Although I won't be able to support a WGLC until I see tested
> > > code that addresses the issues, I do want to support the WG along
> > > its present path to a conclusion...
> 
> it is a matter of just coding a fallback to Cubic if your detected
> RTT is too big. So as far as I'm concerned, we do have running code! 
> 
> preferences for other values, then let me know. If people have time
> and want to implement the Cubic fallback (or even better longer RTT-
> independence), then let me know too.

I noticed that there were some recent changes, so we'll see what
there's time for.

Regards,
Pete :)

> for now to the confines of the L4S design. We'd like to support
> forward progress while facing the issues head on if possible. I'll
> take a shot here as tersely as I can:
> 
> and Prague implementations? One chart which we didn't reference in
> our writeup illustrates the consistent bias for Prague vs CUBIC.
> the same RFC3168 queue (see the issue with tunnels reported above for
> a common path to that), and the intra-flow latency spikes for L4S
> flows
> few years ago (    
> https://github.com/heistp/l4s-tests/#deployments-of-fq_codel). #2 is
> > useful 
> > > > path RTTs, and intra-flow latency spikes upon rate reductions
> > > > in 
> > > > fq_codel.