Subject: Re: [tsvwg] Another tunnel/VPN scenario (was RE: Reasons for WGLC/RFC asap)

From: Bob Briscoe <[email-masked]>
Date: Thu, 3 Dec 2020 12:19:41 +0000

Sebastian, Jonathan,

Encrypted tunnels not revealing the component flows in the payload is a 
correct property of layering, which has been a principle that guides 
protocol design since long before the Internet protocols adopted it. I 
would ask you to imagine standing up (virtually) in an IETF plenary (or 
any networking forum) and blame layering (and/or layered encryption) for 
FQ not being able to isolate component flows. You would be flamed to a 
crisp.

Certainly, a flow (such as L4S) that needs to be isolated from certain 
other flows could be seen as the cause of the need for FQ. But it is 
not. It is the cause of the need for isolation, not specifically /flow/ 
isolation. It is FQ that chooses to rely on identifiers that violate 
layering in order to provide that isolation.

The root cause of the problem can be objectively determined. FQ doesn't 
correctly schedule flows within encrypted tunnels, whether L4S is 
present or not. Therefore L4S cannot be the root cause of this problem.

It is instructive to see how the world looks to people who are stuck so 
far down the FQ rabbit-hole that the walls of the FQ burrow and the 
other rabbits down there give the comforting feeling that FQ is the 
whole world. Except for that small chink of light in the sky when 
looking up out of the rabbit-hole, which is all that is visible of the 
rest of the universe of networking.

Bob

On 20/11/2020 07:56, Sebastian Moeller wrote:
> Ingemar,
>
> encrypted tunnels not revealing the individual component flows in 
> their payload is a feature of encryption and not a failure of flow 
> isolation... Arguably an encrypted tunnel that disguises as a single 
> flow should not allow propagation of ECN codepoints between the inner 
> and outer layer at all, but then that is in the hand of the tunnel 
> operator, not the AQM node.
> It is quite interesting though, how tunneling is brought as an 
> argument against the SCE proposal (only CE is guaranteed to be passed 
> between layers*) and the very moment L4S shows issues with tunneling 
> this is interpreted as someone else's problem. This constant 
> application of double standards alone should be reason to reject the 
> L4S drafts....
>
>
> Best Regards
> Sebastian
>
>
> *) one of the original sins in regards to ECN and tunnels seems to 
> have been not simply requiring complete unconditionally copying of 
> inner ECN bits to outer ECN bits on en-capsulation and outer to inner 
> on de-capsulation and letting the end-points deal with any accidental 
> fall-out (to be complete, a tunnel should either not do any ECN 
> propagation in any direction, or the one described). For rfc3168, I 
> can fully understand why that route was not chosen, but years later 
> for rfc6040 that decision is much harder to rationalize.
>
>
> On 20 November 2020 07:04:56 CET, Ingemar Johansson S 
> <ingemar.s.johansson=[email-masked] 
> <mailto:ingemar.s.johansson=[email-masked]>> wrote:
>
>     Hi David, Pete
>
>     I try to make it clear to me what this scenario show is about and
>     somehow I see it more as a flow isolation problem that makes FQ
>     non-functional rather than an L4S problem?.
>
>     There is of course a possibility that VPNs do not implement
>     RFC6040 properly. I guess for software VPNs it is only an update
>     cycle away, more hard/firmware VPNs can of course be a different
>     story but I guess that, similar to the discussion on home gateways
>     and ECN a few months ago, they can be upgradeable too  ?
>
>     /Ingemar
>
>     *From:* tsvwg <[email-masked]
>     <mailto:[email-masked]>> *On Behalf Of *Black, David
>     *Sent:* den 19 november 2020 22:20
>     *To:* Pete Heist <[email-masked] <mailto:[email-masked]>>
>     *Cc:* tsvwg IETF list <[email-masked] <mailto:[email-masked]>>
>     *Subject:* [tsvwg] Another tunnel/VPN scenario (was RE: Reasons
>     for WGLC/RFC asap)
>
>     [posting as an individual]
>
>     > I'll leave it to the WG to come up with examples of what types
>     of tunnels and traffic scenarios could lead to this,
>
>     > but one example is a user who has a privacy VPN on their PC, and
>     fq_codel on their home gateway.
>
>     > Let's say one flow connects to an L4S capable server, and
>     another flow to a non-L4S, conventional server.
>
>     > The L4S flow will dominate the non-L4S one (whether it's ECN
>     capable or not), probably causing some level
>
>     > of poor service, perhaps for a video stream, download, or whatever.
>
>     It’s more than home gateways – there will be increasing use of
>     VPNs with public or shared WiFi to block snooping by other WiFi
>     devices and/or the access point infrastructure.  In that case, the
>     WiFi access point and nodes between the access point and the VPN
>     gateway can only look at the outer IP header applied by the VPN. 
>     If the VPN preserves packet boundaries and complies with RFC 6040,
>     then ECT(1) in the inner header will show up in the outer header,
>     but not all VPNs do both of those.
>
>     Thanks, --David
>
>     *From:* tsvwg <[email-masked]
>     <mailto:[email-masked]>> *On Behalf Of *Pete Heist
>     *Sent:* Thursday, November 19, 2020 3:03 PM
>     *To:* Gorry Fairhurst
>     *Cc:* tsvwg IETF list
>     *Subject:* Re: [tsvwg] Reasons for WGLC/RFC asap
>
>     [EXTERNAL EMAIL]
>
>     On Thu, 2020-11-19 at 16:34 +0000, Gorry Fairhurst wrote:
>
>         On 19/11/2020 16:22, Pete Heist wrote:
>
>             Hi Koen,
>
>             Rather than thinking of this as advantages and
>             disadvantages to waiting, I see it as an engineering
>             process. It was decided earlier this year that the L4S
>             proposal has enough support to continue, so we're on that
>             path now. Part of that decision, as I understood it, also
>             recognized that there are valid safety concerns around
>             compatibility with existing AQMs, and some solution needs
>             to be devised.
>
>             RFC3168 bottleneck detection was added to TCP Prague,
>             which appears to be difficult to do reliably when there is
>             jitter or cross-flow traffic, and it has since been
>             disabled in the reference implementation. The l4s-ops
>             draft was started, but isn't complete yet and may need WG
>             adoption as part of a LC. We can then decide how effective
>             the proposed mitigations are against the risks and prevalence.
>
>             To start a WGLC now would circumvent that earlier
>             recognition that a safety case needs to be made.
>             Meanwhile, since testing showed that tunnels through
>             RFC3168 FQ AQMs are a straightforward path to unsafe flow
>             interaction, along with other issues relative to the
>             goals, it doesn't seem like the engineering process is
>             done just yet.
>
>         By the way, I liked your data - and it helped me a lot to look
>         at this, thanks very much for doing this.
>
>     I'm glad, as I think we're at our best when we're doing
>     engineering and producing data. I wish it were easier to do!
>
>         It would help me if you clarify what you mean by  "unsafe" -
>         to me "safety" relates to traffic unresponsive to drop, as in
>         CBR traffic, etc. I've not understood how CE-marked traffic
>         can erode safety, but maybe I missed something?
>
>     Sure, so the existing RFC3168 CE signal in use on the Internet
>     today indicates an MD (multiplicative decrease), whereas the
>     redefined CE signal in L4S indicates an AD (additive decrease).
>     Two congestion controls responding to CE in a different way, or
>     one that responds to CE with an AD and one that responds only to
>     drop (i.e. all standard congestion controls that advertise
>     Not-ECT), will not interact safely in the same RFC3168 signaling
>     queue. We're probably on the same page here already, but I'll
>     refer to section 5 of RFC8257.
>
>     That is one of the reasons why ECT(1) is used in L4S to place L4S
>     flows in the L queue- to keep them separate from conventional
>     flows in the C queue. As long as flows have advertised their
>     capability correctly, that works.
>
>     However, existing RFC3168 queues do not have knowledge of L4S,
>     therefore will not know that ECT(1) means that traffic needs to be
>     segregated and signaled in a different way. They will signal a
>     Prague flow, which sets ECT(1), with CE, expecting the flow to
>     respond with an MD, rather than AD. Meanwhile they'll signal an
>     RFC3168 or non-ECN flow with either CE or drop, and in either case
>     the flow will respond with an MD, causing conventional flows to
>     yield to Prague flows to varying degrees depending on the AQM in use.
>
>     Here's an example of CUBIC and Prague when they end up in the same
>     fq_codel queue:
>
>     http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg
>     <https://protect2.fireeye.com/v1/url?k=d323b60b-8cb88f48-d323f690-861fcb972bfc-55ab41b160781c6c&q=1&e=d231a1ce-c8b5-4e7f-a9d7-fb6323a2bf1d&u=http%3A%2F%2Fsce.dnsmgr.net%2Fresults%2Fl4s-2020-11-11T120000-final%2Fl4s-s6-rfc3168-1q%2Fl4s-s6-rfc3168-1q%2Fl4s-s6-rfc3168-1q-ns-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg>
>
>     Here's a more extreme example of Reno and Prague sharing a single
>     PIE queue with ECN enabled (less common):
>
>     http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-reno-pie-50Mbit-20ms_tcp_delivery_with_rtt.svg
>     <https://protect2.fireeye.com/v1/url?k=f275e74d-adeede0e-f275a7d6-861fcb972bfc-9924f35e47dbd9cf&q=1&e=d231a1ce-c8b5-4e7f-a9d7-fb6323a2bf1d&u=http%3A%2F%2Fsce.dnsmgr.net%2Fresults%2Fl4s-2020-11-11T120000-final%2Fl4s-s6-rfc3168-1q%2Fl4s-s6-rfc3168-1q%2Fl4s-s6-rfc3168-1q-ns-prague-vs-reno-pie-50Mbit-20ms_tcp_delivery_with_rtt.svg>
>
>     In the example with PIE, Reno appears to be driven at or close to
>     minimum cwnd. In the fq_codel example, the steady state throughput
>     of Prague:CUBIC is around 19:1. We've seen a range in the Codel
>     case from around 12:1 to 20:1. In my opinion, we could use the
>     word "unsafe" here in both cases.
>
>         I'm not sure why "tunnels have crept in here. There have
>         always been side-effects with classification (and hence
>         scheduling), but I don't see new issues relating to "tunnels"
>         with ECN.
>
>     Tunnels are relevant because they provide an easy practical path
>     to the unsafe flow interaction described above. The widely used
>     fq_codel qdisc has ECN enabled by default. Fortunately, because it
>     has flow-fair queueing, Prague flows and conventional flows are
>     usually placed in a separate queue (hash collisions aside),
>     causing Prague to only affect itself with additional delay (TCP
>     RTT). However, a tunnel's encapsulated packets all share the same
>     fq_codel queue because they all have the same 5-tuple, so there is
>     unsafe interaction between the tunnel's flows. Here we use
>     Wireguard through fq_codel:
>
>     http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s5-tunnel/l4s-s5-tunnel-phys-wireguard-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg
>     <https://protect2.fireeye.com/v1/url?k=ecac4b36-b3377275-ecac0bad-861fcb972bfc-d30217b6aa1796c6&q=1&e=d231a1ce-c8b5-4e7f-a9d7-fb6323a2bf1d&u=http%3A%2F%2Fsce.dnsmgr.net%2Fresults%2Fl4s-2020-11-11T120000-final%2Fl4s-s5-tunnel%2Fl4s-s5-tunnel-phys-wireguard-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg>
>
>     I'll leave it to the WG to come up with examples of what types of
>     tunnels and traffic scenarios could lead to this, but one example
>     is a user who has a privacy VPN on their PC, and fq_codel on their
>     home gateway. Let's say one flow connects to an L4S capable
>     server, and another flow to a non-L4S, conventional server. The
>     L4S flow will dominate the non-L4S one (whether it's ECN capable
>     or not), probably causing some level of poor service, perhaps for
>     a video stream, download, or whatever.
>
>         I'm not commenting on when the Chairs think a WGLC will
>         provide useful information, we'll say that in due course.
>
>     Ok, I trust that we'll engage enough disinterested people into
>     congestion control who will add their input.
>
>     Thanks Gorry for looking this over. :)
>
>         Best wishes,
>
>         Gorry
>
>             Regards,
>
>             Pete
>
>             On Wed, 2020-11-18 at 10:31 +0000, De Schepper, Koen
>             (Nokia - BE/Antwerp) wrote:
>
>                 Hi all,
>
>                 To continue on the discussions in the meeting, a recap
>                 and some extra thoughts. Did I miss some arguments?
>
>                 Benefits to go for WGLC/RFC asap:
>
>                   * There is NOW a big need for solutions that can
>                     support Low Latency for new Interactive applications
>                   * The big L4S benefits were a good reason to justify
>                     the extra network effort to finally implement ECN
>                     in general and AQMs in network equipment
>                   * Timing is optimal now: implementations in NW
>                     equipment are coming and deployment can start now
>                   * Deployment of L4S support will include deployment
>                     of Classic ECN too! So even for the skeptics among
>                     us, that consider that the experiment can fail due
>                     to CCs not performing to expectations, we will
>                     fall back to having Classic ECN support
>                   * Current drafts are about the network part, and are
>                     ready and stable for a very long time now.
>                   * Only dependency to CCs in the drafts are the
>                     mandatory Prague requirements (only required
>                     input/review from future CC developers: are they
>                     feasible for you)
>                   * We have a good baseline for a CC (upstreaming to
>                     Linux is blocked by the non-RFC status)
>                   * Larger scale (outside the lab) experiments are
>                     blocked by non-RFCs status
>                   * It will create the required traction within the CC
>                     community to come up with improvements (if needed
>                     at all for the applications that would benefit
>                     from it; applications that don’t benefit from it
>                     yet, can/will not use it)
>                   * NW operators have benefits now (classic ECN and
>                     good AQMs) and in the future can offer their
>                     customers better Low Latency experience for the
>                     popular interactive applications
>                   * When more L4S CCs are developed, the real
>                     independent evaluation of those can start
>
>                 Disadvantages to wait for WGLC/RFC:
>
>                   * We’ll gets stuck in an analysis paralysis (aren’t
>                     we already?)
>                   * Trust in L4S will vanish
>                   * No signs that we can expect more traction in CC
>                     development; trust and expectations of continuous
>                     delays will not attract people working on it, as
>                     there will be plenty of time before deployments
>                     are materializing
>                   * Product development of L4S will stall and die due
>                     to uncertainty on if L4S will finally materialize
>                   * Product development of Classic ECN will stall and
>                     die due to uncertainty on how L4S will finally
>                     materialize
>
>                 What are the advantages to wait? Do they overcome
>                 these disadvantages?
>
>                 Regards,
>
>                 Koen.
>
>     -->
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
                PRIVILEGED AND CONFIDENTIAL

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 3 Dec 2020 14:40:40 +0100

Bob,

more beow in-line, prefixed [SM].

> On Dec 3, 2020, at 13:19, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian, Jonathan,
> 
> Encrypted tunnels not revealing the component flows in the payload is a correct property of layering, which has been a principle that guides protocol design since long before the Internet protocols adopted it. I would ask you to imagine standing up (virtually) in an IETF plenary (or any networking forum) and blame layering (and/or layered encryption) for FQ not being able to isolate component flows. You would be flamed to a crisp.

	[SM] Sure, but by that same token passing of ECN codepoints between inner and outer is also a layering violation, if we did not do this, we would have no issue. But still "moving" ECN codepoints around at e- and de-tunneling is "endorsed" by the IETF. The same IETF now should make sure that these recommendations do not lead to negative outcomes, like some flows taking over most of the tunnel, by say consciously redefining the response to those ECN signals by transport protocols. 

> 
> Certainly, a flow (such as L4S) that needs to be isolated from certain other flows could be seen as the cause of the need for FQ. But it is not. It is the cause of the need for isolation, not specifically /flow/ isolation. It is FQ that chooses to rely on identifiers that violate layering in order to provide that isolation.

	[SM] Well, as long as all flows behave TCP-friendly (as they arguably should) that isolation-failure is pretty benign, no? The issue here is that L4S proposes a new way of ECN-signalling that has exciting new TCP-unfriendly failure modes, like a shared tunnel with rfc3168 flows over a rfc3168 bottleneck. IMHO, if somebody uses a tunnel, they deserve to be treated like a single flow, what is not okay, is to have some mis-designed protocols take over the tunnel for themselves.
	IMHO, looking at essential IP and TCP/UDP header fields for flow definition is the only sane way of doing it (we can haggle which to include), any field that can be gamed, will be gamed, so flows need to be defined as little game-able as possible, and if that means looking into protocol headers so be it.

> 
> The root cause of the problem can be objectively determined. FQ doesn't correctly schedule flows within encrypted tunnels, whether L4S is present or not. Therefore L4S cannot be the root cause of this problem.

	[SM] No arguable FQ is doing what it is supposed, if something presents itself to the network as a single flow it gets treated as such, if the source wanted different treatment, it knows what to do....
But again it is L4S that turns this into a problem, because in the condition we currently discuss, it fails to follow the principle of TCP-fairness... So you basically claim, if you unfairly tackle someone while the referee isn't watching that constitutes no foul. Not sure I agree with that position.

	IMHO, the root cause of this issue is still L4S hare-brained design of thinking that re-defining the meaning of CE is a safe proposition (see how I picked up your Leporidae-theme from below?). The tunnel with the rfc3168 is only the example the demonstrates that just saying anybody who cares should use an FQ-AQM is not a solution, but honestly, that is a position team L4S introduced to the discussion as a means to keep declaring the fall-out from re-defining CE "not a big deal".
Again to be clear, the problem is that L4S is designed to be actively hostile towards/over rea rfc3168 bottlenecks, and the ideal fix would be to change L4S here. there have been several proposals how to achieve that, e.g. by Jake Holland, that all have been essentially ignored by team L4S (sure such a change would requre some re-engineering, so the argument that engineering would be required is IMHO not a real blocker for Jake's proposal).

> 
> It is instructive to see how the world looks to people who are stuck so far down the FQ rabbit-hole
> that the walls of the FQ burrow and the other rabbits down there give the comforting feeling that FQ is the whole world. Except for that small chink of light in the sky when looking up out of the rabbit-hole, which is all that is visible of the rest of the universe of networking.

	[SM] Does this actually contain actionable content you want an answer to? If so please clarify, otherwise I will ignore that as an unwelcome attempt at derailing the discussion.
	I understand, that you dislike FQ (you have not been shy or subtle about that position), but the issue at hand is really an issue about L4S's proposed schemes inherent lack of safety.

Best Regards
	Sebastian

>> encrypted tunnels not revealing the individual component flows in their payload is a feature of encryption and not a failure of flow isolation... Arguably an encrypted tunnel that disguises as a single flow should not allow propagation of ECN codepoints between the inner and outer layer at all, but then that is in the hand of the tunnel operator, not the AQM node.
>> 	It is quite interesting though, how tunneling is brought as an argument against the SCE proposal (only CE is guaranteed to be passed between layers*) and the very moment L4S shows issues with tunneling this is interpreted as someone else's problem. This constant application of double standards alone should be reason to reject the L4S drafts....
>> 
>> 
>> Best Regards
>> 	Sebastian
>> 
>> 
>> *) one of the original sins in regards to ECN and tunnels seems to have been not simply requiring complete unconditionally copying of inner ECN bits to outer ECN bits on en-capsulation and outer to inner on de-capsulation and letting the end-points deal with any accidental fall-out (to be complete, a tunnel should either not do any ECN propagation in any direction, or the one described). For rfc3168, I can fully understand why that route was not chosen, but years later for rfc6040 that decision is much harder to rationalize. 
>> 
>> 
>> On 20 November 2020 07:04:56 CET, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
>> Hi David, Pete
>>  
>> I try to make it clear to me what this scenario show is about and somehow I see it more as a flow isolation problem that makes FQ non-functional rather than an L4S problem?. 
>>  
>> There is of course a possibility that VPNs do not implement RFC6040 properly. I guess for software VPNs it is only an update cycle away, more hard/firmware VPNs can of course be a different story but I guess that, similar to the discussion on home gateways and ECN a few months ago, they can be upgradeable too  ?
>>  
>> /Ingemar
>>  
>>  
>> From: tsvwg <[email-masked]> On Behalf Of Black, David
>> Sent: den 19 november 2020 22:20
>> To: Pete Heist <[email-masked]>
>> Cc: tsvwg IETF list <[email-masked]>
>> Subject: [tsvwg] Another tunnel/VPN scenario (was RE: Reasons for WGLC/RFC asap)
>>  
>> > I'll leave it to the WG to come up with examples of what types of tunnels and traffic scenarios could lead to this,
>> > but one example is a user who has a privacy VPN on their PC, and fq_codel on their home gateway.
>> > Let's say one flow connects to an L4S capable server, and another flow to a non-L4S, conventional server.
>> > The L4S flow will dominate the non-L4S one (whether it's ECN capable or not), probably causing some level
>> It’s more than home gateways – there will be increasing use of VPNs with public or shared WiFi to block snooping by other WiFi devices and/or the access point infrastructure.  In that case, the WiFi access point and nodes between the access point and the VPN gateway can only look at the outer IP header applied by the VPN.  If the VPN preserves packet boundaries and complies with RFC 6040, then ECT(1) in the inner header will show up in the outer header, but not all VPNs do both of those.
>>  
>> Thanks, --David
>>  
>> From: tsvwg <[email-masked]> On Behalf Of Pete Heist
>> Sent: Thursday, November 19, 2020 3:03 PM
>> To: Gorry Fairhurst
>> Cc: tsvwg IETF list
>> Subject: Re: [tsvwg] Reasons for WGLC/RFC asap
>>  
>> [EXTERNAL EMAIL] 
>> 
>> Rather than thinking of this as advantages and disadvantages to waiting, I see it as an engineering process. It was decided earlier this year that the L4S proposal has enough support to continue, so we're on that path now. Part of that decision, as I understood it, also recognized that there are valid safety concerns around compatibility with existing AQMs, and some solution needs to be devised.
>>  
>> RFC3168 bottleneck detection was added to TCP Prague, which appears to be difficult to do reliably when there is jitter or cross-flow traffic, and it has since been disabled in the reference implementation. The l4s-ops draft was started, but isn't complete yet and may need WG adoption as part of a LC. We can then decide how effective the proposed mitigations are against the risks and prevalence.
>>  
>> To start a WGLC now would circumvent that earlier recognition that a safety case needs to be made. Meanwhile, since testing showed that tunnels through RFC3168 FQ AQMs are a straightforward path to unsafe flow interaction, along with other issues relative to the goals, it doesn't seem like the engineering process is done just yet.
>>  
>> By the way, I liked your data - and it helped me a lot to look at this, thanks very much for doing this.
>> 
>> I'm glad, as I think we're at our best when we're doing engineering and producing data. I wish it were easier to do!
>> It would help me if you clarify what you mean by  "unsafe" - to me "safety" relates to traffic unresponsive to drop, as in CBR traffic, etc. I've not understood how CE-marked traffic can erode safety, but maybe I missed something?
>> 
>> Sure, so the existing RFC3168 CE signal in use on the Internet today indicates an MD (multiplicative decrease), whereas the redefined CE signal in L4S indicates an AD (additive decrease). Two congestion controls responding to CE in a different way, or one that responds to CE with an AD and one that responds only to drop (i.e. all standard congestion controls that advertise Not-ECT), will not interact safely in the same RFC3168 signaling queue. We're probably on the same page here already, but I'll refer to section 5 of RFC8257.
>>  
>> That is one of the reasons why ECT(1) is used in L4S to place L4S flows in the L queue- to keep them separate from conventional flows in the C queue. As long as flows have advertised their capability correctly, that works.
>>  
>> However, existing RFC3168 queues do not have knowledge of L4S, therefore will not know that ECT(1) means that traffic needs to be segregated and signaled in a different way. They will signal a Prague flow, which sets ECT(1), with CE, expecting the flow to respond with an MD, rather than AD. Meanwhile they'll signal an RFC3168 or non-ECN flow with either CE or drop, and in either case the flow will respond with an MD, causing conventional flows to yield to Prague flows to varying degrees depending on the AQM in use.
>>  
>> Here's an example of CUBIC and Prague when they end up in the same fq_codel queue:
>> Here's a more extreme example of Reno and Prague sharing a single PIE queue with ECN enabled (less common):
>> In the example with PIE, Reno appears to be driven at or close to minimum cwnd. In the fq_codel example, the steady state throughput of Prague:CUBIC is around 19:1. We've seen a range in the Codel case from around 12:1 to 20:1. In my opinion, we could use the word "unsafe" here in both cases.
>> I'm not sure why "tunnels have crept in here. There have always been side-effects with classification (and hence scheduling), but I don't see new issues relating to "tunnels" with ECN.
>> 
>> Tunnels are relevant because they provide an easy practical path to the unsafe flow interaction described above. The widely used fq_codel qdisc has ECN enabled by default. Fortunately, because it has flow-fair queueing, Prague flows and conventional flows are usually placed in a separate queue (hash collisions aside), causing Prague to only affect itself with additional delay (TCP RTT). However, a tunnel's encapsulated packets all share the same fq_codel queue because they all have the same 5-tuple, so there is unsafe interaction between the tunnel's flows. Here we use Wireguard through fq_codel:
>>  
>> I'll leave it to the WG to come up with examples of what types of tunnels and traffic scenarios could lead to this, but one example is a user who has a privacy VPN on their PC, and fq_codel on their home gateway. Let's say one flow connects to an L4S capable server, and another flow to a non-L4S, conventional server. The L4S flow will dominate the non-L4S one (whether it's ECN capable or not), probably causing some level of poor service, perhaps for a video stream, download, or whatever.
>> I'm not commenting on when the Chairs think a WGLC will provide useful information, we'll say that in due course.
>> 
>> Ok, I trust that we'll engage enough disinterested people into congestion control who will add their input.
>>  
>> On Wed, 2020-11-18 at 10:31 +0000, De Schepper, Koen (Nokia - BE/Antwerp) wrote:
>>  
>> Hi all,
>>  
>> To continue on the discussions in the meeting, a recap and some extra thoughts. Did I miss some arguments?
>>  
>> 	• There is NOW a big need for solutions that can support Low Latency for new Interactive applications
>> 	• The big L4S benefits were a good reason to justify the extra network effort to finally implement ECN in general and AQMs in network equipment
>> 	• Timing is optimal now: implementations in NW equipment are coming and deployment can start now
>> 	• Deployment of L4S support will include deployment of Classic ECN too! So even for the skeptics among us, that consider that the experiment can fail due to CCs not performing to expectations, we will fall back to having Classic ECN support
>> 	• Current drafts are about the network part, and are ready and stable for a very long time now.
>> 	• Only dependency to CCs in the drafts are the mandatory Prague requirements (only required input/review from future CC developers: are they feasible for you)
>> 	• We have a good baseline for a CC (upstreaming to Linux is blocked by the non-RFC status)
>> 	• Larger scale (outside the lab) experiments are blocked by non-RFCs status
>> 	• It will create the required traction within the CC community to come up with improvements (if needed at all for the applications that would benefit from it; applications that don’t benefit from it yet, can/will not use it)
>> 	• NW operators have benefits now (classic ECN and good AQMs) and in the future can offer their customers better Low Latency experience for the popular interactive applications
>> 	• When more L4S CCs are developed, the real independent evaluation of those can start
>>  
>> 	• We’ll gets stuck in an analysis paralysis (aren’t we already?)
>> 	• Trust in L4S will vanish
>> 	• No signs that we can expect more traction in CC development; trust and expectations of continuous delays will not attract people working on it, as there will be plenty of time before deployments are materializing
>> 	• Product development of L4S will stall and die due to uncertainty on if L4S will finally materialize
>> 	• Product development of Classic ECN will stall and die due to uncertainty on how L4S will finally materialize
>>  
>> What are the advantages to wait? Do they overcome these disadvantages?
>>  
>> Regards,
>> Koen.
>>  
>>  
>>  
>> 
>> -->
> 
> --

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 3 Dec 2020 17:52:43 +0200

> On 3 Dec, 2020, at 2:19 pm, Bob Briscoe <[email-masked]> wrote:
> 

I did not make any such argument about layering, so I will treat this as the strawman it is, by ignoring it.

> The root cause of the problem can be objectively determined. FQ doesn't correctly schedule flows within encrypted tunnels, whether L4S is present or not. Therefore L4S cannot be the root cause of t…

This is where you are incorrect.  If FQ cannot distinguish between the flows encapsulated within a tunnel, its behaviour merely reverts to that of a single queue, by design.  This also means that FQ-AQMs effectively revert to single-queue AQMs in that case.  Existing traffic is perfectly happy with that arrangement, and it is generally an improvement over single queues *without* AQM, which is presently the status quo over much of the Internet.

RFC-3168 and RFC-8511 lay out certain congestion control behaviours which permit ECN and loss signalling of congestion to coexist, and that arrangement works quite well at present.  Transports obeying these requirements (and those of TCP friendliness without ECN) coexist without starving each other out of the network.  Even though they do not always share the link with perfect fairness (or even RTT-fairness), it is within some reasonable factor.

Specifically, RFC-3168 expects AQMs to signal by loss on Not-ECT packets at the same rate as they signal by CE marks on ECT packets, to treat ECT(0) and ECT(1) as equivalent, and for congestion control to respond in the same way to CE marks as to packet loss.  RFC-8511 relaxes RFC-3168 slightly, and specifies a minimum response to single CE marks (per RTT) of a 15% reduction of cwnd (for window based transports) or send rate (for rate based transports), ie. a Multiplicative Decrease event.

The introduction of L4S traffic upsets this situation, by violating the above-mentioned provisions of these two RFCs.  L4S transports cause substantially more harm to other transports, when they happen to share an AQM queue, than existing transports do.  Redefining the semantics of the CE mark is the single element that, if removed, makes all the coexistence problems on existing networks go away.

Frankly, the sooner the WG understands and accepts that basic fact, the sooner we can move to a viable solution - one which does not redefine CE from the semantics established by RFC-3168 and RFC-8511, and hence does not place burdens on networks which have no interest in the L4S experiment.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 3 Dec 2020 16:17:55 +0000

Sebastian,

> 	[SM] Sure, but by that same token passing of ECN codepoints between inner and outer is also a layering violation, if we did not do this, we would have no issue. But still "moving" ECN codepoints a…

[BB] I don't know which school you learned networking at, but passing 
information into a layer at one end or out of it at the other is how 
layers work. ECN was designed to be fully in line with layering.

The IETF has no obligation to make functions work that violate layering 
(like FQ). Layer violations have to be considered as an optimization 
that works when the higher layer is accessible and doesn't when it 
isn't. Same with other similar mechanisms like firewalls. Remember 
FQ-CoDel is not standards track.

> 	[SM] Well, as long as all flows behave TCP-friendly (as they arguably should) that isolation-failure is pretty benign, no?

[BB] FQ's failure to isolate flows that end up in a tunnel together 
overrides any flows that are trying to be TCP-friendly and forces them 
not to be TCP-friendly. You don't seem to have grocked that there are 
always two sides to this argument. I put the other side and recognized 
your side. Then I gave the principle (layering) that sits as the judge 
between the two - determining the side that the IETF and the networking 
industry as a whole will always take.

You can't keep playing the game of dressing up this argument from only 
one viewpoint, as if that somehow proves the other viewpoint isn't valid.

> The issue here is that L4S proposes a new way of ECN-signalling that has exciting new TCP-unfriendly failure modes, like a shared tunnel with rfc3168 flows over a rfc3168 bottleneck. IMHO, if somebody uses a tunnel, they deserve to be treated like a single flow, what is not okay, is to have some mis-designed protocols take over the tunnel for themselves.

[BB] An application endpoint is not in control of whether its flow is 
aggregated with others in a tunnel.

The whole of networking works by encapsulation - typically multiple 
layers of it. In networking, if you want network equipment to do 
something with a flow ID, you put another transport header inside the 
outer (typically UDP encapsulation). For instance, see how GUE ensures 
equal cost multipath routing works.

> Again to be clear, the problem is that L4S is designed to be actively hostile towards/over rea rfc3168 bottlenecks, and the ideal fix would be to change L4S here. there have been several proposals…

[BB] I worked with Jake to write-up all the pros and cons, which we put 
in the ecn-l4s-id draft after Jake approved it. You will see one pro and 
many, many cons.

> 	[SM] Does this actually contain actionable content you want an answer to? If so please clarify, otherwise I will ignore that as an unwelcome attempt at derailing the discussion.

[BB] When you produce any actionable content, please let us know.

> 	I understand, that you dislike FQ (you have not been shy or subtle about that position), but the issue at hand is really an issue about L4S's proposed schemes inherent lack of safety.

[BB] You will have seen that I have now put FQ on equal footing with 
DualQ in the latest L4S drafts, which I also explained in my status 
update talk in tsvwg at IETF-109.  Although FQ is contrary to the e2e 
argument and violates layering, I always see networking as a tussle 
between bell heads and net heads, and try to accommodate both where I can.

Bob

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 3 Dec 2020 16:40:59 +0000

Just on that last point, perhaps we should be clearer by what 
"experiment" means.

> Frankly, the sooner the WG understands and accepts that basic fact, the sooner we can move to a viable solution - one which does not redefine CE from the semantics established by RFC-3168 and RFC-8…

I think we need to be clear that I understand the proposal is to an 
Internet-wide experimental deployment.

If succesful, I'd expect we'll discover things in the deployment that 
will make us consider new things, before this is approved as a PS. The 
WG might later decide to obsolete RFC-3168, or L4S, or neither, or both. 
That will be to decide in future. In persepective, about 15 years passed 
before the previous EXP use of ECT(1) was made made historic, enabling this.

Gorry

-- 
G. Fairhurst, School of Engineering

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Thu, 3 Dec 2020 09:43:44 -0800 (PST)

[ Charset UTF-8 unsupported, converting... ]
> before the previous EXP use of ECT(1) was made made historic, enabling this.

That 15 years is one reason the opponents of L4S are so very concerned
about the lack of solid, operable positive test data on the current
"state of art" L4S, aka the lack of a viable transport protocol that
COULD be depoloyed on the internet using the L4S middle box signalling.

If a mistake is made here in going forward with this experiment as
it exists today the failure recovery is a very long tail.

-- 
Rod Grimes                                                 [email-masked]

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 3 Dec 2020 21:38:55 +0200

> On 3 Dec, 2020, at 6:40 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
> I think we need to be clear that I understand the proposal is to an Internet-wide experimental deployment.

An Internet-wide experiment will inevitably need to deal with a lot of "legacy" networks which have never even heard of L4S, have no pressing desire to change their equipment to accommodate it, and might find it challenging (or at least costly) to do so even if they desired to.  This especially includes those networks which have already invested in deploying plain old ECN, most of whom will have done so relatively recently and are still depreciating those assets.

That's precisely why L4S needs to show that it is safe to deploy into legacy networks, which may depend on existing RFC-3168 assumptions for their operation, before it can be approved for an Internet-wide experimental deployment.  We have been trying to make that point for two years already, including through several of the issues (still open) raised at the previous L4S WGLC at Montreal.  Meanwhile, ECN deployment has been increasing (see Jake's data).

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 3 Dec 2020 22:36:19 +0100

Bob,

see [SM2] for comments below.

> On Dec 3, 2020, at 17:17, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> [BB] I don't know which school you learned networking at,

	[SM2] @chairs, @Bob, really? There is nothing technical in that question and nothing advancing our discussion or civility. 

> but passing information into a layer at one end or out of it at the other is how layers work. ECN was designed to be fully in line with layering

	[SM2] Is it? In an encrypted tunnel it can be easily argued, that passing information between inner and outer layers is a layering violation. And as far as I can tell that argument has been made and recognized. Sure there can be value in passing ECN codepoints between inner and outer layers, but it certainly is not strict maintenance of layers. I am sure a similar discussion can be had about flow labels of IPv6 in IPv6 tunnels, copying them from inner to outer on encapsulation can have value, but needs to be weighted against the risk of establishing an information exfiltration channel. But that is really a whole different discussion than the question why L4S deals badly with rfc3169 signalling. 

> 
> The IETF has no obligation to make functions work that violate layering (like FQ). Layer violations have to be considered as an optimization that works when the higher layer is accessible and doesn't when it isn't. Same with other similar mechanisms like firewalls. Remember FQ-CoDel is not standards track.

	[SM2] FQ is a red herring here, the issue is L4S's failure to equitably share over single queue rfc3168 bottlenecks. I have stated that explicity twice in the e-mail you relied to. As an a-side, last time I looked L4S' internet drafts also aim for experimental track, like e.g. FQ_CoDel...

> [BB] FQ's failure to isolate flows that end up in a tunnel together overrides any flows that are trying to be TCP-friendly and forces them not to be TCP-friendly.

	[SM2] How that? Inside each of stochastic FQs bins, flows can and will share the same congestion signal probability, just as they would do in a lower queue count AQM (say 2 queues). Whether a flow is TCP-friendly or not, is not a function of the bottleneck AQM, but of the flow's own congestion response. 

	The issue here is, to repeat for your education, that L4S's designed CC response is not coping fairly with rfc3168 CE marks in a TCP friendly fashion, nothing more and nothing less; FQ or no-FQ on the bottleneck. Except that for non-tunneled flows FQ can actually recover some level of equitable sharing in spite of L4S in-appropriate CC response to rfc3168 CEs. But if a tunnel hides the information that FQ needs to isolate flows that behavior regresses to the normal behavior for non-FQ rfc3168 AQMs, sure.
	If I look at it, FQ is not a complete solution for all network issues (and nobody claimed it was), but it helps the normal situation quite a lot. Especially if the FQ-AQM is close to the ISP to end-user transition, where tunneled traffic is going to be rare (or can be treated specially be the end-user).

> You don't seem to have grocked that there are always two sides to this argument. I put the other side and recognized your side.
> Then I gave the principle (layering) that sits as the judge between the two - determining the side that the IETF and the networking industry as a whole will always take.

	[SM2] That seems true, and pretty much orthogonal to the discussion we are having. 

> 
> You can't keep playing the game of dressing up this argument from only one viewpoint, as if that somehow proves the other viewpoint isn't valid.

	[SM2] Which view-point again? As far as I a concerned, you are still trying to edrail the discussion of L4S inherent lack of safety into a discussion about anything else. 

> [BB] An application endpoint is not in control of whether its flow is aggregated with others in a tunnel.

	[SM2] "Something" is the entity that encapsulated flows into a tunnel, obviously. But this is a bit theoretical, as none of the choke points that see large scale tunneled traffic will see any deployment of AQMs anytime soon anyway. But humor me and show how juniper, cisco and co. are close to releasing DualQ implementations for their big iron routers... the reason, why tunneling came up again is simply that arguably the rfc3168 AQM with widest deployment is fq_codel, and team L4S (not team FQ whoever that is) keeps claiming that FQ will make L4S' re-definition of the CE response "safe".; but in the light of tunneling that claim simply is not true. Also not really a reason to go off on an unprovoked rant about FQ and layer violations.

> 
> The whole of networking works by encapsulation - typically multiple layers of it. In networking, if you want network equipment to do something with a flow ID, you put another transport header inside the outer (typically UDP encapsulation). For instance, see how GUE ensures equal cost multipath routing works.

	[SM2] Again that seems to be orthogonal to the L4S issue. I still have a hard time believing, that ab ISP savvy enough for that level of traffic engineering at the same time would set up an AQM with known deficits to such encapsulated traffic an i would expect these tunnels to end at transitions between ASs, and not start at the ISP to customer edge. I will not enter a theoretical discussion about the pro and cons of tunneling in general, but restrict myself to the part where L4S demonstrates un-safe behavior. Existing tunnels, do exactly that, they exist already, but L4S is still in the pre-deploymet phase and can (and should) be fixed before roll-out is attempted.

> [BB] I worked with Jake to write-up all the pros and cons, which we put in the ecn-l4s-id draft after Jake approved it. You will see one pro and many, many cons.

	[SM2] You might have realized, that I do not agree with all/most of your judgement calls. These lists of yours tend to tell more about your "political" stance than about hard technical limits...

> [BB] When you produce any actionable content, please let us know.

	[SM2] Classy, I would say, classy. But also pretty much expected, by now I have seen your debating style often enough, to realize that this is a sign that you realize that your argument do not hold much water. The next step will likely be that you fall silent... and abandon this sub-thread, fine with me.

> [BB] You will have seen that I have now put FQ on equal footing with DualQ in the latest L4S drafts, which I also explained in my status update talk in tsvwg at IETF-109.  Although FQ is contrary to the e2e argument and violates layering, I always see networking as a tussle between bell heads and net heads, and try to accommodate both where I can.

	[SM2] Oh, FQ seems to be your personal nemesis, but leave me out of this, please. That is not my battle, out of experience I can tell you that FQ-AQMs at the end-customer ISP edge work pretty well; but I also already tried to express the issue that neither FQ-AQMs not L4S AQMs actually offer the targeted unfairness that end-users seem to desire once more approximate (as already claimed by dualpi2) or stricter (as in stochastic fair queueing) flow fairness does not solve the issues. In that case typically the important traffic that merits un-equal treatment does not have simple characteristics like shortest RTT but really depends on the user's judgement. 

	No, I am concerned the L4S as designed and implemented does not meet its promises and frankly does not work well enough to consider deployment for anything but the over-tested short RTT, low-hop count fast-track for which it was designed. I am also concerned that team L4S tries to paper over catastrophic failure in the declared reference AQM by adding an increasing list of requirements onto L4S compatible protocols, with actually enforcing the required behavior at run time. And as TCP Prague demonstrates the best effort of coming up with an L4S compatible transport protocol falls noticeably short of what can be considered a transport protocol for general purpose internet transport. This last point has been driven home prominently by Koen's proposal to have TCP Prague fall back to a CUBIC type response for RTTs > 80 ms... after fudging TCP Prague's CC response dynamics to make up for DualPI2's equitable sharing break-down at short RTTs. 

	How about you go and demonstrate safe, reliable and robust functionality of L4S AQM and transport protocol over the existing internet, before wasting more time on telling me about how networking should have been thought at my school, okay? Humor me and demonstrate that my concerns and objections are unfounded, and L4S is truly the best invention in networking since sliced bread.

Regards
	Sebastian

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 3 Dec 2020 17:17:51 -0500

(chair hat on)

I wish this thread was making progress in any way, but I don't see it, 
and it's become a bit heated again.

I think we can probably close this thread, since the scenario described 
is well understood, usual suspects have clearly expressed their 
thoughts, and going back and forth doesn't seem to be convincing to one 
another in any way.

FYI, there are 762 people on this mailing list, and we're aware that the 
tone getting too hot discourages some from participating in the 
conversation.  I *would* like to hear from others or anyone with new 
perspectives to contribute, rather than just the handful of people that 
we are hearing a lot from.

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 4 Dec 2020 01:00:21 +0200

On 3 Dec, 2020, at 11:36 pm, Sebastian Moeller <[email-masked]> wrote:

> …team L4S (not team FQ whoever that is) keeps claiming that FQ will make L4S' re-definition of the CE response "safe".; but in the light of tunneling that claim simply is not true. Also not really a reason to go off on an unprovoked rant about FQ and layer violations.

Just to put this completely to bed:

- If you remove FQ from the equation, and just have single-queue RFC-3168 AQMs, then L4S' problems get worse, not better.  Every instance, rather than only a limited subset of instances, of L4S sharing an AQM bottleneck with conventional traffic would be doing so in a shared queue, and we know perfectly well where that story ends (see Issue #16).  Therefore, FQ is not the cause of the problem, but a partial workaround; it works well when applicable, but there are scenarios where it does not apply, whence the behaviour is merely as if FQ were not present.

- If you remove AQMs from the equation entirely, then the coexistence problem goes away, but also L4S shows no benefits over the status quo, because it relies on an AQM to inform it of the bottleneck queue status.  Both L4S and conventional traffic would experience a lot of queue delay.

- If you remove L4S from the equation, then the problems associated with it go away completely.  Conventional traffic coexists with other conventional traffic just fine, whether in a shared queue or with FQ.  Therefore, L4S is the cause of the problem, because it is the single element that you can remove (while changing nothing else) to solve the problem.

In causal analysis, this is a no-brainer.  L4S, not FQ, is the problem.

With that high-level insight, it is then straightforward to drill deeper into the system and find the design element within L4S that is the *root* cause of the coexistence problems.  Everything leads back to the design decision to change the meaning of CE, slavishly following the (bad) example of DCTCP.  All subsequent design of L4S has been driven by the need to mitigate the consequences of that basic choice, and ultimately it has not worked - fundamentally because existing networks (as opposed to those specially prepared for L4S) see only the behaviour inherited from DCTCP.

In SCE, we made a different choice.  We chose from the outset to keep CE with its existing meaning, and even to use the existing ECE/CWR feedback mechanism for it.  For the high-fidelity signal we followed the suggestion in RFC-3168 of using the spare ECN codepoint, which seemed very natural, and we then had to decide on a feedback mechanism for it.  All of our subsequent work on SCE has been driven from the natural consequences of these early decisions, and some WG feedback on what behaviours are more or less desirable.  But it has resulted in a scheme that is inherently compatible with existing networks and existing traffic, unlike L4S.

Wes wrote:

> I wish this thread was making progress in any way, but I don't see it, and it's become a bit heated again.

I, and probably Sebastian too, feel some need to robustly counter disinformation when we see it.  That is unfortunately something we have to do rather a lot when L4S comes up.  Today's discussion about FQ is emblematic.

I agree with Wes in wishing that more people would get involved in this topic.  They should look at the test data that Pete, Jake and I have organised, maybe try to replicate our findings, and come to an informed opinion as to whether L4S (as a current WG item) remains worthwhile and/or can be salvaged somehow, or not.  They could also try some mathematical analysis to see what the theoretically predicted behaviour would be, and whether our tests reproduce that accurately.  Third-party validation (or repudiation) would be welcomed.

In connection with that, I would point to the lack of progress in addressing Issue #16 by technical means, in more than a year since it was first raised.  The heuristic bottleneck type detector was a valiant effort, but was not successful.  No better alternative has been put forward - except for changing the signalling mechanism, at least two versions of which the L4S team has flatly rejected.  I think we must consider the signalling mechanism to be a defining feature of L4S at this point, such that they share fates.

May I ask the Chairs whether a WGLC would be capable of de-adopting L4S from the WG agenda?  If so, then I might just remove my objection to having another WGLC for L4S.  At present, it seems like one of the most promising avenues for making concrete progress towards a worthwhile experiment in high-fidelity congestion control.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Thu, 03 Dec 2020 20:54:09 -0500

On Thu, 2020-12-03 at 16:40 +0000, Gorry Fairhurst wrote:
> > Frankly, the sooner the WG understands and accepts that basic fact,
> > the sooner we can move to a viable solution - one which does not
> > redefine CE from the semantics established by RFC-3168 and RFC-
> > 8511, and hence does not place burdens on networks which have no
> > interest in the L4S experiment.
> 

Have single network experiments been attempted? Have cooperating
peering network experiments been attempted? Why is an *Internet-wide*
experiment the appropriate next step, especially when potential
detrimental impacts to traffic flows of non-participating networks &
hosts have been identified?

> before the previous EXP use of ECT(1) was made made historic,

Is there a planned deadline to evaluate the results of the experiment,
and either advance to PS or revert the allocation of ECT(1)?

Regards,

// Steve

---

From: Michael Welzl <[email-masked]>
Date: Fri, 4 Dec 2020 10:08:13 +0100

Hi all,

I’m reacting to this email. I’ve been following this thread from a distance, mostly for its entertainment value (stuff like "I don't know which school you learned networking at”).

So, personally, I think that a decision to go forward should be grounded in data - and I see it as an asset for the group that there’s an opposing team which presents a *critical* analysis. This should weigh more than data presented by the proponents, unless it’s provably erroneous (using broken code or whatever).  I have to apologize for not consulting the list of issues - but, from these emails, it seems that there are two major remaining concerns:

1) RTT unfairness among L4S flows:

If I understood this correctly from emails, this issue has been addressed somewhat ok, albeit perhaps with a hack. Unless this is truly extreme (to the degree of making some flows almost entirely unusable), I wouldn’t worry too much about it, as it is a matter of shooting yourself in the foot if things work very poorly. The Internet has a way of self-regulating such too-aggressive behavior: when BIC was found to be a bit too aggressive, it was replaced with Cubic. Cubic used beta=0.8 for a while, which was found too aggressive, and now it uses 0.7. BBRv1 has received a lot of criticism for being too aggressive, but not much roll-out has been documented… and Google folks have been quick to try to address this with BBRv2. “The bully wins” isn’t really how congestion control tends to play out, perhaps because even a single host produces multiple flows, and people also live in households with multiple heterogeneous hosts.

2) Unfairness towards RFC-3168 style ECN:

This is a very different matter. Recently, there was some discussion of the slowly increasing 3168-style ECN deployment, and Bob has asked if these are all FQ cases (or else they might be irrelevant).
I don’t think that’s the right conversation to have. Imagine a scenario where L4S is deployed on some hosts feeding into a bottleneck, yet some other hosts don’t participate in the L4S experiment (e.g., these hosts could run a different OS). Now assume that the non-L4S-participating hosts want to try out 3168-style ECN **at this point in time** - and they find it performing terribly poor, worse than without using ECN, concluding that 3168-style ECN isn’t even worth trying. Then, the only possibility for improvement is to also participate in the L4S experiment - and what if this, then, also doesn’t hold its promises (e.g., because the only existing end system implementation doesn’t work well) ?   Won’t this lead to an overall conclusion that ECN isn’t worth it, and never has been?

I would also like to remind people that there were two planned experiments: L4S and ABE (RFC 8511). Code for ABE exists but it isn’t really turned on by anyone, as far as I know - perhaps because people are waiting for this discussion’s dust to settle before even trying something else with ECN. I don’t know. Anyway, IIRC, the plan was to allow both experiments to play out individually, without one eliminating the possibility of even trying the other.

In conclusion: I’m inclined to say “let’s test this”, but only if point 2 is convincingly addressed by L4S. That hasn’t been the case for the last, what, 5 years or so, and unless this data is based on wrong or broken code, it doesn’t convince me otherwise:  https://github.com/heistp/l4s-tests#unsafety-in-shared-rfc3168-queues <https://github.com/heistp/l4s-tests#unsafety-in-shared-rfc3168-queues>
IIRC this was also the original plan.

Cheers,
Michael

PS: my personal view on L4S, in general: the use case is thin. L4S is “sold” as minimizing latency, but this is also easily achieved by a tiny DropTail queue. Less easily, and with its own issues, this is also achieved by FQ (or variants thereof). The large majority of interactive latency-critical flows has a low rate (and even high-rate interactive traffic is probably crafted to be adaptive and work below the average capacity limit, to avoid a latency build-up). A short DropTail queue or FQ would work reasonably well for such flows. L4S can attain high throughput without building a large queue, which is great, of course — for non-interactive traffic. For non-interactive traffic, though, I don’t understand why an e2e codepoint is needed: put a PEP there, let it pull data as early as it can from the sender, use L4S with a DSCP across the bottleneck towards the receiver. Why is this not the L4S deployment model, at least while it’s an experiment? Is this just because the IETF doesn’t approve of PEPs?

> On Dec 3, 2020, at 11:17 PM, Wesley Eddy <[email-masked]> wrote:
> 
> (chair hat on)
> 
> I think we can probably close this thread, since the scenario described is well understood, usual suspects have clearly expressed their thoughts, and going back and forth doesn't seem to be convincing to one another in any way.
> 
> FYI, there are 762 people on this mailing list, and we're aware that the tone getting too hot discourages some from participating in the conversation.  I *would* like to hear from others or anyone with new perspectives to contribute, rather than just the handful of people that we are hearing a lot from.
> 
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 4 Dec 2020 12:38:48 +0200

> On 4 Dec, 2020, at 11:08 am, Michael Welzl <[email-masked]> wrote:
> 
> Imagine a scenario where L4S is deployed on some hosts feeding into a bottleneck, yet some other hosts don’t participate in the L4S experiment (e.g., these hosts could run a different OS). Now assume that the non-L4S-participating hosts want to try out 3168-style ECN **at this point in time** - and they find it performing terribly poor, worse than without using ECN, concluding that 3168-style ECN isn’t even worth trying. Then, the only possibility for improvement is to also participate in the L4S experiment - and what if this, then, also doesn’t hold its promises (e.g., because the only existing end system implementation doesn’t work well) ?   Won’t this lead to an overall conclusion that ECN isn’t worth it, and never has been?

In this situation, the effects would be seen by the non-L4S hosts even *before* they enabled ECN.  Here's an example of what they might encounter, noting that a Not-ECT CUBIC flow is involved here:

http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-cubic-noecn-pie-50Mbit-20ms_tcp_delivery_with_rtt.svg

For insight into this, consider that RFC-3168 AQMs are required to drop Not-ECT packets at the same rate as they mark ECT traffic.  That is, the decision to apply a congestion signal to a packet is made independently of its ECT status, and the latter only determines whether it can be a mark, or must be a drop.  The L4S traffic still looks like ECT packets (because RFC-3168 treats ECT1 as equivalent to ECT0), so the marking and dropping rates still go up in concert and the conventional traffic still gets squashed - only now it also has to perform retransmissions, so the goodput actually suffers a bit more than with ECN enabled.

The conclusion that a user or ISP (who isn't familiar with L4S but is simply trying to roll out ECN) might draw from this is that AQM is bad and unreliable, rather than ECN per se; they would actually see a slight improvement from enabling ECN at the endpoint, but a much bigger one from disabling the AQM.  This runs contrary to the established wisdom that AQM reduces latency (which is good) without having too much effect on throughput (which would be bad).

Another possible conclusion that a more paranoid network operator could draw is that ECN *in general* is a potential DoS attack vector and must therefore be blackholed.  That would be bad for *everyone* including L4S, and would also prevent reverting to RFC-3168 deployment upon failure of the L4S experiment.  I think that is a substantial risk that must also be weighed.

 - Jonathan Morton

---

From: Michael Welzl <[email-masked]>
Date: Fri, 4 Dec 2020 12:33:24 +0100

> On Dec 4, 2020, at 11:38 AM, Jonathan Morton <[email-masked]> wrote:
> 

Right; bad! But the inherent problem is the same: TCP Prague’s inability to detect the 3168-marking AQM algorithm. I thought that a mechanism was added, and then there were discussions of having it or not having it?  Sorry, I didn’t follow this closely enough.

> Another possible conclusion that a more paranoid network operator could draw is that ECN *in general* is a potential DoS attack vector and must therefore be blackholed.  That would be bad for *ever…

I agree with that.

Cheers,
Michael

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 4 Dec 2020 13:45:32 +0200

> On 4 Dec, 2020, at 1:33 pm, Michael Welzl <[email-masked]> wrote:
> 

Right, there was a heuristic added to TCP Prague to (attempt to) detect if the bottleneck was RFC-3168 or L4S.  In the datasets from around March this year, we showed that it didn't work reliably, with both false-positive and false-negative results in a variety of reasonably common scenarios.  This led to both the L4S "benefits" being disabled, and a continuation of the harm to conventional flows, depending on which way the failure went.

The code is still there but has been disabled by default, so we're effectively back to not having it.  That is reflected in our latest test data.

I believe the current proposals from L4S are:

1: Use the heuristic data in manual network-operations interventions, not automatically.

2: Have TCP Prague treat longer-RTT paths as RFC-3168 but shorter ones as L4S.  I assume, charitably, that this would be accompanied by a change in ECT codepoint at origin.

Those proposals do not seem very convincing to me, but I am just one voice in this WG.

 - Jonathan Morton

---

From: Michael Welzl <[email-masked]>
Date: Fri, 4 Dec 2020 12:48:59 +0100

> On Dec 4, 2020, at 12:45 PM, Jonathan Morton <[email-masked]> wrote:
> 

Yeah, so I have added my voice for this particular issue.

Cheers,
Michael

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 4 Dec 2020 12:32:22 +0000

Hi all,

to add one more option.

To be clearly any issues discussed only occurred if RCF3168-AQMs (without FQ) are deployed in the network (no matter if any traffic is actually RFC3168-ECN enabled or not). 

My understanding of the current deployment status is that only ECN-AQMs with FG support are deployed today and I don't think we should put a lot of effort in a complicated RFC3168-AQM detection mechanism which might negative impact the L4S experiment if we have no evidence that these queues are actually deployed.

Further I would like to note that RCF3168 AQMs are already negatively impacting non ECN traffic and advantaging ECN traffic. However, I think it's actually a feature of L4S to provide better performance that non-ECN and therefore providing a deployment incentive for L4S, as long as non-L4S is not starved entirely. We really, really must stop taking about fairness as equal flow sharing. That is not the reality today (e.g. video traffic can take larger shares on low bandwidth links and that keeps it working) and it is not desired at all because not all flows are equal!!! The endpoints know what is required to make their application work and as long as there is a circuit breaker that avoids complete starving or collapse, the evolution of the Internet depends on this control in the endpoint and future applications that potentially have very different requirements. Enforcing equal sharing in the network hinders this evolution.

I also would like to note that L4S is not only about lower latency. Latency is the huge problem we have in the Internet today because the current network was optimized for high bandwidth applications, however, many of the critical things we do on the Internet today actually is more sensitive to latency. This problem is still not fully solved, event hough smaller queues and AQM deployments are a good step in the right direction. L4S goes even further and the point is not only about reducing latency but to enable the deployment of a completely new congestion control regime with takes into account all the lessons learnt from e.g. data center deployment where we not have to be bounded by today's limitation of "old" congestion controls and co-existence. L4S is exactly a way to transmission to this new regime without starving "old" traffic but there also need to be incentives to actually move to the new world. That's what I would like to see and why I'm existed about L4S.

Mirja

    Cheers,
    Michael

---

From: Michael Welzl <[email-masked]>
Date: Fri, 4 Dec 2020 14:27:08 +0100

Hi,

A few answers in line:

> On Dec 4, 2020, at 1:32 PM, Mirja Kuehlewind <[email-masked]> wrote:
> 
> Hi all,
> 
> to add one more option.
> 

If that’s the case, I would say that it would be a more reasonable sequence to first move RFC 3168 to historic.
If we can’t find consensus on this, then how can there be consensus to go out and deploy a mechanism that essentially starves traffic which uses this other mechanism?

> Further I would like to note that RCF3168 AQMs are already negatively impacting non ECN traffic and advantaging ECN traffic. However, I think it's actually a feature of L4S to provide better performance that non-ECN and therefore providing a deployment incentive for L4S, as long as non-L4S is not starved entirely.

Starving entirely, or something *very* close to that, is what’s on the table - unless these plots were made with broken code or completely invalid scenarios for some reason:
https://github.com/heistp/l4s-tests#unsafety-in-shared-rfc3168-queues <https://github.com/heistp/l4s-tests#unsafety-in-shared-rfc3168-queues>

Cheers,
Michael

> We really, really must stop taking about fairness as equal flow sharing. That is not the reality today (e.g. video traffic can take larger shares on low bandwidth links and that keeps it working) and it is not desired at all because not all flows are equal!!! The endpoints know what is required to make their application work and as long as there is a circuit breaker that avoids complete starving or collapse, the evolution of the Internet depends on this control in the endpoint and future applications that potentially have very different requirements. Enforcing equal sharing in the network hinders this evolution.
>

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 4 Dec 2020 14:38:35 +0100

Hi Mirja,

more below in-line, prefixed with [SM].

> On Dec 4, 2020, at 13:32, Mirja Kuehlewind <mirja.kuehlewind=[email-masked]> wrote:
> 
> Hi all,
> 
> to add one more option.
> 

	[SM] So you seem to reject the tunnel argument, Could you please elaborate why tunnels seem ignorable in this context, but are a big argument for re-defining CE? These two positions seem logically hard to bring into alignment. 

> 
> Further I would like to note that RCF3168 AQMs are already negatively impacting non ECN traffic and advantaging ECN traffic.

	[SM] You must mean that rfc3168 enabled flows do not suffer the other-wise required retransmission after a drop and get a slightly faster congestion feed-back? That is a rather small benefit over non-ECN flows, but sure there is a rationale for ECN usage.

> However, I think it's actually a feature of L4S to provide better performance that non-ECN and therefore providing a deployment incentive for L4S,

	[SM] There is a difference between performing better by doing something better and "making the existing traffic artificially slower" but that is what L4S does (it works on both ends), stacking the deck against non-L4S traffic.

> as long as non-L4S is not starved entirely.

	[SM] Please define what exactly you consider "starved entirely" to mean otherwise this is not helpful.

> We really, really must stop taking about fairness as equal flow sharing.

	[SM] Yes, the paper about the "harm" framework that Wes posted earlier seems to be a much better basis here than a simplistic "all flows need to be strictly equal" strawmen criterion.

> That is not the reality today (e.g. video traffic can take larger shares on low bandwidth links and that keeps it working)

	[SM] You wish, please talk to end users that want at the same time use concurrent video streaming and jitter sensitive on-line gaming over their internet access link, and the heroic measures they are willing to take to make this work. It is NOT working as desired out of the box in spite of DASH video being adaptive and games requiring very little traffic in comparison. The solution would be to switch video over to CBR type of streaming instead of the current burtsy video delivery (but that is unlikely to change).

IMHO L4S will not change much here because a) it still aims to offer rough per flow fairness (at least for flows with similar network paths) and b) the real solution is controlled/targeted un-fairness where a low latency channel is carved out that works in spite of other flows not cooperating (L4S requires implicit coordination/cooperation of all flows to achieve its means, which is to stay civil optimistic).

> and it is not desired at all because not all flows are equal!!!

	[SM] Now, if only we had a reliably and robust way to rank flows by importance that is actually available at the bottleneck link we would be set. Not amount of exclamation marks is going to solve that problem, that importance of flows is a badly defined problem. If our flows cross on say our shared IPS's transit uplink, which is more important? And who should be the arbiter of importance, you, me, the ISP, the upstream ISP? That gets complicated quickly, no?

> The endpoints know what is required to make their application work and as long as there is a circuit breaker that avoids complete starving or collapse, the evolution of the Internet depends on this control in the endpoint and future applications that potentially have very different requirements. Enforcing equal sharing in the network hinders this evolution.

	[SM] Well, the arguments for equal-ish sharing are:
a) simple to understand, predict and measure/debug (also conceptually easy to implement).
b) avoids starvation as best as possible as evenly as possible
c) is rarely pessimal (and almost never optimal), often "good enough".

Compare this with your proposed "anything goes" approach (which does not reflect the current internet which seems mostly rough equitable sharing in nature)
a) extremely hard to make predictions unless the end point controls all flows over the bottleneck
b) has not inherent measures against starvation
c) Has the potential to be optimal, but that requires a method to rate relative importance/value of each packet that rarely exist at the points of congestion. 

How should the routers at a peering point between two AS know, which of the flows in my network I value most? Simply they can't and hence will not come up with the theoretically optimal sharing behavior. I really see no evolutionary argument for anything goes here.

> 
> I also would like to note that L4S is not only about lower latency. Latency is the huge problem we have in the Internet today because the current network was optimized for high bandwidth applications, however, many of the critical things we do on the Internet today actually is more sensitive to latency. This problem is still not fully solved, event hough smaller queues and AQM deployments are a good step in the right direction.

	[SM] Mirja, L4S offer really only very little advancement over the state of the art AQMs, 5ms average queueing delay is current reality, L4S' 1 ms (99.9 quantile) queueing delay really will not change much here, yes 1ms is smaller than 5ms, but please show a realistic  scenario where that difference matters.

> L4S goes even further and the point is not only about reducing latency but to enable the deployment of a completely new congestion control regime with takes into account all the lessons learnt from e.g. data center deployment where we not have to be bounded by today's limitation of "old" congestion controls and co-existence.

	[SM] I do smell second system syndrome here. Instead of aiming for a revolution, how about evolving the existing CCs instead? The current attempts at making DCTCP fit for the wider internet in the guise of TCP Prague are quite disappointing in what they actually deliver. To be blunt TCP Prague demonstrates quite well that the initial assumption DCTCP would work well over the internet if only it was safe to do so was simply wrong. The long initial ramp up time and the massively increased RTT-bias as well as the failure to compete well with cubic flows in FIFO bottlenecks are clear indicators that a new L4S reference transport protocol needs to be developed.

> L4S is exactly a way to transmission to this new regime without starving "old" traffic but there also need to be incentives to actually move to the new world. That's what I would like to see and why I'm existed about L4S.

	[SM] That is a procedural argument that seems to take L4S's promises at face value, while ignoring all the data that demonstrate L4S has still a long way to go to actually deliver on its promises. 
	I also do not believe it to be an acceptable way to create incentives by essentially making existing transport protocols perform worse (over L4s controlled bottlenecks). But that is what L4S does.

Best Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 8 Dec 2020 21:29:52 +0000

Steven,

> hosts have been identified?

[BB] As I understand it, until the IETF assigns ECT(1) to the L4S 
experiment, no-one should be deploying L4S on the public Internet, 
whether in single networks, or pairs of peered networks. This confines 
us to experiments where one can only try to generate realistic traffic.

Or do you mean a single *isolated* network (or peered *isolated* 
networks)? If so, isn't that what DCTCP has been doing for some years now?

Put another way, if implementers/operators started deploying L4S on 
ECT(1), even in single networks but within the public Internet, wouldn't 
that mean that ECT(1) had become de facto assigned to L4S?

Bob

---

From: Steven Blake <[email-masked]>
Date: Tue, 08 Dec 2020 18:00:40 -0500

> networks)? If so, isn't that what DCTCP has been doing for some years

Yes, I mean isolated network(s). IETF is not a regulatory agency. It
cannot prevent interested operators from conducting experiments *within
the confines of their own networks* using whatever signaling they want
(hint: there already is a field defined with an experimental range of
values allocated for steering packets into queues).

Has DCTCP already been tested using Dual-Q outside of the lab or
datacenters?

> that mean that ECT(1) had become de facto assigned to L4S?

L4S experimentation *within the confines of participating operator
networks* is not dependent on assignment of ECT(1). This has been true
and obvious for years. Every counter argument that I have seen against
using experimental DSCPs to signal L4S traffic in these experiments (at
least initially) has been predicated on ease of widespread deployment,
which should not even be an issue at this stage of experimentation.

The safe and sane thing to do in these initial experiments is to use an
experimental DSCP value to signal L4S/TCP Prague packets and use ECT(1)
to signal congestion in the Dual-Q low-latency queue, avoiding
ambiguity with CE signaling. 

Regards,

// Steve

---

From: Greg White <[email-masked]>
Date: Wed, 9 Dec 2020 21:56:16 +0000

Sebastian,

As usual, there was a lot of hyperbole and noise in your response, most of which I'll continue to ignore.  But, I will respond to this:

This underscores a perennial misunderstanding of the benefits of L4S.  It isn't about average latency.  Current "state of the art" AQMs result in P99.99 latencies of 40-120 ms (depending on the traffic load), compared against ~6 ms at P99.99 for L4S.  For a cloud-rendered game or VR application, P99.99 packet latency occurs on average once every ~4 seconds (of course, packet latency may not be iid, so in reality high latency events are unlikely to be quite that frequent).   Since RTTs of greater than 20ms have been shown to cause nausea in VR applications, this is a realistic scenario where the difference *really* matters.

Put another way, if, of that 20 ms motion-to-photon budget for VR, a generous 10 ms is allowed for queuing delay (leaving only 10 ms for motion capture, uplink propagation, rendering, compression, downlink propagation, decompression, display), current "state of the art" AQMs would result in 10-50% packet loss (due to late packet arrivals) as opposed to <0.001% with L4S.

-Greg

On 12/4/20, 6:38 AM, "tsvwg on behalf of Sebastian Moeller" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Mirja,

    Best Regards
    	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 10 Dec 2020 00:08:37 +0100

Greg,

> On Dec 9, 2020, at 22:56, Greg White <[email-masked]> wrote:
> 
> Sebastian,
> 

	[SM] I believe you might not have read that post, what in there is so hyperbolic that it offended your good taste? My predictions of how L4S is going t behave are all extrapolations from data (mostly Pete's, some from team L4S), if you believe these to be wrong, please show how and why.  

> This underscores a perennial misunderstanding of the benefits of L4S.  It isn't about average latency.  Current "state of the art" AQMs result in P99.99 latencies of 40-120 ms (depending on the traffic load),

	[SM] This is not what I see on my link, sorry. Here are the cake statistics from running 3 concurrent speedtests on three different devices (one of the speedtests exercised 8 bidirectional concurrent flows each marked with one of the CS0 to CS7 diffserv codepoints to exercise all of cake's priority tins).

qdisc cake 80df: dev pppoe-wan root refcnt 2 bandwidth 31Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 88 
 Sent 1692154603 bytes 4053117 pkt (dropped 2367, overlimits 3194719 requeues 0) 
 backlog 38896b 28p requeues 0
 memory used: 663232b of 4Mb
 capacity estimate: 31Mbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       88 /    1542
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh       1937Kbit       31Mbit     7750Kbit
  target          9.4ms        5.0ms        5.0ms
  interval      104.4ms      100.0ms      100.0ms
  pk_delay       48.8ms       25.0ms       12.5ms
  av_delay        8.1ms        4.7ms        3.7ms
  sp_delay        1.4ms        349us        311us
  backlog         3036b       28348b        7512b
  pkts            48019      3879642       127851
  bytes        19893008   1594898953     80834953
  way_inds            0        93129          167
  way_miss            5        80699          678
  way_cols            0            4            0
  drops               0         2367            0
  marks             262         1322          405
  ack_drop            0            0            0
  sp_flows            1            7            2
  bk_flows            1            8            2
  un_flows            0            0            0
  max_len          1492         1492         1492
  quantum           300          946          300

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ---------------- 
 Sent 8865345923 bytes 7688386 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80e0: dev ifb4pppoe-wan root refcnt 2 bandwidth 95Mbit diffserv3 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 50 mpu 88 
 Sent 8862364695 bytes 7686372 pkt (dropped 1971, overlimits 11290382 requeues 0) 
 backlog 64156b 43p requeues 0
 memory used: 567Kb of 4750000b
 capacity estimate: 95Mbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       88 /    1542
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh       5937Kbit       95Mbit    23750Kbit
  target          5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms
  pk_delay        2.0ms        9.2ms        102us
  av_delay        1.1ms        4.5ms         15us
  sp_delay          4us        522us         10us
  backlog            0b       64156b           0b
  pkts            71084      7607860         9442
  bytes       102481496   8762307523       556904
  way_inds            0       223028            0
  way_miss          177        77358          264
  way_cols            0            3            0
  drops              18         1953            0
  marks               0         1014            0
  ack_drop            0            0            0
  sp_flows            1           17            1
  bk_flows            0            8            0
  un_flows            0            0            0
  max_len          1492         1492          638
  quantum           300         1514          724

Not only does the average delay (av_delay) in the non Bulk (cake's name for the scavenger background class) below 5ms, no even, the peak delay (P100 in your nomenclature) stays well below your predicted 40 to 120ms. These statistics come from an uptime of 8 hours (my ISP reconnects my PPPoE session one per day, at which point the stats get cleared). Now that is an anecdotal data point, but at least it is a real data point. 

> compared against ~6 ms at P99.99 for L4S.  

	[SM] I accept your claim, but want to see real data with e.g. real speedtests as artificial load generators over the real existing internet, not simulated runs in a simulated network. 

> For a cloud-rendered game or VR application, P99.99 packet latency occurs on average once every ~4 seconds (of course, packet latency may not be iid, so in reality high latency events are unlikely to be quite that frequent).  

	[SM] Well, cloud rendered games are already a commercial reality, so the state of the art can not be so terrible that people are not willing to play.

> Since RTTs of greater than 20ms have been shown to cause nausea in VR applications, this is a realistic scenario where the difference *really* matters.
> 

	[SM] Greg, I believe you are trying to paraphrase John Carmack (see for example https://danluu.com/latency-mitigation/, would have been nice if you included a citation). There he also says "A total system latency of 50 milliseconds will feel responsive, but still subtly lagging." And that 50ms goal is well within reach with the state of the art AQMs. But even 20 ms is within reach if the motion tracking is done predictively (think Kalman filters) in which case network delays can be speculated over.

	But the fact alone that we have to go that deep into the weeds of an obscure example like VR-with-network-rendering to find a single example where L4S's claimed (not realistically proven yet) low queueing delay might be relevant is IMHO telling. Also do you have data showing your claim  "current "state of the art" AQMs would result in 10-50% packet loss (due to late packet arrivals)* as opposed to <0.001% with L4S" is actually true for the existing L4S AQM and transport protocols. To me that reads like a best case prediction and not like hard data. 
         One could call framing predictions as if they were data as "noise" and "hyperbolic", if one would be so inclined... But humor me, and show that these are numbers from real tests over the real internet... 

IMHO the fact that the best you can offer is this rather contrived VR-in-the-cloud example** basically demonstrates my point that L4S offers only very little over the state of the art, but does so at a considerable cost. 

Best Regards
	Sebastian

*) That is not how packet loss works. A late packet is not a lost packet, and in this example displaying a frame that is 20+Xms in the past is better than not changing the frame at all. 

**) Which has a very simple solution, don't render VR via the cloud, the won 10ms processing time can be put to good use for e.g. taking later motion data into account for tighter integration between somatosensation and visual inputs... I understand why you resort to this example, but really, this is simply a problem, where cutting out the network completely is the best solution, at least for motion sensor integration....

---

From: Greg White <[email-masked]>
Date: Thu, 10 Dec 2020 20:41:59 +0000

I'm not sure why I take the bait on these...

Sebastian,

You asked for a realistic scenario where a dramatic reduction in latency variation matters, and I gave you one (well, two).  Just because you don't see the value in improving latency and latency variation doesn't mean there is no value.  There is a strong interest by many in this group (and the broader networking industry) in solutions to Internet latency / latency variation, and high-fidelity congestion signaling is a key and necessary component.  Does it immediately solve all sources of latency? No.  Is TCP Prague (or BBRv2, or SCReAM), as currently implemented, perfect? No.  Is there an issue with RFC3168 interoperability?  Some say yes, some say no.  

But we've heard from a number of very experienced transport protocol and congestion control researchers who see L4S congestion signaling as a feasible path, and we've heard from a large number of network operators who also see it as a feasible path and are willing to make the investment in deploying it.  

Also:

50ms is not the industry target for MTP latency for VR, it is generally 20ms or less.
https://developer.oculus.com/learn/bp-rendering/ 
https://www-file.huawei.com/-/media/CORPORATE/PDF/ilab/cloud_vr_oriented_bearer_network_white_paper_en_v2.pdf 
https://www.cs.unc.edu/~zhengf/ISMAR_2014_Latency.pdf

In isochronous applications, late packets generally *do* mean lost packets.  Current VR gear runs at 90 fps, and cloud games run at up to 120 fps. A 100ms latency spike can mean 9-12 frame times with no data received, and then 9-12 frames arriving back-to-back.  Adaptive dejitter buffers can reduce packet drop rate, but only at the expense of added (and variable) delay.  Either way it is a degradation.

To Wes's point, I'm going to try to leave it there on this thread rather than encouraging even more back and forth.  Don't interpret the lack of further response as agreement.

-Greg

On 12/9/20, 4:08 PM, "Sebastian Moeller" <[email-masked]> wrote:

    Greg,

                       Bulk  Best Effort        Voice

                       Bulk  Best Effort        Voice

    Best Regards
    	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 11 Dec 2020 00:05:08 +0100

Hi Greg,

> On Dec 10, 2020, at 21:41, Greg White <[email-masked]> wrote:
> 

	[SM] You might not realize it, but instead of the "moral high ground" for which you might be aiming, statements like this reveal that your objectivity in this matter is not really that much different from how you probably perceive mine to be. I might not be the most civil discussion partner, but I can and will be convinced by facts and real data, by pure statements without supporting data, not so much.  

> 
> Sebastian,
> 
> You asked for a realistic scenario where a dramatic reduction in latency variation matters,

	[SM] Well going from 5 to 1 is not dramatic, sorry. But other than that, I am happy to agree that latency and latency variation do matter. That is why participated early on in the bufferbloat effort (at a very minor position mainly testing and reporting) and why I have been operating a latency reducing AQM on my home networks for the last 10 years.

> and I gave you one (well, two).  Just because you don't see the value in improving latency and latency variation doesn't mean there is no value.

	[SM] That is not what I argued. I argued that the additional  latency variation reduction that L4S brings to the table is simply not worth the cost L4S incurs. That is different from claiming it has no value. If we lived in a pure over-buffered FIFO world with expected queueing delays in the multiple 100ms range, I would welcome L4S and what it offers, but that simply is not the reality anymore, we are talking about average queueing delays of ~5ms that L4S needs to compete with.

>  There is a strong interest by many in this group (and the broader networking industry) in solutions to Internet latency / latency variation, and high-fidelity congestion signaling is a key and necessary component.  

	[SM] I keep repeating that the interest in HFCC seems to be the main reason why the idea of granting RFC status to the L4S drafts is still entertained, in spite of 
a) the almost complete lack of relevant testing data from team L4S (I include ableLabs here after revisiting the Low Latency DOCSIS specs) and 
b) the fact that the reference protocol is still in early development and admittedly not nearly  finished (evident in Koen's recent proposal to change TCP Prague to switch die a CUBIC like response for RTT >= 80ms, with that numeric  cut-off RTT value apparently purely driven by the fact that Pete's tests use 80 ms as one of the test RTTs) and 
c) the fact that the reference AQM, unlike other AQMs, requires active participation of the un-finished transport protocols to achieve the rough equitable sharing the L4S drafts seem to promise, hinting that the coupling design might be terminally unsafe/flawed.

> Does it immediately solve all sources of latency? No.  Is TCP Prague (or BBRv2, or SCReAM), as currently implemented, perfect? No.  

	[SM] Look Greg, the issue is not that the L4S transport(s) are not polished enough, but that e.g. TCP Prague clearly is notbaked-enough to be considered ready for deployment, just look at its performance in the most likely bottleneck system a flw might encounter on the open internet,  the dreaded FIFO: https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667

Look, how TCP Prague is clearly not useful in a world where it encounters a FIFO, as it severely throttles itself if it competes with CUBIC (independent of RTT). And neither QUIC, nor BBRv2 nor SCReAM actually attempt to implement any/all of L4S' protocol requirements and hence can not even be realistically tested during the planned experiment. Unless you ignore your own requirements and treat them to the LL queue without making sure they are compliant, but at that point you invalidate your argument that you somehow need RFC status before experimentation can start (because if you are willing to ignore these RFC during the actual experiment it seems unclear why you needRFC status at all).

> Is there an issue with RFC3168 interoperability?  Some say yes, some say no.  

	[SM] Well, for anybody still in the "no" camp, please have a look at https://camo.githubusercontent.com/42b2427b18a22c9c2f181942fa9fbd08e5b678606f8b3f9c75262e4dfd6d4f1c/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f6c34732d73362d726663333136382f6c34732d73362d726663333136382d6e732d7072616775652d76732d63756269632d6e6f65636e2d66715f636f64656c5f31715f2d35304d6269742d32306d735f7463705f64656c69766572795f776974685f7274742e737667
	It is a clear fact that there are severe "RFC3168 interoperability issues" unless you want to use a different term for TCP Prague unfairly driving competing non-L4S flows into starvation.

	We could be of different opinion on how to evaluate that fact (at least as an intellectual exercise), but the interoperability issue is not a matter of opinion.

> But we've heard from a number of very experienced transport protocol and congestion control researchers who see L4S congestion signaling as a feasible path, and we've heard from a large number of n…

	[SM] And all of the above have so far refrained from answering the explicit question I posed several times that they show the data that they have that convinced them that L4S is safe to deploy and will actually deliver on its promises over the existing internet. Sure in lieu of data we might resort to asking experts on predictions and opinions, but here we are talking about an issue that can and should be answered by experiments, data and facts. 

> https://developer.oculus.com/learn/bp-rendering/ 

	[SM] This is pretty thin on details or references, but since Carmack worked at Occulus no wonder we see his 20ms number pop up again...

> https://www-file.huawei.com/-/media/CORPORATE/PDF/ilab/cloud_vr_oriented_bearer_network_white_paper_en_v2.pdf 

	[SM] Mmmh, that is pretty slick and interestin (also pretty short on real experimental data). Interestingly they actually describe how the network should evolve to deliver immersive VR from the cloud that might actually work reliably, as it aims for carving out reserved rate channels to ultimately deliver sensor data to cloud and cloud rendered frames to the display with reliable real-time guarantees. IMHO maybe a bit over engineered, but certainly something that can work robustly and reliably. But that is quite a different beast than L4S...
So, while meh on the 20ms claim (no supporting data just listed as a number), +1 for coming up with a reliable way to assure RT delivery of data that seems not based on wishful thinking.

> https://www.cs.unc.edu/~zhengf/ISMAR_2014_Latency.pdf

	[SM] Believe it or not, I have actually cited a reference where the magic 20ms came from...  But none of this is actual peer-reviewed psychophysics research making the whole 20ms or less a bit of a cargo cult thing... Now there is actual research out there that confirms some of that (especially in the context of nausea induced by commercial/military flight simulators), but I will not waste time on doing somebody else's literature reseach.

> 
> In isochronous applications, late packets generally *do* mean lost packets.  

	[SM] Well, sure, but VR rendering is not necessarily an isochronous application, unless you make it such, and for in-the-cloud-rendering I am not sure that structuring your application with hard-realtime requirements is that great a design. 

> Current VR gear runs at 90 fps, and cloud games run at up to 120 fps. A 100ms latency spike

	[SM] See my links data, where the peak delay in the high priority tier one would use for real-time-ish data was 12.5ms (measured over all flows in that tier, not all flows will actually have encountered that level of delay), not 100ms, with average delay at 3.7ms. Why are we talking about a 100ms spike again?

> can mean 9-12 frame times with no data received, and then 9-12 frames arriving back-to-back.  Adaptive dejitter buffers can reduce packet drop rate, but only at the expense of added (and variable) delay.  Either way it is a degradation.

	[SM] Sure, but again, if you need hard realtime guarantees, do not outsource your rendering to compute units connected over a best effort "bus" (network). Really, that is a typical, "doctor it hurts when I do X" type of situation, with the obvious solution, "don't do X then" ;)

> To Wes's point, I'm going to try to leave it there on this thread rather than encouraging even more back and forth.  Don't interpret the lack of further response as agreement.

	[SM] ??? That ties in nicely with your patronizing introductory statements, but what it really shows that you are out of real arguments. I note that you simply ignored the parts in my response where I supported my arguments with actual data (and kept using numbers)... We could have an actual discussion instead....

Best Regards
	Sebastian