Subject: [iccrg] L4S related activity in 3GPP

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 9 Nov 2019 16:00:11 +0000

Gentlepeople!

This 3GPP SA2 contribution is recently submitted

https://www.3gpp.org/ftp/tsg_sa/WG2_Arch/TSGS2_136_Reno/Docs/S2-1911250.zip

Title “5QI values for efficient handling of congestion control marked IP
packets”

The contribution introduces L4S to the SA2 audience and explains where the
technology is useful. The contribution also proposes a method to bake L4S
into the 5G QoS framework.

If agenda time permits, it will be discussed at the SA WG2 meeting in Reno
on Nov 18-22, hopefully with good results. 

Also a RAN2 contribution was submitted and discussed at an earlier RAN2
meeting.

https://www.3gpp.org/ftp/TSG_RAN/WG2_RL2/TSGR2_107bis/Docs/R2-1913888.zip

Enjoy

/Ingemar

================================

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

Network Protocols & E2E Performance

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked]

www.ericsson.com

      The best way to find out if you 

  can trust somebody is to trust them               

              Ernest Hemingway

=================================

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 9 Nov 2019 18:13:12 +0100

Dear Ingemar,

thanks for posting this. I read "This document describes an Active Queue Management mechanism, ‘L4S’, that can support low-latency services in a way that does not degrade the performance of other traffic." and I wonder whether 3GPP has run any experiments to confirm/support this claim, that you are willing to share?

Sidenote: in the powerpoint file it says about L4s " Standardized in IETF [1]", which seems premature, in the process of being standardized seems a better description of the status quo...

I ask about test data from your side as the recent test data is quite sobering from a number of angles, including quality of isolation:

	Looking at the current L4S tests from the various testbeds I see severe breakdown of the promised isolation between bulk and the "low-latency" queue, apparently making the current implementation, if not the concept itself IMHO unfit for release into the wider internet, before these issues have been sufficiently addressed. Especially the observation that the weighted scheduler built in as a safety mechanism in the dualQ aqm seems to trigger considerably more often than expected/predicted from reading the dualQ draft (giving 1/16 of the bandwidth to on of two classes _might_ be justified as a temporary stop-gap measure in extremely unlikely cases, but this already triggers in a simple one-flow-per queue scenario at low RTTs). In short it currently looks like the "does not degrade performance of other traffic" part is not yet finished. 

	To illustrate my criticism, here is a link to a test comparing the simplest functionality test imaginable, running one L4S-style and one non-L4S-style TCP connection through the L4S dual queue AQM at a intranet like RTT (nomnally 0ms, but in reality):
https://l4s.cablelabs.com/l4s-testing/key_plots/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.png
As can be seen the L4S flow (labeled prague) ends up very quickly taking ~90% of the available bandwidth, leaving only 10% to the non-L4S flow (labeled cubic), wit increasing RTT this bias gets less extreme, but even at 80ms the prague flow still gets around 1.5 times the bandwidth of the cubic flow (https://l4s.cablelabs.com/l4s-testing/key_plots/batch-l4s-s1-2-cubic-vs-prague-50Mbit-80ms_var.png), indicating an RTT dependent performance degradation for "other traffic". Test from another testbed confirm the break-down of fairness between L4S and non-L4S traffic at short RTTs (https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s1-2/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_fixed.png).

	Personally, I can not help thinking that it might be better to first make sure there is a complete, real-world tested implementation that reliably demonstrates that dualQ delivers the required isolation between the two traffic classes and that the L4S approach actually delivers the reported reliable low latency performance under adverse real-world conditions while not degrading the performance of other traffic, before trying to codify L4S into RFCs.

Best Regards
	Sebastian

> On Nov 9, 2019, at 17:00, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Gentlepeople!
>  
> Title “5QI values for efficient handling of congestion control marked IP packets”
> The contribution introduces L4S to the SA2 audience and explains where the technology is useful. The contribution also proposes a method to bake L4S into the 5G QoS framework.
> If agenda time permits, it will be discussed at the SA WG2 meeting in Reno on Nov 18-22, hopefully with good results. 
>  
> Also a RAN2 contribution was submitted and discussed at an earlier RAN2 meeting.

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sat, 9 Nov 2019 10:04:01 -0800 (PST)

> Title ?5QI values for efficient handling of congestion control marked IP
> packets?
> 

Can you elaborate on the item "L4S adoption in the industry" from slide 2?
So far as I am aware there is not even a completed prototype
implementation of L4S.

And in slide 4 "Standardized in IETF [1]" is a premature statement
about the actual state of the L4S drafts.

In slide 5, the graphs, what are the blue line and the red line?
Are they simply 2 identical flow attempts?
Also no data on this graph about B or D values, though I can
speculate it is 250mb/s and 25 ms, but that falls out when
I look at the L4S forwad packet delay graph as the delay
there is below that of my 25mS derived D, which could not
occur.  Could you please provide a description of the tooling
data, and parameters so that we might independently validate
these graphs?

Regards,
Rod

> Labratoriegr?nd 11
> 971 28, Lule?, Sweden

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
Rod Grimes                                                 [email-masked]

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 9 Nov 2019 19:24:21 +0000

Hi Sebastian 

Please see inline below
/Ingemar

> -----Original Message-----
> From: Sebastian Moeller <[email-masked]>
> Sent: den 9 november 2019 18:13
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>
> Cc: [email-masked]; [email-masked]; [email-masked]; Ingemar Johansson S
> <[email-masked]>; [email-masked]
> Subject: Re: [tsvwg] L4S related activity in 3GPP
> 
> Dear Ingemar,
> 
> thanks for posting this. I read "This document describes an Active Queue
> Management mechanism, ‘L4S’, that can support low-latency services in a way
> that does not degrade the performance of other traffic." and I wonder whether
> 3GPP has run any experiments to confirm/support this claim, that you are willing
> to share?

[IJ] We have done simulation studies with our proprietary 4G/5G system simulation tools but I can not disclose the results from these.
> 
> Sidenote: in the powerpoint file it says about L4s " Standardized in IETF [1]",
> which seems premature, in the process of being standardized seems a better
> description of the status quo...

[IJ] Yes, you are right about this, it should have been "Subject to standardization..."

> 
> 
> I ask about test data from your side as the recent test data is quite sobering
> from a number of angles, including quality of isolation:

[IJ] First of all the outline of the proposal for L4S in 5G is to use separate radio bearers for L4S and classic (or normal or non-L4S) traffic. This means that it is the 5G QoS framework and the scheduling in the RAN (Radio Access Network) that dictates the fairness. But of course there are wireline transport link in the core network that may be using DualQ when that becomes commercially available. The bitrates over these links are however likely to be considerably higher than 50Mbps.

> 
> 	Looking at the current L4S tests from the various testbeds I see severe
> breakdown of the promised isolation between bulk and the "low-latency" queue,
> apparently making the current implementation, if not the concept itself IMHO
> unfit for release into the wider internet, before these issues have been
> sufficiently addressed. Especially the observation that the weighted scheduler
> built in as a safety mechanism in the dualQ aqm seems to trigger considerably
> more often than expected/predicted from reading the dualQ draft (giving 1/16
> of the bandwidth to on of two classes _might_ be justified as a temporary stop-
> gap measure in extremely unlikely cases, but this already triggers in a simple
> one-flow-per queue scenario at low RTTs). In short it currently looks like the
> "does not degrade performance of other traffic" part is not yet finished.
> 
> 	To illustrate my criticism, here is a link to a test comparing the simplest
> functionality test imaginable, running one L4S-style and one non-L4S-style TCP
> connection through the L4S dual queue AQM at a intranet like RTT (nomnally
> 0ms, but in reality):
> https://protect2.fireeye.com/v1/url?k=fcd5c73d-a05c68f4-fcd587a6-
> 0cc47ad93e1c-6acfac650abe7f09&q=1&e=6cd6c372-b712-46be-8b10-
> 22955dec7b38&u=https%3A%2F%2Fl4s.cablelabs.com%2Fl4s-
> testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.png
> As can be seen the L4S flow (labeled prague) ends up very quickly taking ~90% of
> the available bandwidth, leaving only 10% to the non-L4S flow (labeled cubic),
> wit increasing RTT this bias gets less extreme, but even at 80ms the prague flow
> still gets around 1.5 times the bandwidth of the cubic flow
> (https://protect2.fireeye.com/v1/url?k=77272802-2bae87cb-77276899-
> 0cc47ad93e1c-56d0590db27309ef&q=1&e=6cd6c372-b712-46be-8b10-
> 22955dec7b38&u=https%3A%2F%2Fl4s.cablelabs.com%2Fl4s-
> testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-80ms_var.png),
> indicating an RTT dependent performance degradation for "other traffic". Test
> from another testbed confirm the break-down of fairness between L4S and non-
> L4S traffic at short RTTs (https://protect2.fireeye.com/v1/url?k=5710f0ef-
> 0b995f26-5710b074-0cc47ad93e1c-418920d155a62bf0&q=1&e=6cd6c372-
> b712-46be-8b10-
> 22955dec7b38&u=https%3A%2F%2Fwww.heistp.net%2Fdownloads%2Fsce-l4s-
> bakeoff%2Fbakeoff-2019-09-13T045427-r1%2Fl4s-s1-2%2Fbatch-l4s-s1-2-
> cubic-vs-prague-50Mbit-0ms_fixed.png).

[IJ] I understand that the 0 RTT results can be because the congestion window is only a few segments large (my guess is ~3 to 4 segments ?). I know that this is a problem with e.g. DCTCP.  The question is whether this is a permanent problem or something that will be solved over time?. What is more interesting to me and what makes L4S attractive for the use cases outlined in the 3GPP contributions, is the very low queue delay with L4S. There are other promising features with L4S. One being that it can enable congestion control algorithms to quickly grab free capacity. 5G access can have large variations in link throughput for example related to dual connectivity and L4S can be helpful here as well.

> 
> 	Personally, I can not help thinking that it might be better to first make
> sure there is a complete, real-world tested implementation that reliably
> demonstrates that dualQ delivers the required isolation between the two traffic
> classes and that the L4S approach actually delivers the reported reliable low
> latency performance under adverse real-world conditions while not degrading
> the performance of other traffic, before trying to codify L4S into RFCs.
> 
> 
> Best Regards
> 	Sebastian
> 
> 
> > On Nov 9, 2019, at 17:00, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> > Gentlepeople!
> >
> > https://protect2.fireeye.com/v1/url?k=803d870e-dcb428c7-803dc795-0cc47
> > ad93e1c-77a60175f8460bc6&q=1&e=6cd6c372-b712-46be-8b10-
> 22955dec7b38&u=
> >
> https%3A%2F%2Fwww.3gpp.org%2Fftp%2Ftsg_sa%2FWG2_Arch%2FTSGS2_136
> _Reno%
> > 2FDocs%2FS2-1911250.zip Title “5QI values for efficient handling of
> > congestion control marked IP packets”
> technology is useful. The contribution also proposes a method to bake L4S into
> the 5G QoS framework.
> > https://protect2.fireeye.com/v1/url?k=06eb4ef7-5a62e13e-06eb0e6c-0cc47
> > ad93e1c-f387514c9f633599&q=1&e=6cd6c372-b712-46be-8b10-
> 22955dec7b38&u=
> >
> https%3A%2F%2Fwww.3gpp.org%2Fftp%2FTSG_RAN%2FWG2_RL2%2FTSGR2_1
> 07bis%2F
> > Docs%2FR2-1913888.zip
> >
> > Enjoy
> > /Ingemar

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 9 Nov 2019 21:38:00 +0100

Dear Ingemar,

more in-line below.

> On Nov 9, 2019, at 20:24, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Sebastian 
> 
> Please see inline below
> /Ingemar
> 

	[SM] Ah, I see, not posting internal data is fine, but by all means, go and perform real-world measurements as well with all the parts you actually want to roll-out (in other words testing with dctcp for example is scientifically interesting, unless you expect your users to actually use dctcp on your links).

> [IJ] Yes, you are right about this, it should have been "Subject to standardization..."

	[SM] Fair enough, as much as I regret to say it, standardization is quite likely.

> [IJ] First of all the outline of the proposal for L4S in 5G is to use separate radio bearers for L4S and classic (or normal or non-L4S) traffic. This means that it is the 5G QoS framework and the scheduling in the RAN (Radio Access Network) that dictates the fairness.

	[SM] Sounds like a workable approach, that side steps the dualQ implementation/theory issues.

> But of course there are wireline transport link in the core network that may be using DualQ when that becomes commercially available. The bitrates over these links are however likely to be considerably higher than 50Mbps.

	That does not matter, the issue is not the limit of 50 Mbps, but the fact that dualQ, at last the current implementation in conjunction with the current TCP prague implementation, simply do not reliably and robustly do what the rfc draft claims. My fear is that dualQ, with its weak coupling vie marking probabilities, simply is a design, that only guarantees approximate fairness. In other words I doubt that dualQ works as a naive reading of the RFC implies. But if I  understood Bob correct, the only gurantee he is willing to give is L4S does not starve non-L4S traffic. Which IMHO is not sufficient to merit inflicting the L4S experiment on the wider internet. Now, that is not my call to make, but the IETF process allows me t voice my concerns, and I just use this opportunity in an attempt to minimize undesired side-effects on the "normal" internet.

> [IJ] I understand that the 0 RTT results can be because the congestion window is only a few segments large (my guess is ~3 to 4 segments ?). I know that this is a problem with e.g. DCTCP.  The question is whether this is a permanent problem or something that will be solved over time?.

	[SM] With DualQ, I assume it is a structural problem, instead of designing this as a robust system with two fair sharing queues it does its clever "indirect coupling" via marking/dropping probabilities. Have a look at https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-09-13T045427-r1/l4s-s2-2/batch-l4s-s2-2-cubic-vs-prague-50Mbit-10ms_fixed.png for how sharing in a FQ system looks. Even with the leaky ECT(1) as classifier abuse it shoukd be simple to fair queue between EXT(1) and not ECH(1) flows and have robust separation. This is not rocket science, and mostly, as far as I can tell a solved problem.

> What is more interesting to me and what makes L4S attractive for the use cases outlined in the 3GPP contributions, is the very low queue delay with L4S.

	[SM] Well, I believe low queueing delay can be achieved without dualQ and without abusing the ECT(1) codepoint as a classifier (the desired classifier requires one complete independent bit, ECT(1) is just 1/2 of a bit, and hence has failure modes in separating L4S-style flows and non-L4S flows). The idea of using the finer-grained congestion signaling of DCTCP over the wider internet seems sane see the SCE draft proposal as an alternative that would/will allow dctcp-style ECN signaling over the internet in a truly backward compatible fashion...

> There are other promising features with L4S. One being that it can enable congestion control algorithms to quickly grab free capacity.

	[SM] When you say L4S, do you mean the congestion signaling? Then https://tools.ietf.org/html/draft-morton-taht-sce-00 should have you covered, without the nasty side-effects of the current L4S proposal. Or do you mean the actual congestion controller as implemented in TCP prague? 

> 5G access can have large variations in link throughput for example related to dual connectivity and L4S can be helpful here as well.

	[SM] Dealing better with variable rate links is certainly desirable, but I am not sure that DualQ/TCP Prague really are the best or only tools for that purpose. At least not until all of the kinks are ironed out. Again, none of this is my call to make, so all I am voicing is my educated subjective assessments.

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 9 Nov 2019 22:44:10 +0200

> On 9 Nov, 2019, at 10:38 pm, Sebastian Moeller <[email-masked]> wrote:
> 

Nitpick: the correct reference is now https://tools.ietf.org/html/draft-morton-tsvwg-sce-01 and significantly updated relative to that old version.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 9 Nov 2019 22:44:54 +0000

On 2019-11-09, 11:24, "Ingemar Johansson S" <ingemar.s.johansson=[email-masked]> wrote:
> One being that it can enable congestion control algorithms to
> quickly grab free capacity. 5G access can have large variations
> in link throughput for example related to dual connectivity and
> L4S can be helpful here as well.

Hi Ingemar,

Not sure how closely you've been following, but the L4S team recently
reported fixing a bug in their implementation that might change this
conclusion, and you probably should be aware of it if that's part of
the driver here.  Specifically the alpha was set to 1 instead of 0.
https://l4s.cablelabs.com/

Changing this to avoid some problematic behavior also causes a  more
gradual throughput ramp instead, so you might want to re-test your
conclusions about how fast it can converge on the fluctuating capacity
after that bug-fix is applied.

The charts show some example differences on startup throughput (along
with the problematic RTT response that was fixed by this).

With the fix:
https://l4s.cablelabs.com/l4s-testing/key_plots/batch-l4s-s6-1-prague-50Mbit-80ms_var.png

Without the fix:
https://l4s.cablelabs.com/l4s-testing/key_plots/batch-l4s-s6-prague_alpha-prague-50Mbit-80ms-alpha0_var.png

Cheers and HTH,
Jake

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 10 Nov 2019 10:02:40 +0000

Hi Jake, please see inline

> From: Holland, Jake <[email-masked]>
> Sent: den 9 november 2019 23:45
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Sebastian Moeller
> <[email-masked]>
> Cc: [email-masked]; [email-masked]; [email-masked]; Ingemar Johansson S
> On 2019-11-09, 11:24, "Ingemar Johansson S"
> > One being that it can enable congestion control algorithms to quickly
> > grab free capacity. 5G access can have large variations in link
> > throughput for example related to dual connectivity and L4S can be
> > helpful here as well.
>
> Hi Ingemar,
>
> reported
> fixing a bug in their implementation that might change this conclusion, and 
> you
> probably should be aware of it if that's part of the driver here. 
> Specifically the
> alpha was set to 1 instead of 0.
> https://protect2.fireeye.com/v1/url?k=5f1e4a97-03ca43f1-5f1e0a0c-
> 8610d8a762ca-aeefb3a777d636fc&q=1&e=e1f0f0ad-9613-4463-9a7f-
> a3b14a2b8481&u=https%3A%2F%2Fl4s.cablelabs.com%2F
>
[IJ] That is the initial alpha and AFAIK, alpha is used in a similar way to 
how it is used with DCTCP. i.e. dictates how much CWND is reduced on CE marks. 
It does therefore not affect the ramp-up when more capacity becomes available
https://github.com/L4STeam/linux/blob/testing/net/ipv4/tcp_prague.c

> Changing this to avoid some problematic behavior also causes a  more gradual
> throughput ramp instead, so you might want to re-test your conclusions about
> how fast it can converge on the fluctuating capacity after that bug-fix is 
> applied.
>
> The charts show some example differences on startup throughput (along with
> the problematic RTT response that was fixed by this).
>
> With the fix:
> https://protect2.fireeye.com/v1/url?k=6f13dc09-33c7d56f-6f139c92-
> 8610d8a762ca-a748163f08dce827&q=1&e=e1f0f0ad-9613-4463-9a7f-
> a3b14a2b8481&u=https%3A%2F%2Fl4s.cablelabs.com%2Fl4s-
> testing%2Fkey_plots%2Fbatch-l4s-s6-1-prague-50Mbit-80ms_var.png
>
> Without the fix:
> https://protect2.fireeye.com/v1/url?k=5bfc2d0a-0728246c-5bfc6d91-
> 8610d8a762ca-05cd77703d5d8069&q=1&e=e1f0f0ad-9613-4463-9a7f-
> a3b14a2b8481&u=https%3A%2F%2Fl4s.cablelabs.com%2Fl4s-
> testing%2Fkey_plots%2Fbatch-l4s-s6-prague_alpha-prague-50Mbit-80ms-
> alpha0_var.png
[IJ] This looks like the normal exit from slowstart to me. With an initial 
alpha=1.0 the congestion window is reduced by 50%, whereas in the initial 
alpha=0.0 it takes a few round trips for alpha to increase sufficiently to 
reduce the congestion window. In the alpha=1.0 the congestion controls enter 
congestion avoidance and Additive Increase applies. I see that the current TCP 
Prague code does not implement the novel paced chirping (please correct) and 
that can change the behavior a lot.

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 10 Nov 2019 10:20:11 +0000

Hi Sebastian
Please see inline

/Ingemar

> Sent: den 9 november 2019 21:38
> To: Ingemar Johansson S <[email-masked]>
> Cc: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; [email-masked];
> [email-masked]; [email-masked]; [email-masked]; Bob Briscoe
> ([email-masked]) <[email-masked]>
> > On Nov 9, 2019, at 20:24, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > Hi Sebastian
> >
> > Please see inline below
> > /Ingemar
> >
> >> Cc: [email-masked]; [email-masked]; [email-masked]; Ingemar Johansson
> >> S <[email-masked]>; [email-masked]
> >> thanks for posting this. I read "This document describes an Active
> >> Queue Management mechanism, ‘L4S’, that can support low-latency
> >> services in a way that does not degrade the performance of other
> >> traffic." and I wonder whether 3GPP has run any experiments to
> >> confirm/support this claim, that you are willing to share?
> >
> > [IJ] We have done simulation studies with our proprietary 4G/5G system
> simulation tools but I can not disclose the results from these.
> 
> 	[SM] Ah, I see, not posting internal data is fine, but by all means, go and
> perform real-world measurements as well with all the parts you actually want to
> roll-out (in other words testing with dctcp for example is scientifically
> interesting, unless you expect your users to actually use dctcp on your links).

[IJ] I hope to have data to show later on

> 
> 
> >>
> >> Sidenote: in the powerpoint file it says about L4s " Standardized in
> >> IETF [1]", which seems premature, in the process of being
> >> standardized seems a better description of the status quo...
> >
> >> I ask about test data from your side as the recent test data is quite
> >> sobering from a number of angles, including quality of isolation:
> >
> > [IJ] First of all the outline of the proposal for L4S in 5G is to use separate radio
> bearers for L4S and classic (or normal or non-L4S) traffic. This means that it is
> the 5G QoS framework and the scheduling in the RAN (Radio Access Network)
> that dictates the fairness.
> 

[IJ] The proposed approach for 5G is for practical reasons, we believe that it is more implementation and standardization friendly as it will utilize the existing QoS framework as much as possible. 

> 
> 
> > But of course there are wireline transport link in the core network that may be
> using DualQ when that becomes commercially available. The bitrates over these
> links are however likely to be considerably higher than 50Mbps.
> 
> 	That does not matter, the issue is not the limit of 50 Mbps, but the fact
> that dualQ, at last the current implementation in conjunction with the current
> TCP prague implementation, simply do not reliably and robustly do what the rfc
> draft claims. My fear is that dualQ, with its weak coupling vie marking
> probabilities, simply is a design, that only guarantees approximate fairness. In
> other words I doubt that dualQ works as a naive reading of the RFC implies. But
> if I  understood Bob correct, the only gurantee he is willing to give is L4S does
> not starve non-L4S traffic. Which IMHO is not sufficient to merit inflicting the
> L4S experiment on the wider internet. Now, that is not my call to make, but the
> IETF process allows me t voice my concerns, and I just use this opportunity in an
> attempt to minimize undesired side-effects on the "normal" internet.
> 
> >
> >>
> >> 	Looking at the current L4S tests from the various testbeds I see
> >> severe breakdown of the promised isolation between bulk and the
> >> "low-latency" queue, apparently making the current implementation, if
> >> not the concept itself IMHO unfit for release into the wider
> >> internet, before these issues have been sufficiently addressed.
> >> Especially the observation that the weighted scheduler built in as a
> >> safety mechanism in the dualQ aqm seems to trigger considerably more
> >> often than expected/predicted from reading the dualQ draft (giving
> >> 1/16 of the bandwidth to on of two classes _might_ be justified as a
> >> temporary stop- gap measure in extremely unlikely cases, but this
> >> already triggers in a simple one-flow-per queue scenario at low RTTs). In
> short it currently looks like the "does not degrade performance of other traffic"
> part is not yet finished.
> >>
> >> 	To illustrate my criticism, here is a link to a test comparing the
> >> simplest functionality test imaginable, running one L4S-style and one
> >> non-L4S-style TCP connection through the L4S dual queue AQM at a
> >> intranet like RTT (nomnally 0ms, but in reality):
> >> testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.p
> >> ng As can be seen the L4S flow (labeled prague) ends up very quickly
> >> taking ~90% of the available bandwidth, leaving only 10% to the
> >> non-L4S flow (labeled cubic), wit increasing RTT this bias gets less
> >> extreme, but even at 80ms the prague flow still gets around 1.5 times
> >> the bandwidth of the cubic flow
> >> testing%2Fkey_plots%2Fbatch-l4s-s1-2-cubic-vs-prague-50Mbit-80ms_var.
> >> png), indicating an RTT dependent performance degradation for "other
> >> traffic". Test from another testbed confirm the break-down of
> >> fairness between L4S and non- L4S traffic at short RTTs
> >> (https://protect2.fireeye.com/v1/url?k=5710f0ef-
> > [IJ] I understand that the 0 RTT results can be because the congestion window
> is only a few segments large (my guess is ~3 to 4 segments ?). I know that this is
> a problem with e.g. DCTCP.  The question is whether this is a permanent
> problem or something that will be solved over time?.
> 
> 	[SM] With DualQ, I assume it is a structural problem, instead of
> designing this as a robust system with two fair sharing queues it does its clever
> "indirect coupling" via marking/dropping probabilities. Have a look at
> https://protect2.fireeye.com/v1/url?k=53bd001b-0f37caa5-53bd4080-
> 869a17b5b21b-e4edb2934547a534&q=1&e=314d9ef9-5cc0-4c98-a599-
> 59f030e6cb2f&u=https%3A%2F%2Fwww.heistp.net%2Fdownloads%2Fsce-l4s-
> bakeoff%2Fbakeoff-2019-09-13T045427-r1%2Fl4s-s2-2%2Fbatch-l4s-s2-2-
> cubic-vs-prague-50Mbit-10ms_fixed.png for how sharing in a FQ system looks.
> Even with the leaky ECT(1) as classifier abuse it shoukd be simple to fair queue
> between EXT(1) and not ECH(1) flows and have robust separation. This is not
> rocket science, and mostly, as far as I can tell a solved problem.
> 
> > What is more interesting to me and what makes L4S attractive for the use
> cases outlined in the 3GPP contributions, is the very low queue delay with L4S.
> 
> 	[SM] Well, I believe low queueing delay can be achieved without dualQ
> and without abusing the ECT(1) codepoint as a classifier (the desired classifier
> requires one complete independent bit, ECT(1) is just 1/2 of a bit, and hence has
> failure modes in separating L4S-style flows and non-L4S flows). The idea of using
> the finer-grained congestion signaling of DCTCP over the wider internet seems
> sane see the SCE draft proposal as an alternative that would/will allow dctcp-
> style ECN signaling over the internet in a truly backward compatible fashion...
> 
> 
> > There are other promising features with L4S. One being that it can enable
> congestion control algorithms to quickly grab free capacity.
> 
> 	[SM] When you say L4S, do you mean the congestion signaling? Then
> https://tools.ietf.org/html/draft-morton-taht-sce-00 should have you covered,
> without the nasty side-effects of the current L4S proposal. Or do you mean the
> actual congestion controller as implemented in TCP prague?
> 
> 
> > 5G access can have large variations in link throughput for example related to
> dual connectivity and L4S can be helpful here as well.
> 
> 	[SM] Dealing better with variable rate links is certainly desirable, but I
> am not sure that DualQ/TCP Prague really are the best or only tools for that
> purpose. At least not until all of the kinks are ironed out. Again, none of this is
> my call to make, so all I am voicing is my educated subjective assessments.
> 
> 
> Best Regards
> 	Sebastian
> 
> >
> >>
> >> 	Personally, I can not help thinking that it might be better to first
> >> make sure there is a complete, real-world tested implementation that
> >> reliably demonstrates that dualQ delivers the required isolation
> >> between the two traffic classes and that the L4S approach actually
> >> delivers the reported reliable low latency performance under adverse
> >> real-world conditions while not degrading the performance of other traffic,
> before trying to codify L4S into RFCs.
> >>
> >>
> >> Best Regards
> >> 	Sebastian
> >>
> >>
> >> technology is useful. The contribution also proposes a method to bake
> >> L4S into the 5G QoS framework.
> >>> If agenda time permits, it will be discussed at the SA WG2 meeting
> >>> in Reno on
> >> Nov 18-22, hopefully with good results.
> >>>
> >>>      The best way to find out if you  can trust somebody is to trust
> >>> them
> >>>              Ernest Hemingway

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 10 Nov 2019 11:45:48 +0100

> On Nov 10, 2019, at 11:02, Ingemar Johansson S <[email-masked]> wrote:
> 

	[SM] has any one actually tested paced chirping (PC) over real-life bonded links? As far as I can see PC pretty much relies on a) little to no re-ordering of packets (weird given that L4S also wants to allow larger re-ordering tolerance by recommending/mandating the use of RACK) and b) that the packet delivery time of each packet reflects a common path characteristic (which is less true for a bonded link).
	IMHO this just another piece of insufficiently tested technology thrown into the hodgepodge that is L4S. The way I see it, L4S combined a number of clever ideas, that might work out in reality, but might also be "too clever" for the internet.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 10 Nov 2019 11:50:27 +0100

Hi Ingemar,

> On Nov 10, 2019, at 11:20, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Sebastian
> Please see inline
> 
> /Ingemar
> 

	[SM] Excellent, looking forward to that.

> [IJ] The proposed approach for 5G is for practical reasons, we believe that it is more implementation and standardization friendly as it will utilize the existing QoS framework as much as possible. 

	[SM] Well, what ever the rationale, by avoiding the insufficiently isolating DualQ AQM you will save yourself from a number of headaches; I simply assume that you implement stricter airtime sharing between the two bearers than dualQ, and if you err you err on the side of conserving normal traffic (unlike DualQ's "L4S first" approach). But you still are saddled with L4S poor choice of ECT(1) as heuristic to separate L4S from non-L4S traffic. Calling that aETC(1) classifier seems plain wrong, as classifier implies strict separation.

Best Regards
	Sebastian

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 13 Nov 2019 06:57:51 +0000

On 2019-11-10, 02:46, "Sebastian Moeller" <[email-masked]> wrote:
> [SM] has any one actually tested paced chirping (PC) over real-life
> bonded links? As far as I can see PC pretty much relies on a) little
> to no re-ordering of packets (weird given that L4S also wants to allow
> larger re-ordering tolerance by recommending/mandating the use of RACK)
> and b) that the packet delivery time of each packet reflects a common
> path characteristic (which is less true for a bonded link).

I'd certainly love to see paced chirping fleshed out.  If there's a robust
version, it seems a promising way to avoid overshooting on slow start for
just about everyone (so maybe we could all get rid of that latency spike
on scenario 5), and I'd think it could have applications even outside of
slow start anywhere you have a way to back off before pushing to overflow
(e.g. BBR, or something using SCE, as well as L4S).

Of course as the paper said, it's not ready yet, but it does also seem
potentially helpful for the use case of highly variable link throughput.
To me it seems worth keeping in mind and examining further, whether
or not you think L4S with ECT(1) will end up being viable.

Best,
Jake

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 13 Nov 2019 09:03:43 +0100

Hi Jake,

I guess my tone was more negative than it should have been, PC is a clever idea, and it would be great if it succeeds in the real world. More below in-line.

> On Nov 13, 2019, at 07:57, Holland, Jake <[email-masked]> wrote:
> 
> I'd certainly love to see paced chirping fleshed out.

	[SM] No objection. The promise of it is certainly interesting. What I would like to see first, before getting too excited is rigorous testing to demonstrate its robustness and functionality even under adverse conditions. Of the top of my head, here is a list of things that I would like to see tested:

1) robustness against bi-directional path saturation
2) robustness against asymmetric paths
3) robustness against bonded links (possibility of re-ordering and inter packet delay does not reflect a single patch characteristic) I believe paired-packet probes have similar issues.
4) effect of high prevalence of paced chirpers in the network (these are active probes, at what point do the probes of different flow start to interfere with each other and ist that problematic)
5) conceptually, if I understand correctly it actually measured return latency, while ideally it should measure one-way delays, what are the consequences of that mismatch.

Again, would great if this works reliably and robustly, but as far as I can see so far all we know is that it works under some conditions.

>  If there's a robust
> version,

	[SM] +1

> it seems a promising way to avoid overshooting on slow start for
> on scenario 5),

	[SM] In the new scenario 5? I guess there not re-defining CE would be the cleanest solution...

> and I'd think it could have applications even outside of

	[SM] Yes, that is why getting it tested under adverse conditions seems like an important next step!

> potentially helpful for the use case of highly variable link throughput.

	[SM] Excuse my ignorance, but in its proposed role as slow start replacement, it would only trigger rarely (on flow start-up and on certain time-out conditions, no)?

> or not you think L4S with ECT(1) will end up being viable.

	[SM] I apologize for lumping to many things together in the referenced email, it is just that the whole L4S set of concepts and drafts, IHO all could do with more thorough testing...

Best Regards
	Sebastian