Subject: Re: [tsvwg] [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

From: Bob Briscoe <[email-masked]>
Date: Tue, 19 Mar 2019 01:06:57 +0000

David,

On 17/03/2019 18:07, David P. Reed wrote:
>
> Vint -
>
> BBR is the end-to-end control logic that adjusts the source rate to 
> match the share of the bolttleneck link it should use.
>
> It depends on getting reliable current congestion information via 
> packet drops and/or ECN.
>
> So the proposal by these guys (not the cable guys) is an attempt to 
> improve the quality of the congestion signal inserted by the router 
> with the bottleneck outbound link.
>
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.

> THe cable guys are trying to get a "private" field in the IP header 
> for their own use.
>

There is nothing private about this codepoint, and there never has been. 
Here's some data points:

* The IP header codepoint in question (ECT(1) in the ECN field) was 
proposed for use as an alternative ECN behaviour in July 2105 in the 
IETF AQM WG and the IETF's transport area WG (which handles all ECN 
matters).
* A year later there followed a packed IETF BoF on the subject (after 2 
open Bar BoFs).
* Long discussion ensued on the merits of different IP header field 
combinations, on both these IETF lists, involving people active on this 
list (bloat), including Dave Taht, who is acknowledged for his 
contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of 
draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG and 
co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to 
lay out the process for reclaiming and reusing the necessary codepoints.
* This work and the process of freeing up codepoints has been very 
visible at every IETF ever since, with multiple drafts to fix other 
aspects of the protocols working their way through the IETF process in 
multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.

Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van 
Jacobson's first presentation of CoDel at the IETF in July 2012. VJ 
overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a paper
* Then I (working for BT) brought the work into the EU RITE project 
(Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from Alcatel 
Lucent and myself (I had just switched from BT to Simula), along with 
Olga Bondarenko (now Albisser), a PhD student at Simula who now works 
for Microsoft (on something else) and is still doing her PhD part-time 
with Simula
   o By that time, Al-Lu and Simula had a cool prototype running.
   o This was demonstrated publicly for the first time in the IETF AQM 
WG over DC+core+backhaul+DSL+home networks.
     https://riteproject.eu/dctth/#1511dispatchwg
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ 
packet from all the following simultaneous applications achieving ~1ms 
queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):
   o cloud-rendered remote presence in a racing car within a VR headset
   o the interactive cloud-rendered video already linked above
   o an online gaming benchmark
   o HAS video streaming
   o a number of bulk file downloads
   o a heavy synthetic load of web browsing

L4S has never been access-technology-specific. Indeed the cable industry 
has been funding my work at the IETF to help encourage a wider L4S 
ecosystem. There is nothing private to the cable industry in this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access 
technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the 
low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was 
published, with a fully working version to be used and abused.
    Now at: https://github.com/L4STeam/sch_dualpi2_upstream
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as 
well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
   o Ingemar Johansson from Ericsson was one of the proponents, focused 
on using L4S in LTE
   o along with Kevin Smith from Vodafone and
   o Praveen Balasubramanian from Microsoft (who maintains the Windows 
TCP stack, including DCTCP).
   o Ingemar has since written an open-source L4S variant of the SCReAM 
congestion controller for real-time media: 
https://github.com/EricssonResearch/scream/
   o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the 
necessary feedback in TCP https://github.com/mirjak/linux-accecn
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired 
me later in the year to help develop and specify it, along with support 
for L4S
   o In Jan 2019 the Low Latency DOCSIS spec was published and is now 
being implemented.
   o You can find the primary companies involved at the end of the White 
Paper. 
https://cablela.bs/low-latency-docsis-technology-overview-february-2019
   o Operators:
     Liberty Global
     Charter
     Rogers
     Comcast
     Shaw
     Cox Communications
o Equipment vendors
     ARRIS
     Huawei
     Broadcom
     Intel
     Casa
     Nokia
     Cisco
     Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the 
necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the 
moment
     (the developer wants to announce it themselves, so I have been 
asked not to identify them).

There's a lot more stuff been going on, but I've tried to pick out 
highlights.

All this is good healthy development of much lower latency for Internet 
technology.

I find it extremely disappointing that some people on this list are 
taking such a negative attitude to the major development in their own 
field that they seem not to have noticed since it first hit the 
limelight in 2015.

L4S has been open-sourced since 2016 so that everyone can develop it and 
make it better...

If I was in this position, having overlooked something important for 
multiple years, I would certainly not condemn it while I was trying to 
understand what it was and how it worked. Can I suggest everyone takes a 
step back, and suspends judgement until they have understood the 
potential, the goals and the depth of what they have missed. People who 
know me, know that I am very careful with Internet architecture, and 
particularly with balancing public policy against commercial issues. 
Please presume respect unless proven otherwise.

Best Regards

Bob

PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into 
the L4S queue. But only if it can keep latency down below around 1ms. 
Currently those ~15-25ms delay spikes would not pass muster. Indeed, its 
delay is not consistently low enough between the spikes either.

> -----Original Message-----
> From: "Vint Cerf" <[email-masked]>
> Sent: Saturday, March 16, 2019 5:57pm
> To: "Holland, Jake" <[email-masked]>
> Cc: "Mikael Abrahamsson" <[email-masked]>, "David P. Reed" 
> <[email-masked]>, "[email-masked]" 
> <[email-masked]>, "bloat" <[email-masked]>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] 
> Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> where does BBR fit into all this?
> v
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On 2019-03-15, 11:37, "Mikael Abrahamsson" <[email-masked]
>     <mailto:[email-masked]>> wrote:
>         L4S has a much better possibility of actually getting
>     deployment into the
>         wider Internet packet-moving equipment than anything being
>     talked about
>         here. Same with PIE as opposed to FQ_CODEL. I know it's might
>     not be as
>         good, but it fits better into actual silicon and it's being
>     proposed by
>         people who actually have better channels into the people
>     setting hard
>         requirements.
>
>         I suggest you consider joining them instead of opposing them.
>
>
>     Hi Mikael,
>
>     I agree it makes sense that fq_anything has issues when you're talking
>     about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
>     makes better sense there.
>
>     But fq_x makes great sense and provides real value for the uplink in a
>     home, small office, coffee shop, etc. (if you run the final rate limit
>     on the home side of the access link.)  I'm thinking maybe there's a
>     disconnect here driven by the different use cases for where AQMs
>     can go.
>
>     The thing is, each of these is the most likely congestion point at
>     different times, and it's worthwhile for each of them to be able to
>     AQM (and mark packets) under congestion.
>
>     One of the several things that bothers me with L4S is that I've seen
>     precious little concern over interfering with the ability for another
>     different AQM in-path to mark packets, and because it changes the
>     semantics of CE, you can't have both working at the same time unless
>     they both do L4S.
>
>     SCE needs a lot of details filled in, but it's so much cleaner that it
>     seems to me there's reasonably obvious answers to all (or almost
>     all) of
>     those detail questions, and because the semantics are so much cleaner,
>     it's much easier to tell it's non-harmful.
>
>     <aside regarding="non-harmful">
>     The point you raised in another thread about reordering is mostly
>     well-taken, and a good counterpoint to the claim "non-harmful relative
>     to L4S".
>
>     To me it seems sad and dumb that switches ended up trying to make
>     ordering guarantees at cost of switching performance, because if it's
>     useful to put ordering in the switch, then it must be equally
>     useful to
>     put it in the receiver's NIC or OS.
>
>     So why isn't it in all the receivers' NIC or OS (where it would render
>     the switch's ordering efforts moot) instead of in all the switches?
>
>     I'm guessing the answer is a competition trap for the switch vendors,
>     plus "with ordering goes faster than without, when you benchmark the
>     switch with typical load and current (non-RACK) receivers".
>
>     If that's the case, it seems like the drive for a competitive
>     advantage
>     caused deployment of a packet ordering workaround in the wrong network
>     location(s), out of a pure misalignment of incentives.
>
>     RACK rates to fix that in the end, but a lot of damage is already
>     done,
>     and the L4S approach gives switches a flag that can double as
>     proof that
>     RACK is there on the receiver, so they can stop trying to order those
>     packets.
>
>     So point granted, I understand and agree there's a cost to abandoning
>     that advantage.
>     </aside>
>
>     But as you also said so well in another thread, this is
>     important.  ("The
>     last unicorn", IIRC.)  How much does it matter if there's a
>     feature that
>     has value today, but only until RACK is widely deployed? If you were
>     convinced RACK would roll out everywhere within 3 years and SCE would
>     produce better results than L4S over the following 15 years, would
>     that
>     change your mind?
>
>     It would for me, and that's why I'd like to see SCE explored before
>     making a call.  I think at its core, it provides the same thing
>     L4S does
>     (a high-fidelity explicit congestion signal for the sender), but with
>     much cleaner semantics that can be incrementally added to congestion
>     controls that people are already using.
>
>     Granted, it still remains to be seen whether SCE in practice can match
>     the results of L4S, and L4S was here first.  But it seems to me
>     L4S comes
>     with some problems that have not yet been examined, and that are
>     nicely
>     dodged by a SCE-based approach.
>
>     If L4S really is as good as they seem to think, I could imagine
>     getting
>     behind it, but I don't think that's proven yet.  I'm not certain, but
>     all the comparative analyses I remember seeing have been from more or
>     less the same team, and I'm not convinced they don't have some
>     misaligned incentives of their own.
>
>     I understand a lot of work has gone into L4S, but this move to jump it
>     from interesting experiment to de-facto standard without a more
>     critical
>     review that digs deeper into some of the potential deployment problems
>     has me concerned.
>
>     If it really does turn out to be good enough to be permanent, I'm not
>     opposed to it, but I'm just not convinced that it's non-harmful,
>     and my
>     default position is that the cleaner solution is going to be better in
>     the long run, if they can do the same job.
>
>     It's not that I want it to be a fight, but I do want to end up
>     with the
>     best solution we can get.  We only have the one internet.
>
>     Just my 2c.
>
>     -Jake
>
>
>     _______________________________________________
>     Ecn-sane mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://lists.bufferbloat.net/listinfo/ecn-sane
>
>
> -- 
> New postal address:
> Google
> 1875 Explorer Street, 10th Floor
> Reston, VA 20190
>
> _______________________________________________
> Bloat mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/bloat

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Dave Taht <[email-masked]>
Date: Tue, 19 Mar 2019 04:18:56 +0100

On Tue, Mar 19, 2019 at 2:07 AM Bob Briscoe <[email-masked]> wrote:
> BBR is the end-to-end control logic that adjusts the source rate to match the share of the bolttleneck link it should use.
>
>
>
> It depends on getting reliable current congestion information via packet drops and/or ECN.
>
>
>
> So the proposal by these guys (not the cable guys) is an attempt to improve the quality of the congestion signal inserted by the router with the bottleneck outbound link.
>

I think the inventor of udp, and the e2e argument has good reason to
blow his top. I also think vint asked a reasonable question - has this
been tested vs bbr?

>
>
> THe cable guys are trying to get a "private" field in the IP header for their own use.
>
>
> There is nothing private about this codepoint, and there never has been. Here's some data points:

I hope https://github.com/dtaht/bufferbloat-rfcs/blob/master/sce/draft-morton-taht-tsvwg-sce.txt
gets comprehensively evaluated in comparison to the more private use
of ect_1 you are talking about,

> * The IP header codepoint in question (ECT(1) in the ECN field) was proposed for use as an alternative ECN behaviour in July 2105 in the IETF AQM WG and the IETF's transport area WG (which handles all ECN matters).

I called for the closure of that wg because it had become an endless
bikeshed. Since the aqm chairs refused to apply long honored
traditions of the ietf - like *running code*. I ended up forming a new
wg

https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/

I like my rules a lot better than what ended up going on here. I do
hope more folk join my wg.

> * A year later there followed a packed IETF BoF on the subject (after 2 open Bar BoFs).

>From which many, including myself, ran screaming. I actually quit the
ietf last prague, and went off to implement real code in real
products. Shipping. I lost track of the numbers after they cracked
10m.

> * Long discussion ensued on the merits of different IP header field combinations, on both these IETF lists, involving people active on this list (bloat), including Dave Taht, who is acknowledged for his contributions in the IETF draft.

I have called repeatedly that my name be removed from the l4s related
drafts, in private, and in public. I in no way endorsed this effort,
except as a (somewhat dubious) experiment.

> * That was when it was decided that ECT(1) was most appropriate.

Love the passive voice there. Everybody else had left the room.

> * The logic of the decision is written up in an appendix of draft-ietf-ecn-l4s-id.

Yea, read that. Lousy logic. My notes from that period said - "prob
won't work in a container/vm/network namespace. design an experiment
to test it when running code arrives." Running code never did.

None of this stuff has been subjected to rigor we put the aqms through
in the aqm group. There's no public ns2, or ns3 models, no public
implementations at all... and y'all are proposing to write tcp prague
from scratch at a hackathon this weekend?

come on.

> * David Black, one of the co-chairs of the IETF's transport area WG and co-author of both the original ECN and Diffserv RFCs, wrote RFC8311 to lay out the process for reclaiming and reusing the necessary codepoints.

yea, that was good. used that for wireguard and now SCE.

> * This work and the process of freeing up codepoints has been very visible at every IETF ever since, with multiple drafts to fix other aspects of the protocols working their way through the IETF process in multiple WGs (tsvwg, tcpm, trill, etc).

amazing levels of funding for that bit. Why on earth couldn't you hire
a decent hacker and get a version of tcpprague.c done 3 years ago?
You're Doing it at a hackathon next week? You know how much testing
even a simple change to tcp or and aqm goes through at google or
bufferbloat.net? Even with running code and widescale live testing?
I've been trying to analyze one tiny change to fq_codel on real
networks for close to a year now.... seems to work, so that's part of
the now SCE enabled repo here:

https://github.com/dtaht/fq_codel_fast

> * In fact my first presentation on it was scheduled directly after Van Jacobson's first presentation of CoDel at the IETF in July 2012. VJ overran by nearly 20mins leaving just 3 mins for my presentation.
> * Then I (working for BT) brought the work into the EU RITE project (Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
> * By 2015 the two main L4S proponents were Koen De Schepper from Alcatel Lucent and myself (I had just switched from BT to Simula), along with Olga Bondarenko (now Albisser), a PhD student at Simula who now works for Microsoft (on something else) and is still doing her PhD part-time with Simula
>   o This was demonstrated publicly for the first time in the IETF AQM WG over DC+core+backhaul+DSL+home networks.

with a contrived benchmark.

> * In May 2016, L4S was demonstrated at MultiMediaSystems'16 with /every/ packet from all the following simultaneous applications achieving ~1ms queuing delay or less over a 40Mb/s Internet access link (7ms base RTT):

not measuring any normal traffic. on a contrived, proprietary,
"bigbuckbunny" benchmark I have blasted for its misrepresentation of
how basic internet protocols work multiple times and suggested instead
that the enormous suite of tests we developed for pie/codel/fq_codel
at least be run. Your group has refused to do so.

Our flent and irtt benchmarks are GPL3, developed in public, with a
bog-std output format and terabytes of data published.They are
licensed that way so that they cannot be gamed.

Now that the code is finally starting to land everyone can go and run
some real benchmarks and form their own opinions.

>   o a heavy synthetic load of web browsing

Not one of these being independently reproducible or repeatable. No
source code, no means to verify.

>
> L4S has never been access-technology-specific. Indeed the cable industry has been funding my work at the IETF to help encourage a wider L4S ecosystem. There is nothing private to the cable industry in this:

This works on wifi? Got code?

> * Al-Lu used DSL as a use-case, but L4S was relevant to all the access technologies they supplied.
> * but BT's initial motivating use-case was to incrementally deploy the low queuing delay of DCTCP over BT's data centre interconnect networks.

Just in terms of deployment fq_codel, in free.fr *alone* is well above
a million units (as of 2015), and persistently in the  top rankings of
http://www.dslreports.com/speedtest/results/bufferbloat?up=1

You guys are at ZERO. Still.

> * In Jul 2016 the open-source Linux repo for the Coupled AQM was published, with a fully working version to be used and abused.

No open source developer can get near that code legally.

https://fsfe.org/activities/os/why-frand-is-bad-for-free-software.en.html

It's not open source with a frand patent. OSI statement here:

https://opensource.org/node/616

Period. I'd asked repeatedly that patent be lifted. crickets. I asked
nokia's contact for the ipr. crickets.

>    Now at: https://github.com/L4STeam/sch_dualpi2_upstream

I am delighted you finally got some code possibly worth reviewing by experts.

> * Of course, DCTCP was already open-sourced in Linux and FreeBSD, as well as available in Windows

Beating that patent took forever. I'm glad microsoft joined OIN.

>   o Ingemar Johansson from Ericsson was one of the proponents, focused on using L4S in LTE
>   o Praveen Balasubramanian from Microsoft (who maintains the Windows TCP stack, including DCTCP).
>   o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/

I have to note that google congestion control (gcc) won in the
marketplace and I have no idea why anyone tries with scream anymore. I
liked some bits of nada.

>   o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn

it does help to submit code to netdev for mainline consideration? this
repo has not had a commit since 2017.

> * In summer 2017 CableLabs started work on Low Latency DOCSIS, and hired me later in the year to help develop and specify it, along with support for L4S
>   o In Jan 2019 the Low Latency DOCSIS spec was published and is now being implemented.

Thought it was just another limited experiment til last month.
*really*. Not once in all our communications in the past 2 years did I
have the slightest clue what was up. Formed my own wg, under rules
true to the ietf's original spirit (
https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/ ) - made
progress, submitted a *goooood * draft, and then, wow, all kinds of
stuff started coming to light that nobody knew about.

>   o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019

>     Cox Communications

I think david's characterization of the "cable industry" being behind
this is accurate.

> * Magnus Westerlund of Ericsson with a team of others has written the necessary ECN feedback into QUIC
where is the code?

> * L4S hardware is also being implemented for hi-speed switches at the moment

>     (the developer wants to announce it themselves, so I have been asked not to identify them).

groovy.

>
> There's a lot more stuff been going on, but I've tried to pick out highlights.
>
> All this is good healthy development of much lower latency for Internet technology.

I look forward to benching it against normal traffic finally, esp
against bbr, and in a realistic scenario with fq_codeled home routers
(and sch_cake), and on wifi.

IMHO there's no L4S deployment plan that makes sense vs the existing
10-100m+ deployment of rfc3168 compliant fq_codel, but, now that
l4s/dualpi/tcppraguecode exists to test -  I started drafting some
experiments to finally take a look at it. Maybe I'll get a chance to
talk about those.

>
>
> I find it extremely disappointing that some people on this list are taking such a negative attitude to the major development in their own field that they seem not to have noticed since it first hit the limelight in 2015.
>
> L4S has been open-sourced since 2016 so that everyone can develop it and make it better...
>
> If I was in this position, having overlooked something important for multiple years, I would certainly not condemn it while I was trying to understand what it was and how it worked. Can I suggest everyone takes a step back, and suspends judgement until they have understood the potential, the goals and the depth of what they have missed. People who know me, know that I am very careful with Internet architecture, and particularly with balancing public policy against commercial issues. Please presume respect unless proven otherwise.
>
> Best Regards
>
>
>
> Bob
>
> PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get into the L4S queue. But only if it can keep latency down below around 1ms. Currently those ~15-25ms delay spikes would not pass muster. Indeed, its delay is not consistently low enough between the spikes either.
>
>
>
>
>
>
>
> Cc: "Mikael Abrahamsson" <[email-masked]>, "David P. Reed" <[email-masked]>, "[email-masked]" <[email-masked]>, "bloat" <[email-masked]>
> Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
> On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <[email-masked]> wrote:
>>
>> On 2019-03-15, 11:37, "Mikael Abrahamsson" <[email-masked]> wrote:
>>     L4S has a much better possibility of actually getting deployment into the
>>     wider Internet packet-moving equipment than anything being talked about
>>     here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
>>     good, but it fits better into actual silicon and it's being proposed by
>>     people who actually have better channels into the people setting hard
>>     requirements.
>>
>> disconnect here driven by the different use cases for where AQMs can go.
>>
>> seems to me there's reasonably obvious answers to all (or almost all) of
>> useful to put ordering in the switch, then it must be equally useful to
>> If that's the case, it seems like the drive for a competitive advantage
>> RACK rates to fix that in the end, but a lot of damage is already done,
>> and the L4S approach gives switches a flag that can double as proof that
>> But as you also said so well in another thread, this is important.  ("The
>> last unicorn", IIRC.)  How much does it matter if there's a feature that
>> produce better results than L4S over the following 15 years, would that
>> change your mind?
>>
>> making a call.  I think at its core, it provides the same thing L4S does
>> the results of L4S, and L4S was here first.  But it seems to me L4S comes
>> with some problems that have not yet been examined, and that are nicely
>> If L4S really is as good as they seem to think, I could imagine getting
>> from interesting experiment to de-facto standard without a more critical
>> opposed to it, but I'm just not convinced that it's non-harmful, and my
>> It's not that I want it to be a fight, but I do want to end up with the
>> [email-masked]

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 19 Mar 2019 11:07:35 +0200

It was suggested that I should copy this point here for more robust discussion.

> On 19 Mar, 2019, at 7:52 am, Greg White <[email-masked]> wrote:
> 
> L4S utilizes TCP Prague, which falls back to traditional congestion control if the bottleneck link doesn't provide isolation.  

You see, this is the part I find difficult to believe that it will operate reliably.  For a start, I have seen no detailed public description of TCP Prague, even though it has supposedly been in "open" development for so long.  My most recent information is that it's essentially a slightly modified DCTCP.

I did find this, which reads like a requirements spec rather than a working algorithm:

" It would take time for endpoints to distinguish classic and L4S ECN
  marking.  An increase in queuing delay or in delay variation would be
  a tell-tale sign, but it is not yet clear where a line would be drawn
  between the two behaviours.  "

Internet history is littered with failed attempts at implementing delay-sensitive TCPs.  I can immediately think of several reasons why delay can and will vary for reasons other than the bottleneck not implementing an isolated queue  (just ask the BBR devs).  The mere presence of a wifi link on the path, even if it is never the bottleneck, would be a trivial and common example.

So please explain (or point to good documentation) how TCP Prague robustly avoids misbehaving in a standard ECN environment, as is presently deployed.

SCE explicitly does not rely on specific changes in behaviour by endpoints.  It just provides a conduit of information from the network to the receiver, in addition to standard ECN behaviour.  The receiver is free to ignore that information, without harming the network, and will naturally behave normally and safely when that information is absent.  We have a proof-of-concept implementation (a trivial mod of sch_codel and sch_fq_codel) which successfully passes this information across the Internet and works with (is transparently ignored by) existing endpoints and middleboxes.

In short, SCE is incrementally deployable by design.

The broader system of feedback and modified congestion control, which I call ELR (Explicit Load Regulation) as an umbrella term, offers benefits which, yes, have not yet been proved - but which are straightforward in concept and should be amenable to analysis.  It seems likely that some work from L4S can be used in this context.

- Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Wed, 20 Mar 2019 00:59:29 +0100

On Tue, Mar 19, 2019 at 5:44 AM Greg White <[email-masked]> wrote:
>
> That is ridiculous.
>
>
>
> You clearly haven’t read the drafts, and so are speaking from a position of ignorance.  Please get informed before making statements like this.

I've read all the drafts, 3 times, and made pithy comments when they
came out. I've been trying to get other knowlegable people in the
daily grind of modern DC netops to read them also for years. I hope
more, now do.

>
>
> There is *absolutely* nothing cable-specific or “private” about L4S.  It is being developed in an open forum, the IETF!!   Yes, the cable industry is adopting L4S – because we recognized its potential.  Others are too, and anyone can.  It is a totally open spec, and has been since the initial drafts came out of the RITE project.  The cable industry was not involved in RITE (in fact the technology was first demonstrated on DSL equipment), and we learned about L4S when the rest of the world did.  We decided to become early adopters.  Yes, we were quiet about the fact that we were planning on adopting it (until now).
>
>
>
> If individuals drop out of participating in the IETF, they shouldn’t be upset if the IETF continues to make progress on developing Internet technology in their absence.  It seems pretty disingenuous for DT to form his own private working group to come up with an incompatible, and limited, alternative to the ongoing work in IETF, then spring it on the IETF and start this FUD war.

We announced it publicly back in august with our charter and goals on
bloat and aqm mailing lists.

Nobody spoke up then saying l4s was still alive either in the ietf or privately.

We published a charter and outline of scope:
https://www.bufferbloat.net/projects/ecn-sane/wiki/

And we formed into teams:
https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/

And published position papers.

6 months of work on dealing with ecn issues finally has as it's first
result, the SCE draft.

You calling me disingenuous is disingenuous.

>
>
> The craziest thing about this whole thread is that there is a heck of a lot in common between L4S and SCE.

Yep, they do!

Yes, I think we achieved consensus long ago that a single queued AQM
cannot work with an earlier indication of congestion than drop without
changing the endpoints. We also achieved consensus that an earlier
congestion notification was potentially useful in some respects.

Repurposing CE, and using up ECT_1, was all that was on the table before.

>More in common than different.  My initial belief was that we all had similar goals (eliminating buffering latency in the internet) and we’d be able to achieve a meeting of the minds on the best way to use ECT(1) to achieve it.

If I could merely have got the cable industry to reduce the buffering
in their cmts ends from 600+ms to 100ms in the last 8 years, I'd have
been happy.

Two things. 1)  I believe - along with just about everybody I know in
the DC business - few that have bothered to think of ecn at all -
(none have deployed it). In every conversation about l4s is the
presumption that the other person i the conversation already thinks
ecn is a good idea. I know a lot of people that don't think it is. Me,
when it comes to ECN I am firmly on the yellow team - chicken - see
above position paper - in general.

2) Bufferbloat.net deployed solutions starting in 2012 that work for
fixing both inbound and outbound buffering. We shipped sch_cake in
august after 3 years of deployment testing.  I would rather like y'all
to add sch_cake to your test matrix.

> Now, I’m not so sure what DT’s motivation is.

My motivation is always to eliminate bufferbloat and build a better
internet. It's the motto on my patreon page.

>
>
>
> If I can boil this down for the people who are jumping into this without reading the drafts:
>
>
>
> Both L4S and SCE are attempting to provide congestion-controlled senders with better congestion signals so that flows can achieve link capacity without buffering delay.
> Both are proposing to use ECT(1) as part of the mechanism, but to use it in different ways.

ECT(1) from a unwritten, specialized tcp that is incompatible with
BBR, Cubic, and Reno.

ECT(1) from a well established, shipping AQM.

> SCE’s usage of ECT(1) potentially allows an automatic fallback to traditional Cubic behavior if the bottleneck link is a single-queue classic-ECN AQM (do any of these exist?), whereas L4S will need to detect such a condition via RTT measurement

Strike "potentially". Does. Instead of writing further emails, I'm
focusing on patching tcptrace and xplot to show this clearly from
packet captures. Perhaps that will be ready by the hackathon.

> L4S’s usage of ECT(1) allows links to identify new senders and take advantage of new sender features like reordering tolerance that can further drive down latency in many common link technologies.

There's nothing in SCE that stops the exact same idea, except that the
behavior is chosen reciever side, not sender side.

> SCE will only work if the bottleneck link implements fq.  Some bottleneck network gear will not be able to implement fq or will not implement it due to its undesirable side effects (see section 6 of RFC 8290).

This is overbroad.

And I didn't write that section.

> L4S will work if the bottleneck link implements *either* fq or dual queue.

And this is not a demonstrated fact yet - *at all*  that needs to
yield to experiment and independent analysis.

That's a good start actually to writing up a good comparison document.
I'll save that and work on it, if
I get a talk slot for it.

>
>
> Beyond that, they are *very,very* similar.

Yes!

>
>
>
> But, L4S has been demonstrated in real equipment and in simulation, and leverages an existing congestion

Not under circumstances I can control. That's Not Science.

>controller that is available in Linux and Windows (with some tweaks).

SCE improves on an existing AQM that the default in roughly 100% of
linux deployments now, all the way up to the new RHEL8 release, which
was the last distro to adopt it, so far as I know. It doesn't require
a specialized tcp to use, either.

>  SCE leverages a paragraph in a draft that describes a first guess about how a congestion controller might work.

We were kind of in a hurry. 'round here, we usually write the code and
test it thoroughly for years, before submitting a draft.

We did part of this work originally with the original definition of
ce_threshold (back in 2013?) when we too had fiddled with DCTCP,
before pacing and BBR had arrived.

Running code is up on several servers now and we hope to have a live
test over the internet at ietf. Maybe tcp-fu will land. Hopefully
tcp-prague will land.

>
>
>
> L4S has defined a congestion feedback mechanism so that these congestion signals can get back to the sender.  SCE offers that “we’ll propose something later”.
>
>
>
> BBR currently does not listen to explicit congestion signals, but it could be updated to do so (for either SCE or L4S).
>
>
>
> -Greg
>
>
>
>
>
> From: Bloat <[email-masked]> on behalf of "David P. Reed" <[email-masked]>
> Date: Sunday, March 17, 2019 at 12:07 PM
> To: Vint Cerf <[email-masked]>
> Cc: bloat <[email-masked]>, "[email-masked]" <[email-masked]>
> Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
>
>
>
> Vint -
>
>
>

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 20 Mar 2019 11:17:11 +0100

Dear All,

> On Mar 20, 2019, at 00:59, Dave Taht <[email-masked]> wrote:
> 
>> [...]
[...]

It would be great if the L4S project maybe could help kick-start independent testing, by creating an sharing two VMs one with the appropriate client side patches and one with a L4S aware AQM (probably curvy RED to avoid the patent issue, assuming the patent does not cover curvyRED). So that it is easier to "kick" the tiers in a way that tests what the L4S project considers compliant clients/AQM. Personally I am interested to see how robust and reliable the  detection of non-L4S CE sources is and how well the L4S side of the AQM will tolerate CE-marked packets from non-L4S senders, or in other words how well the "isolation" works. And also how L4S endpoints will deal with SCE emitting AQMs on their path.
I admit that I have doubts that ECT(1), basically a single "constellation" of a 2-bit bitfield can serve as a replacement for a single independent bit in a single-bit bit-field, that seems required for real isolation of flows of different ECN-response types.

Best Regards
	Sebastian

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 20 Mar 2019 19:04:17 +0000

Hi Bob & Greg,

I agree there has been a reasonably open conversation about the L4S
work, and thanks for all your efforts to make it so.

However, I think there's 2 senses in which "private" might be fair that
I didn't see covered in your rebuttals (merging forks and including
Greg's rebuttal by reference from here:
https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )

Please note:
I don't consider these senses of "private" a disqualifying argument
against the use of L4S, though I do consider them costs that should be
taken into account (and of course opinions may differ here).

With that said, I wondered whether either of you have any responses that
speak to these points:

1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
patent-protected AQM scheduler.

I understand that l4s-id suggests the possibility of an alternate
scheme.  However, comparing the MUSTs of the section 5 requirements
with the claims made by the patent seems to leave no room for an
alternate that would not infringe the patent claims, unless I'm missing
something?

https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
https://patents.google.com/patent/US20170019343A1/en

2. the L4S use of the ECT(1) codepoint privileges the low latency use
case.

As Jonathan Morton pointed out, low latency is only one of several
known use cases that would be worthwhile to identify to an AQM
scheduler, which the L4S approach cannot be extended to handle:
- Minimize Latency
- Minimize Loss
- Minimize Cost
- Maximize Throughput

https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html

I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
operator tuning parameters for a dualq node can possibly allow for
tuning between minimizing loss and latency, as mentioned in section
4.1 of aqm-dualq, but since the signaling is bundled together, it can
only happen for one at a time, if I'm reading it right.

But more importantly, the L4S usage couples the minimized latency use
case to any possibility of getting a high fidelity explicit congestion
signal, so the "maximize throughput" use case can't ever get it.

Regards,
Jake

PS:
If you get a chance, I'm still interested in seeing answers to my
questions about deployment mitigations on the tsvwg list:
https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A

I'm not surprised if it slipped by unnoticed, there have been a lot of
emails on this.  But good answers to those questions would go a long way
toward easing my concerns about the urgency on this discussion.

PPS:
These issues are a bit sideways to my technical reasons for preferring
the SCE formulation of ECT(1), which have more to do with the confusing
semantics and proliferation of corner cases it creates for CE and ECE.
But I thought I’d ask about them since it seemed like maybe there was a
disconnect here.

From: Bob Briscoe <[email-masked]>
Date: 2019-03-18 at 18:07
To: "David P. Reed" <[email-masked]>, Vint Cerf <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>, bloat <[email-masked]>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

David,

Vint -

Some history:
    https://riteproject.eu/dctth/#1511dispatchwg<https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>

   Now at: https://github.com/L4STeam/sch_dualpi2_upstream<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
  o Ingemar has since written an open-source L4S variant of the SCReAM congestion controller for real-time media: https://github.com/EricssonResearch/scream/<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
  o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the necessary feedback in TCP https://github.com/mirjak/linux-accecn<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
  o You can find the primary companies involved at the end of the White Paper. https://cablela.bs/low-latency-docsis-technology-overview-february-2019<https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>

Best Regards

Bob

-----Original Message-----
From: "Vint Cerf" <[email-masked]><mailto:[email-masked]>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <[email-masked]><mailto:[email-masked]>
Cc: "Mikael Abrahamsson" <[email-masked]><mailto:[email-masked]>, "David P. Reed" <[email-masked]><mailto:[email-masked]>, "[email-masked]"<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>, "bloat" <[email-masked]><mailto:[email-masked]>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <[email-masked]<mailto:[email-masked]>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <[email-masked]<mailto:[email-masked]>> wrote:
    L4S has a much better possibility of actually getting deployment into the
    wider Internet packet-moving equipment than anything being talked about
    here. Same with PIE as opposed to FQ_CODEL. I know it's might not be as
    good, but it fits better into actual silicon and it's being proposed by
    people who actually have better channels into the people setting hard
    requirements.

Hi Mikael,

Just my 2c.

-Jake

_______________________________________________
Ecn-sane mailing list
[email-masked]<mailto:[email-masked]>
https://lists.bufferbloat.net/listinfo/ecn-sane<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>

Bloat mailing list

[email-masked]<mailto:[email-masked]>

https://lists.bufferbloat.net/listinfo/bloat<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>

--

Bob Briscoe                               http://bobbriscoe.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 20 Mar 2019 19:39:13 +0000

See below

On 20/03/2019, 19:04, Holland, Jake wrote:
> *From: *Bob Briscoe <[email-masked]>
> *Date: *2019-03-18 at 18:07
> *To: *"David P. Reed" <[email-masked]>, Vint Cerf <[email-masked]>
> *Cc: *tsvwg IETF list <[email-masked]>, bloat 
> <[email-masked]>
> *Subject: *Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] 
>     BBR is the end-to-end control logic that adjusts the source rate
>     to match the share of the bolttleneck link it should use.
>
>     So the proposal by these guys (not the cable guys) is an attempt
>     to improve the quality of the congestion signal inserted by the
>     router with the bottleneck outbound link.
>
> combinations, on both these IETF lists, involving people active on 
> this list (bloat), including Dave Taht, who is acknowledged for his 
> * David Black, one of the co-chairs of the IETF's transport area WG 
> and co-author of both the original ECN and Diffserv RFCs, wrote 
> RFC8311 to lay out the process for reclaiming and reusing the 
> necessary codepoints.
> * By 2015 the two main L4S proponents were Koen De Schepper from 
> Alcatel Lucent and myself (I had just switched from BT to Simula), 
> along with Olga Bondarenko (now Albisser), a PhD student at Simula who 
> now works for Microsoft (on something else) and is still doing her PhD 
> part-time with Simula
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
> * In May 2016, L4S was demonstrated at MultiMediaSystems'16 with 
> /every/ packet from all the following simultaneous applications 
> achieving ~1ms queuing delay or less over a 40Mb/s Internet access 
> link (7ms base RTT):
> L4S has never been access-technology-specific. Indeed the cable 
> industry has been funding my work at the IETF to help encourage a 
> wider L4S ecosystem. There is nothing private to the cable industry in 
> this:
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
> * In summer 2017 CableLabs started work on Low Latency DOCSIS, and 
> hired me later in the year to help develop and specify it, along with 
> support for L4S
>   o You can find the primary companies involved at the end of the 
> White Paper. 
> https://cablela.bs/low-latency-docsis-technology-overview-february-2019 <https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
>   o Operators:
>     Liberty Global
>     Charter
>     Rogers
>     Comcast
>     Shaw
>     Cox Communications
>    o Equipment vendors
>     ARRIS
>     Huawei
>     Broadcom
>     Intel
>     Casa
>     Nokia
>     Cisco
>     Videotron
> understand what it was and how it worked. Can I suggest everyone takes 
> a step back, and suspends judgement until they have understood the 
> potential, the goals and the depth of what they have missed. People 
> who know me, know that I am very careful with Internet architecture, 
> and particularly with balancing public policy against commercial 
> issues. Please presume respect unless proven otherwise.
>
> Best Regards
>
>
>
> Bob
>
> PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get 
> into the L4S queue. But only if it can keep latency down below around 
> 1ms. Currently those ~15-25ms delay spikes would not pass muster. 
> Indeed, its delay is not consistently low enough between the spikes 
> either.
>
>
>
>
>     From: "Vint Cerf" <[email-masked]> <mailto:[email-masked]>
>     To: "Holland, Jake" <[email-masked]> <mailto:[email-masked]>
>     Cc: "Mikael Abrahamsson" <[email-masked]>
>     <mailto:[email-masked]>, "David P. Reed" <[email-masked]>
>     <mailto:[email-masked]>, "[email-masked]"
>     <mailto:[email-masked]>
>     <[email-masked]>
>     <mailto:[email-masked]>, "bloat"
>     <[email-masked]> <mailto:[email-masked]>
>         <mailto:[email-masked]>
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
>
>
>     -- 
>
>     New postal address:
>
>     Google
>
>     [email-masked]  <mailto:[email-masked]>
>
>     https://lists.bufferbloat.net/listinfo/bloat  <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/  <https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
Concerning "Maximize Throughput", if you don't need scalability to very 
high rates, then is your requirement met by TCP-like semantics, as in 
TCP with SACK/loss or even better TCP with ABE/ECT(0)?

I wonder .... if the intent is to scale to really high rates, then the 
control loop delay for the congestion-controller becomes a limiting 
issue, and in that case low-latency is necessary to safely climb the 
rate to the high speed - and conversely to allow the controller to react 
quickly when (or if) that overshoots a capacity bottleneck. In other 
words, is scalable high throughput inseperable from low latency?

Gorry

---

From: Stephen Hemminger <[email-masked]>
Date: Wed, 20 Mar 2019 12:58:22 -0700

On Wed, 20 Mar 2019 19:04:17 +0000

> https://patents.google.com/patent/US20170019343A1/en

Has anyone done a detailed investigation for prior art?
The patent office does not do a good job of looking for prior art,
and the parties in the patent process are not motivated to look.

Other vendors often are not interested either because their own house of
cards built on patents of previous research might come falling down.

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 20 Mar 2019 20:00:13 +0000

On 2019-03-20, 12:40, "Gorry Fairhurst" <[email-masked]> wrote:
    Concerning "Maximize Throughput", if you don't need scalability to very 
    high rates, then is your requirement met by TCP-like semantics, as in 
    TCP with SACK/loss or even better TCP with ABE/ECT(0)?

[JH] A problem with TCP with BE/ECT(0) is that it gets at most 1 ECE signal
per round-trip, so the kind of high-fidelity congestion response I hope to
see out of some upcoming SCE-echo proposal would be very welcome, especially
in BBR (or similar), as well as anything that uses slow-start.

[JH] I agree lower latency, particularly including anything that avoids
buffer-bloat, is an important factor in returning a timely congestion
signal to the sender.

However, there may be a big difference in throughput between a CC that
allows for an increase of, say, 10-20ms, or +.5 base-RTT at a bottleneck,
vs. one that pushes back on anything above 1ms, especially when considering
paths with longer transit times.

In that sense of course a good bandwidth-maximizing approach benefits from
keeping latency low also, but perhaps with different thresholds.
    
    Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 20 Mar 2019 22:05:25 +0200

> On 20 Mar, 2019, at 9:39 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
> Concerning "Maximize Throughput", if you don't need scalability to very high rates, then is your requirement met by TCP-like semantics, as in TCP with SACK/loss or even better TCP with ABE/ECT(0)?

My intention with "Maximise Throughput" is to support the bulk-transfer applications that TCP is commonly used for today.  In Diffserv terminology, you may consider it equivalent to "Best Effort".

As far as I can see, L4S offers "Maximise Throughput" and "Minimise Latency" services, but not the other two.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 20 Mar 2019 20:05:09 +0000

On 2019-03-20, 12:58, "Stephen Hemminger" <[email-masked]> wrote:
    Has anyone done a detailed investigation for prior art?

I don't know, but for serendipitous reasons I recently came across this,
which may be interesting to those who would be interested in digging
further on that question:

http://www.statslab.cam.ac.uk/~frank/PAPERS/tac.pdf
"On packet marking at priority queues"
R. J. Gibbens and F. P. Kelly
IEEE Transactions on Automatic Control 47 (2002) 1016-1020.

I wasn't sure if it was technically prior art or not, and I'm offering
no opinion, but it sounded similar to me in some significant ways.

HTH.

-Jake

---

From: Greg White <[email-masked]>
Date: Wed, 20 Mar 2019 20:55:20 +0000

In normal conditions, L4S offers "Maximize Throughput" +  "Minimize Loss" + "Minimize Latency" all at once.  It doesn't require an application to have to make that false choice (hence the name "Low Latency Low Loss Scalable throughput").  

If an application would prefer to "Minimize Cost", then I suppose it could adjust its congestion control to be less aggressive (assuming it is congestion controlled). Also, as you point out the LEPHB could be an option as well.

What section 4.1 in the dualq draft is referring to is a case where the system needs to protect against unresponsive, overloading flows in the low latency queue.   In that case something has to give (you can't ensure low latency & low loss to e.g. a 100 Mbps unresponsive flow arriving at a 50 Mbps bottleneck).

-Greg

    

     - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Wed, 20 Mar 2019 21:48:58 +0000

I responded to point 2 separately.  In response to point 1, the dual queue mechanism is not the only way to support L4S and TCP Prague.  As we’ve mentioned a time or two, an fq_codel system can also support L4S and TCP Prague.  I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:

At dequeue:
If packet indicates ECT(1):
                If sojourn_time > L4S_threshold:
                                Set CE*, and forward packet
                Else:
                                Forward packet
Else:
                Do all the normal CoDel stuff

In many cases, all of the packets in a single queue will either all be ECT(1) or none of them will.  But, to handle VPNs (where a mix of ECT(1) and non-ECT(1) packets could share a queue), I would think that including the ECN field in the flow hash would keep those packets separate.

*a more sophisticated approach would be to mark CE based on a ramp function between a min_thresh and max_thresh, which could be implemented as a random draw, or via a counting function

From: Bloat <[email-masked]> on behalf of "Holland, Jake" <[email-masked]>
Date: Wednesday, March 20, 2019 at 1:06 PM
To: Bob Briscoe <[email-masked]>, "David P. Reed" <[email-masked]>, Vint Cerf <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>, bloat <[email-masked]>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

Hi Bob & Greg,

Please note:

Regards,
Jake

PS:

PPS:

David,

Vint -

Some history:

Best Regards

Bob

-----Original Message-----
From: "Vint Cerf" <[email-masked]><mailto:[email-masked]>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <[email-masked]><mailto:[email-masked]>
Cc: "Mikael Abrahamsson" <[email-masked]><mailto:[email-masked]>, "David P. Reed" <[email-masked]><mailto:[email-masked]>, "[email-masked]"<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>, "bloat" <[email-masked]><mailto:[email-masked]>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

Hi Mikael,

Just my 2c.

-Jake

Bloat mailing list

--

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 20 Mar 2019 23:56:19 +0200

> On 20 Mar, 2019, at 11:48 pm, Greg White <[email-masked]> wrote:
> 
> the dual queue mechanism is not the only way to support L4S and TCP Prague.  As we’ve mentioned a time or two, an fq_codel system can also support L4S and TCP Prague.  I’m not aware that anyone has implemented it to test it out yet (because so far most interest has been on dual-queue), but I think the simplest version is:
>  
> At dequeue:

The above looks remarkably similar to our proof-of-concept for an SCE middlebox.  Essentially:

At dequeue:
	Do all the normal Codel stuff
	If packet (still) carries ECT(0):
		If sojourn_time > SCE_threshold:
			Set SCE
	Forward packet

And yes, a ramp function between 0 and codel_target would work better.  The above gives us something to test against with minimum hassle.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 20 Mar 2019 22:38:33 +0000

Thanks Greg,

But wouldn’t this potentially violate at least one MUST from section 5.2 of l4s-id?

   The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST
   be roughly proportional to the square of the likelihood that it would
   have marked it if it had been an L4S packet (p_L)
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5.2

Maybe it depends on how far you stretch “roughly”, I guess...  I’m not sure, but I’d imagine some realistic conditions could provide counterexamples, unless there’s some reason I’m missing that prevents it?

Regards,
Jake

From: Greg White <[email-masked]>
Date: 2019-03-20 at 14:49
To: "Holland, Jake" <[email-masked]>, Bob Briscoe <[email-masked]>, "David P. Reed" <[email-masked]>, Vint Cerf <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>, bloat <[email-masked]>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

At dequeue:

Hi Bob & Greg,

Please note:

Regards,
Jake

PS:

PPS:

David,

Vint -

Some history:

Best Regards

Bob

-----Original Message-----
From: "Vint Cerf" <[email-masked]><mailto:[email-masked]>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <[email-masked]><mailto:[email-masked]>
Cc: "Mikael Abrahamsson" <[email-masked]><mailto:[email-masked]>, "David P. Reed" <[email-masked]><mailto:[email-masked]>, "[email-masked]"<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>, "bloat" <[email-masked]><mailto:[email-masked]>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

Hi Mikael,

Just my 2c.

-Jake

Bloat mailing list

--

---

From: Greg White <[email-masked]>
Date: Wed, 20 Mar 2019 22:56:58 +0000

That is an interesting point.  The goal of that MUST statement is to ensure per-flow-fairness.  Since fq provides per-flow-fairness as a guarantee via DRR, there is no need to actively apply that rule, it should just be an emergent property of the scheduling.

-Greg

From: "Holland, Jake" <[email-masked]>
Date: Wednesday, March 20, 2019 at 4:38 PM
To: Greg White <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>, bloat <[email-masked]>
Subject: Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

Thanks Greg,

Regards,
Jake

At dequeue:

Hi Bob & Greg,

Please note:

Regards,
Jake

PS:

PPS:

David,

Vint -

Some history:

Best Regards

Bob

-----Original Message-----
From: "Vint Cerf" <[email-masked]><mailto:[email-masked]>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <[email-masked]><mailto:[email-masked]>
Cc: "Mikael Abrahamsson" <[email-masked]><mailto:[email-masked]>, "David P. Reed" <[email-masked]><mailto:[email-masked]>, "[email-masked]"<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>, "bloat" <[email-masked]><mailto:[email-masked]>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

Hi Mikael,

Just my 2c.

-Jake

Bloat mailing list

--

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 21 Mar 2019 00:29:28 +0100

Jake,

> taken into account (and of course opinions may differ here).
Thanks for engaging in this.
I do also intend to address one or two other recurring questions on the 
bloat list, but I have a lot on at the mo.

> https://patents.google.com/patent/US20170019343A1/en
1/ In 2016, I arranged for the hire of a patent attorney to undertake 
the unusual step of filing a third party observation with the European 
Patent Office. This went through Al-Lu's patent application claim by 
claim pointing to prior art and giving the patent examiner all the 
arguments to reject each claim. However, the examiner chose to take very 
little note of it, which was disappointing and costly for us. The main 
prior art is:
     Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority 
Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
The guys named as inventors in AL-Lu's filing published a paper on PI2 
with me, in which we included a citation of this Gibbens paper as 
inspiration for the coupling. The Gibbens paper was already cited as 
background by other patents, so the EPO has it in their citation index.

The coupling was also based on my prior research with Mirja before I 
started working with the guys from Al-Lu in the RITE European 
Collaborative project. we had to go through a few rejections, but Mirja 
and I finally got this work published in 2014  - still before the 
priority date of the Al-Lu patent application:
     Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using 
Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom 
Workshop on Telecommunications Standards: From Research to Standards 
pp.583-588 (December 2014)

2/ The only claim that I could not find prior art for (in the original 
EU filing) was a very specific claim about using a square root for the 
coupling. The Linux implementation runs this the other way round so that 
it only has to do a squaring. So I figured we were safe from that.

However, until just now, I had not noticed that Al-Lu has 
retrospectively re-written the claims in the US patent and in the EU 
patent application to claim this the other way round - as a squaring. 
And to claim the two random number trick. Both restructuring to use a 
squaring and the two random number trick were definitely my ideas (while 
working for BT in a collaboration with Al-Lu). I have emails to prove 
this (from memory they were actually both in the same email). This is 
important, because a patent has to be about mechanism, not algorithm.

3/ This is a positive development. It means this patent is on very shaky 
legal ground. I have been trying to put pressure on Nokia to license 
this royalty free. But now I see what they have done, I am going to have 
to get a different type of legal advice.

> only happen for one at a time, if I'm reading it right.
Not at all. There is a reason why it's called Low Latency, Low Loss, 
Scalable throughput (L4S).

The L4S definition of ECN marking addresses all four of these at once. 
Strictly it directly addresses all except minimize cost, but minimize 
cost can be built around the system. This was the subject of my PhD. I 
haven't described L4S in these terms, because most people are only 
interested in the latency. But this is the underlying reason for my 
obsession with ECN.

Frank Kelly predicted that queuing delay would be removed from the 
optimization as it was minimized. With L4S we've got very close to that.

ECN removes all congestion loss.

And the use of a inverse linear congestion controller gives the scalable 
throughput. I shall be touching on this in my talk for netdev tomorrow, 
but it's not really a subject for an implementation conference.

Minimize cost is something you do by combining the congestion signals 
across a network. So any AQM is part of that. And congestion controllers 
are the other part - they implicitly optimize cost, using the congestion 
signals as shadow prices. The square root in classic TCP distorts this, 
but DCTCP's inverse linear controller gives proportional fairness 
directly. Without a weight term in the congestion controller, there is 
not really an economic optimization, but that can be built onto a 
proportionally fair system and competition will gradually cause that to 
happen (or regulation as a proxy for competition). These are very long 
term processes though.

> signal, so the "maximize throughput" use case can't ever get it.
Eh? There's definitely a misunderstanding or a difference in terminology 
between us here. The whole point of using a congestion controller like 
DCTCP is so that flow rate can scale indefinitely with capacity. Van 
Jacobson actually noted that the original TCP was unscalable in a 
footnote to the tech report version of the SIGCOMM paper.

The high fidelity congestion signal of what we call scalable congestion 
controllers (like DCTCP) is inversely proportional to the window. So as 
window scales up, the congestion signal scales down, so that their 
product remains constant. That means the number of ECN marks per RTT is 
scale-invariant. So the control signal remains just as tight at any scale.

Cheers

Bob

> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
>   o Operators:
>     Liberty Global
>     Charter
>     Rogers
>     Comcast
>     Shaw
>     Cox Communications
>    o Equipment vendors
>     ARRIS
>     Huawei
>     Broadcom
>     Intel
>     Casa
>     Nokia
>     Cisco
>     Videotron

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 21 Mar 2019 01:51:07 +0200

> On 21 Mar, 2019, at 1:29 am, Bob Briscoe <[email-masked]> wrote:
> 

> Eh? There's definitely a misunderstanding or a difference in terminology between us here. The whole point of using a congestion controller like DCTCP is so that flow rate can scale indefinitely with capacity. Van Jacobson actually noted that the original TCP was unscalable in a footnote to the tech report version of the SIGCOMM paper.
> 
> The high fidelity congestion signal of what we call scalable congestion controllers (like DCTCP) is inversely proportional to the window. So as window scales up, the congestion signal scales down, so that their product remains constant. That means the number of ECN marks per RTT is scale-invariant. So the control signal remains just as tight at any scale. 

If you'll indulge me for a moment, I'd like to lay out a compromise scenario where a lot of L4S' stated goals are still met.

There is no dualQ.  There is an AQM at the bottleneck link, of unspecified type, which implements SCE.  Assume that it produces CE marks like a conventional AQM, and also produces SCE marks like an L4S AQM produces CE.

A sender implements DCTCP-SCE, which is essentially Paced NewReno modified to subtract half of all acked data that was SCE-marked from its cwnd.  (This is equivalent to the DCTCP algorithm with g=1 and an arbitrarily small measurement window, but acting on SCE instead of CE.)  Any SCE mark also kicks it out of slow-start.

The means by which SCE information gets back to the sender is left vague for now; it's an orthogonal problem with several viable solutions.

What is missing from this scenario, from L4S' point of view?  And why have I been able to describe it so succinctly?

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 21 Mar 2019 00:41:06 +0000

Hi Greg,

On 2019-03-20, 13:55, "Greg White" <[email-masked]> wrote:
    In normal conditions, L4S offers "Maximize Throughput" +  "Minimize Loss" + "Minimize Latency" all at once.  It doesn't require an application to have to make that false choice (hence the name "Low Latency Low Loss Scalable throughput").  

[JH] This is an interesting claim, and I'm eager to see
how well it holds up under scrutiny and deployment.

I guess I'm not sure what exactly "normal" means, but I
would expect that there are a lot of cases that occur
frequently in practice where tradeoffs have to be made
between throughput, loss, and latency.

I'm finding I struggle to nail down exactly what I expect
from scenarios like a short-RTT L4S flow competing with a
long-RTT L4S flow (from transit delay) and with a BBR flow,
and likewise when a short and a long RTT L4S flow are
competing with a bunch of independent slow-start flows,
but if the L4S cases do indeed get a better throughput than
SCE-based approaches under the wide variety of situations
normal internet use can fall into, I think that would
convince me it's optimizing all of them at once, and it's
a mistake to call it focused on the latency use case.

But for now, I hope you'll forgive a little bit of
skepticism...  I find this stuff complicated, and it's hard
for me to put high confidence on some of the predictions.

Best regards,
Jake

    
    -Greg

    

        
        
         - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 21 Mar 2019 07:04:34 +0100

Jonathan,

> What is missing from this scenario, from L4S' point of view?  And why have I been able to describe it so succinctly?
My goal is also to tighten the EWMA parameter, g, in DCTCP to 1 (or 2). 
That is why we have recommended a queuing-time-based ramp AQM for the 
Low Latency queue, which so far works equivalently to the step with g 
set to its current default of 1/16. We have been doing experiments on 
this for some time. But it is important to assess each change one at a 
time.

Congestion controls are tricky to get stable in all situations. So it is 
important to separate ideas and research from engineering of more mature 
approaches that are ready for more widespread experimentation on the 
public Internet. Our goal with L4S was to use proven algorithms, and put 
in place mechanism to allow those algorithms to evolve.

As regards the desire to use SCE instead of the L4S approach of using a 
classifier, please answer all the reasons I gave for why that won't 
work, which I sent in response to your draft some days ago. The main one 
is incremental deployment: the source does not identify its packets as 
distinct from others, so the source needs the network to use some other 
identifier if it wants the network to put it in a queue with latency 
that is isolated from packets not using the scheme. The only way I can 
see to so this would be to use per-flow-queuing. I think that is an 
unstated assumption of SCE.

In contrast, L4S works with either per-flow queuing or dualQ, so it is 
more appropriate for a wider spread of scenarios. Again, in that same 
unanswered email, I described a way L4S can use per-flow queuing, and 
Greg has since given pseudocode. There are other problems with doing the 
codepoints the SCE way round - pls see that email.

There has been a general statement that the SCE way round is purer. 
However, that concept is in the eye of the beholder. The SCE way round 
does not allow the ECN field to be used as a classifier, so you don't 
get the benefit above about support for per-flow-queueing and dual 
queue. You also don't get the benefit of being able to relax 
resequencing in the network, because the network has no classifier to 
look at. For these, the SCE codepoint would need to be combined with a 
DSCP, and I assume you don't want to do that.

The L4S way round signifies an alternative meaning of the ECN field, 
which is exactly what it is. The problem of having to guess which type 
of packet a CE used to be has been roundly discussed at the IETF in TCPM 
and TSVWG WGs and it has been decided it is a non-problem if it is 
assumed to have been ECT(1) even if it was not - as written up in 
draft-ietf-ecn-l4s-id. And that discussion assumed TCP with 3DupACK 
reordering tolerance, not the more liberal use of RACK (or a RACK-like 
approach in other transports). With a RACK-like approach, it becomes 
even less of a problem.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 21 Mar 2019 09:46:39 +0200

> On 21 Mar, 2019, at 8:04 am, Bob Briscoe <[email-masked]> wrote:

> Congestion controls are tricky to get stable in all situations. So it is important to separate ideas and research from engineering of more mature approaches that are ready for more widespread experimentation on the public Internet. Our goal with L4S was to use proven algorithms, and put in place mechanism to allow those algorithms to evolve.

I hope that from my example, you can see how to adapt a more flexible and "mature" version of DCTCP to use SCE.  You should be able to use the same algorithms that you've worked so hard on; only the signalling method changes, and the trigger for falling back to Classic ECN behaviour is explicit (a plain old CE mark).

As for "proven algorithms", it was conclusively proven that DCTCP was *not* compatible with Classic ECN middleboxes, and had only been proven to work in tightly controlled environments.  I am told that TCP Prague has a failsafe, but I do not yet understand how that failsafe works, and what I have been told sounds fragile.  I am honestly perplexed that no explanation of this is forthcoming.

SCE works transparently with every deployed and proven congestion control algorithm out there, which simply ignores the information SCE provides.  Adaptations of some of those algorithms to incorporate SCE information seem to be straightforward to implement, especially since ns-3 now supports AccECN, so initial full-system experiments should be forthcoming quite soon.  We should even be able to rehabilitate DCTCP without resorting to failsafe workarounds - which *should* have you guys jumping for joy, in theory.

> As regards the desire to use SCE instead of the L4S approach of using a classifier, please answer all the reasons I gave for why that won't work, which I sent in response to your draft some days ago.

I'm afraid that must have got lost in the noise.  There *was* a lot of noise; it gave me a headache.

Regardless, I haven't seen any real claims that SCE won't work, except for some quibbles about RTT-fair convergence with single queues, which I subsequently found an elegant way to address.  We do have a bit of a publication bottleneck over here at the moment; limited manpower.

I have mainly seen claims that SCE isn't a one-for-one replacement for L4S using exactly the same mechanisms and infrastructure as L4S does.  Which is true, but unhelpful, because that would make SCE literally identical to L4S with no advantages of its own.  I'm willing to point out ways to implement L4S' goals using SCE; see below.

> The main one is incremental deployment: the source does not identify its packets as distinct from others, so the source needs the network to use some other identifier if it wants the network to put it in a queue with latency that is isolated from packets not using the scheme. The only way I can see to so this would be to use per-flow-queuing. I think that is an unstated assumption of SCE.

Strictly minimising latency for the individual flow, in the face of competing non-SCE traffic sharing a single queue, is not a goal of SCE per se; I consider it an orthogonal problem which is better addressed by existing solutions.  Coexisting with existing endpoints, existing traffic and existing middleboxes is paramount, and forms our main argument for incremental deployability.

Solutions already available include FQ and Diffserv.  I'll grant you that FQ is easier to implement at lowish speeds, where a cheap CPU can be loaded with flexible software to do the job.  You appear to be more focused on relatively high link capacities, as that is your main argument against FQ.  I'll just note in passing that good FQ can extract a lot of responsiveness from relatively low-capacity links.

Diffserv is widely deployed (in terms of hardware capabilities) and should be a natural fit for distinguishing classes of traffic from each other.  It is rarely used by applications because the networks tend to corrupt it in transit, and rarely make good use of the information into the bargain.  It strikes me that the cable industry may have more influence over that than I do.

> The SCE way round does not allow the ECN field to be used as a classifier…

The ECN field was never intended to be used as a classifier, except to distinguish Not-ECT flows from ECT flows (which a middlebox does need to know, to choose between mark and drop behaviours).  It was intended to be used to convey congestion information from the network to the receiver.  SCE adheres to that ideal.

There is a perfectly good and under-utilised 6-bit field for carrying classifier information, right there in the same byte as the ECN field.  You might want to ask the LE PHB guys for advice on making good use of it.

> You also don't get the benefit of being able to relax resequencing in the network, because the network has no classifier to look at.

My position is that the network is already free to relax resequencing semantics, regardless of the traffic carried.  IP does not guarantee anything about packet ordering, and protocols built on top of it have always had to cope with that, one way or another.

Wifi's head-of-line blocking while performing link-level retries can induce inter-flow coupled delays of many seconds in extreme cases, destroying reliability completely.  Recent work already reduces the effort the Linux wifi stack puts into link-level retries, given that most transports and protocols can survive some level of random loss.  This is done without relying on any classifier, because it benefits all traffic.

On high-capacity bonded links, the likelihood that two packets sent near-simultaneously on different component links, and consequently reordered, both belong to the same flow and will trigger a spurious retransmission, seems to be low enough to not care about, even with existing 3-dupack sensitive TCPs.  Therefore, relaxing resequencing requirements on these links should already be safe.  Perhaps you have hard data showing otherwise?

> …the SCE codepoint would need to be combined with a DSCP, and I assume you don't want to do that.

The SCE codepoint does not need to be combined with a DSCP.  Whether or not a DSCP assignment fits a given application is completely orthogonal to SCE.

You could quite reasonably implement something that looks very like DualQ using a trivial DSCP classifier instead of an ECN-based classifier, and that would be absolutely fine, and it would work with SCE.  It's just not necessary to make SCE work in the first place.

Meanwhile, I still have not seen a detailed answer as to how, precisely, TCP Prague reliably distinguishes a Classic ECN middlebox from an L4S one, in order to activate its failsafe mechanism.  Without that, I'm afraid I must assume that TCP Prague is not incrementally deployable.

Indeed, I was under the impression that DualQ and the use of ECT(1) as a classifier stemmed from this incompatibility, rather than being considered features in their own right.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 21 Mar 2019 09:02:00 +0100

Just to rapidly reply,

> The ECN field was never intended to be used as a classifier, except to distinguish Not-ECT flows from ECT flows (which a middlebox does need to know, to choose between mark and drop behaviours).  I…

Each PHB has a forwarding behaviour a DSCP re-marking behaviour and an 
ECN marking behaviour. The ECN field is the claissifer for the ECN 
marking behaviour.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 21 Mar 2019 09:45:18 +0100

> On Mar 21, 2019, at 07:04, Bob Briscoe <[email-masked]> wrote:
> 
> [...]

	Is there any work planned showing why the heuristic based on CE is not going to cause problems? Because as it stands ECT(1) might be a useful piece of information for a traffic classifier, but it certainly is not a reliable traffic category identifier (unless you are interested in the category: packets that promise to respond in the L4S-way if they should encounter congestion).

> The main one is incremental deployment: the source does not identify its packets as distinct from others, so the source needs the network to use some other identifier if it wants the network to put…

	You write a whole section on alternative labeling schemes in https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-B that you rule out, but that is based on hand-waving over the fact that CE-marking destroys the neat L4S labeling by ECT(1) scheme in two ways (mis-classifiaction of by AQM and mis-interpretation by end-point). 
You write: "Given shortage of codepoints, both L4S and classic ECN sides of an AQM would have to use the same CE codepoint to indicate that a packet had experienced congestion.  If a packet that had already been marked CE in an upstream buffer arrived at a subsequent AQM, this AQM would then have to guess whether to classify CE packets as L4S or classic ECN.  Choosing the L4S treatment would be a safer choice, because then a few classic packets might arrive early, rather than a few L4S packets  arriving late;" but in all honestly this simply assumes the rate of CE-marked packets of non-L4S flows will be low, otherwise your four Ls will suffer.

Regarding the second ambiguity you write:
"A.1.4.  Fall back to Reno-friendly congestion control on classic ECN bottlenecks [side note on https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#page-23 bottleneck appears in normal font instead of bold]
It would take time for endpoints to distinguish classic and L4S ECN marking.  An increase in queuing delay or in delay variation would be a tell-tale sign, but it is not yet clear where a line would be drawn between the two behaviours.  It might be possible to cache what was learned about the path to help subsequent attempts to detect the type of marking."
This has issues that IMHO need empirical data instead of hand-waving. Competent AQM solutions will control traffic rates without introducing massively increasing delays which will make it harder to do a heuristic based on RTT or RTT variation, this is the same mis-design LEDBAT suffers from, making inportant decisions based on un-reliable information... And trying to cache the L4S-ness of the active AQM assumes a steady-state behaviour of the network, which will not work for all cases.

> 
> In contrast, L4S works with either per-flow queuing or dualQ, so it is more appropriate for a wider spread of scenarios. Again, in that same unanswered email, I described a way L4S can use per-flow queuing, and Greg has since given pseudocode. There are other problems with doing the codepoints the SCE way round - pls see that email.
> 
> There has been a general statement that the SCE way round is purer. However, that concept is in the eye of the beholder. The SCE way round does not allow the ECN field to be used as a classifier,

	Nor does the proposed L4S interpretation, as elaborated above.

> so you don't get the benefit above about support for per-flow-queueing and dual queue. You also don't get the benefit of being able to relax resequencing in the network,

	I am still failing to grasp how this is supposed to work, and I had a look at the RACK RFC already. IMHO the link technology people should think about how much slack they require and ask the RACK developers to add that as a minimum to the specification, assuming it is reasonable. Say, lower level ARQ is expected to take 5ms worst-case, than ask RACK to specify a minimum reordering window of 10ms. The current RACK draft with re-ordering window bound to be <= RTT will not give reliable re-odering allowance to the lower levels unless they measure the RTT for each flow independently, I am sure that that is not going to fly...

> because the network has no classifier to look at.

	Same here CE-marked packets have no reliable label, and I assume that with L4S at steady-state and link capacity one can expect quite a number of CE-marked packets.
I begin to wonder whether there is a nomenclature mismatch here, and I have a definition of classification that is alien to the field here.

> For these, the SCE codepoint would need to be combined with a DSCP, and I assume you don't want to do that.
> 
> The L4S way round signifies an alternative meaning of the ECN field, which is exactly what it is. The problem of having to guess which type of packet a CE used to be has been roundly discussed at the IETF in TCPM and TSVWG WGs and it has been decided it is a non-problem

	This is not something to "decide" but something to experimentally test, but I guess I am just getting disillusioned how internet sausage is made...

Best Regards
	Sebastian

> if it is assumed to have been ECT(1) even if it was not - as written up in draft-ietf-ecn-l4s-id. And that discussion assumed TCP with 3DupACK reordering tolerance, not the more liberal use of RACK (or a RACK-like approach in other transports). With a RACK-like approach, it becomes even less of a problem.
> 
> 
> Bob
> 
>>  - Jonathan Morton
>> 
> 
> --

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 21 Mar 2019 09:49:02 +0100

Hi,

Am 21.03.19 um 09:02 schrieb Bob Briscoe:
>> The ECN field was never intended to be used as a classifier, except to
>> distinguish Not-ECT flows from ECT flows (which a middlebox does need
>> to know, to choose between mark and drop behaviours).  It was intended
>> to be used to convey congestion information from the network to the
>> receiver.  SCE adheres to that ideal.
> 

That's exactly the reason, why using ECT(1) as classifier for L4S
behavior is not the right choice. L4S should use a DSCP for
classification, because it is actually defining a PHB.

Regards
 Roland

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 21 Mar 2019 14:24:04 +0100

Roland,

> classification, because it is actually defining a PHB.

1/ First Terminology
The definition of 'PHB' includes the drop or ECN-marking behaviour. For 
instance, you see this in WRED or in PCN (Pre-Congestion Notification).  
If you want to solely talk about scheduling, pls say the scheduling PHB.

2/ The architectural intent of the ECN field

For many years (long before we thought of L4S) I have been making sure 
that ECN propagation through the layers supports the duality of ECN 
behaviours as both a classifier (on the way down from L7/L4 to L3/2) and 
as a return value (on the way back up).

The architecture of ECN is determined by the valid codepoint 
transitions. They are:
1. 00->11
2. 10->11
3. 01->11
4. 10->01

The first three were in RFC3168, but it did not preclude the fourth.
The fourth was first standardized in RFC6660 (which I co-authored). This 
had to be isolated from the e2e use of ECN by inclusion of a DSCP as well.

The relatively late addition of the fourth approach means that an 
attempt to mark using the SCE approach (10->01) is more likely to find 
that it gets reversed when the outer header is decapsulated, if the 
decapsulator hasn't been updated to the latest RFC that catered for this 
fourth transition (RFC6040, also co-authored by me).

L4S follows the original RFC3168 approach
SCE uses the fourth

So, SCE proposes to use /a/ correct approach, but it might not work.
Whereas L4S uses the original correct approach.

3a/ DualQ L4S AQMs
With the DualQ, the difference between the two queues is both in their 
ECN marking behaviour and in their forwarding/scheduling behaviour. 
However, whenever there's traffic in the classic queue the coupling 
between the AQMs overrides the network scheduler. The coupling is solely 
ECN behaviour not scheduling behaviour. So the primary difference 
between the queues is in their ECN-marking behaviour.

What do I mean by "the coupling overrides the network scheduler"? The 
network scheduler certainly does give priority to L4S packets whenever 
they arrive, but the coupling makes the L4S sources control how often 
packets arrive. It's tough to reason about, because we haven't had a 
mechanism like this before.

2b/ FQ L4S AQMs
If the AQM is implemented with per flow queues, the picture is clearer. 
The only difference between the queues is in the ECN marking behaviour 
of the different AQMs.

Bob

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Fri, 22 Mar 2019 13:53:35 +0100

Hi Bob,

see inline.

> If you want to solely talk about scheduling, pls say the scheduling PHB.

I thought that I'm well versed with Diffserv terminology, but I'm not
aware that a Diffserv PHB requires the definition of an ECN marking
behavior. In fact ECN is orthogonal to Diffserv as both RFCs 2474 and
2475 do not even mention ECN. RFC 2475:
"A per-hop behavior (PHB) is a description of the externally
observable forwarding behavior of a DS node applied to a particular
DS behavior aggregate." and "Useful behavioral distinctions
are mainly observed when multiple behavior aggregates compete for
buffer and bandwidth resources on a node."

Usually, there are different mechanisms how to implement a PHB,
e.g., for EF one could use a tail drop queue and Simple Priority
Queueing, Weighted Fair Queueing, or Deficit Round Robin and so
on. Consequently, queueing and scheduling behavior are used to
_implement_ a PHB, i.e., IMHO it makes sense to distinguish between
the PHB as externally observable behavior and a specific _PHB
implementation_ as also pointed out in RFC2475:
   PHBs are implemented in nodes by means of some buffer management and
   packet scheduling mechanisms.  PHBs are defined in terms of behavior
   characteristics relevant to service provisioning policies, and not in
   terms of particular implementation mechanisms.

So some of the Diffserv PHBs do _not_ require using an AQM,
which is often the basis for ECN marking, e.g., for EF
tail drop should be sufficient. For other PHBs it may be
useful to say something about ECN usage (as I did for LE).

RFC 2475:

   PHBs may be specified in terms of their resource (e.g., buffer,
   bandwidth) priority relative to other PHBs, or in terms of their
   relative observable traffic characteristics (e.g., delay, loss).

I think that L4S therefore specifies such a PHB as it is defined
in relation to the default PHB (as in the L4S arch draft
"Classic service").

> The architecture of ECN is determined by the valid codepoint

I wouldn't say that it's determined solely by the transitions.

> So, SCE proposes to use /a/ correct approach, but it might not work.

In case of nodes that implement RFC6040? I think that it would
be useful to measure how many boxes out there actually do this
(or how widespread is ECN usage actually, e.g., how many boxes
actually set CE on congestion? MAMI results anyone?).

> Whereas L4S uses the original correct approach.

Which might also not work...in case RFC3168 boxes set CE, so
the L4S receivers/senders cannot know that the CE wasn't set
by an L4S node.

> mechanism like this before.

Yes, the DualQ mechanism is actually nice, but what I particularly don't
like is to fix the coupling into nodes in this way. If the congestion
control behavior is different from your expectation it will not work
(as already experienced with BBR) properly. This would ossify congestion
control evolution and I see this a very big disadvantage of this
approach.

> The only difference between the queues is in the ECN marking behaviour

This would at least avoid the baked-in coupling law problem...

Regards
 Roland

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Mar 2019 02:34:59 +0100

Alex, inline...

On 24/03/2019 21:15, [email-masked] wrote:
>
> Hi Bob,
>
>
> I note that all the non-dependent claims of US20170019343A1 (claims 1,14,22) seem to assume use of the proportional-integral controller (Note, I am not a lawyer, and especially not a patent lawyer).
Yes, as I understand it, Nokia's intention with this filing was to cover 
use of the PI controller in particular, in combination with various 
other ideas.

> In Appendix B of draft-briscoe-tsvwg-aqm-dualq-coupled, an alternate algorithm 'Curvy RED' seems to replace PI, but it is noted that 'the Curvy RED algorithm has not been maintained to the same degree as the DualPI2 algorithm '.
>
> Can you comment on whether the Curvy RED algorithm could form a non-patent-encumbered dualq? In particular:
>   - Why wasn't curvy red further developed? Was it found to contain some deficiency? Are you intending to present it as an alternative?
We just didn't develop it further, cos we were getting better results 
with PI2. However, I am aware of a hardware implementation based on 
Curvy RED going on at the moment, and you will see there have recently 
been review comments on that Curvy RED appendix on the list.

So, even tho PI might be better, Curvy RED (or another AQM) might be 
preferable for other reasons that performance (e.g. ease of 
implementation, or similarity to an existing hardware implementation).

And indeed, there's nothing to stop anyone using other AQMs, either to 
work round the IPR, or because they're preferable in their own right - 
the DualQ Coupled AQM is intentionally a framework into which you drop 2 
AQMs.

>   - Does Curvy RED actually completely replace PI?
Yes.
>   - Can we have reasonable assurance that no patents will surface covering Curvy RED?
Well, I developed the idea of Curvy RED and I / my employer (BT) did not 
file any IPR at the time. I got approval to publish a tech report 
jointly with Al-Lu. http://bobbriscoe.net/pubs.html#CRED-insights

That was May 2015, so given nothing has surfaced by now, there can't be 
anything from that time from us (where us = Al-Lu & BT).

Of course, I cannot guarantee that there is not another patent in the 
system from some other random company that my searches haven't found. 
There are large numbers of AQM patents. Also, I cannot guarantee that an 
implementer working now isn't filing patents around their 
implementation. All we can do is publish as much as possible as early as 
possible to try to keep some areas of the field open.

Bob
>
> Thanks,
> Alex
>
>
> On Wednesday, March 20, 2019, 11:29:38 PM GMT, Bob Briscoe <[email-masked]> wrote:
>
>
>
>
>
>
> 1/ In 2016, I arranged for the hire of a patent attorney to undertake the unusual step of filing a third party observation with the European Patent Office. This went through Al-Lu's patent application claim by claim pointing to prior art and giving the patent examiner all the arguments to reject each claim. However, the examiner chose to take very little note of it, which was disappointing and costly for us. The main prior art is:
>      Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
> The guys named as inventors in AL-Lu's filing published a paper on PI2 with me, in which we included a citation of this Gibbens paper as inspiration for the coupling. The Gibbens paper was already cited as background by other patents, so the EPO has it in their citation index.
>
> The coupling was also based on my prior research with Mirja before I started working with the guys from Al-Lu in the RITE European Collaborative project. we had to go through a few rejections, but Mirja and I finally got this work published in 2014  - still before the priority date of the Al-Lu patent application:
>      Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom Workshop on Telecommunications Standards: From Research to Standards pp.583-588 (December 2014)
>
> 2/ The only claim that I could not find prior art for (in the original EU filing) was a very specific claim about using a square root for the coupling. The Linux implementation runs this the other way round so that it only has to do a squaring. So I figured we were safe from that.
>
> However, until just now, I had not noticed that Al-Lu has retrospectively re-written the claims in the US patent and in the EU patent application to claim this the other way round - as a squaring. And to claim the two random number trick. Both restructuring to use a squaring and the two random number trick were definitely my ideas (while working for BT in a collaboration with Al-Lu). I have emails to prove this (from memory they were actually both in the same email). This is important, because a patent has to be about mechanism, not algorithm.
>
> 3/ This is a positive development. It means this patent is on very shaky legal ground. I have been trying to put pressure on Nokia to license this royalty free. But now I see what they have done, I am going to have to get a different type of legal advice.
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Mar 2019 03:47:41 +0100

Roland,

On 22/03/2019 13:53, Bless, Roland (TM) wrote:
> behavior.
Ah well, I do not think that any living human has visited all the dark 
corners of the Diffserv world.

I didn't mean (or say) that a Diffserv PHB /requires/ an ECN marking 
behaviour, just that if you are talking about a Diffserv PHB that 
includes an ECN marking behaviour, it helps to say when you are solely 
talking about the scheduling part of the PHB.

In many cases when there is no ECN marking behaviour, it makes no 
difference if you omit the word scheduling, cos that is all there is to 
the behaviour.

> In fact ECN is orthogonal to Diffserv as both RFCs 2474 and
Even the original experimental ECN spec RFC2481 was published just after 
2474 & 2475. So you wouldn't expect the original Diffserv specs to 
mention something that didn't exist then.

>     relative observable traffic characteristics (e.g., delay, loss).
Since RFC2474 & 2475, AQM behaviour and/or ECN marking behaviour has 
become part of some Diffserv PHBs. E.g. WRED in AF. See any of the 
tables in RFC4594 that have an AQM column.

The need for ECN marking behaviour (rather than just AQM behaviour in 
general) as part of a PHB became necessary during the definition of PCN. 
Jo Babiarz and Kwok Ho Chan were both authors of 4594 and of many of the 
PCN specs, and proposed the term 'marking behaviour' as part of the PHB. 
You will find ECN marking behaviours are central to, for instance, RFC5670.

As I've pointed out already, the transitions used by SCE were already in 
the PCN baseline encoding spec [RFC6660], except only defined if 
accompanied by a specific DSCP (which was subsequently standardized as 
EF-ADMIT).

> in relation to the default PHB (as in the L4S arch draft
No. The L4S use of ECN is orthogonal to Diffserv, and can be associated 
with more than one scheduling PHB in a queuing hierarchy. See 
draft-briscoe-l4s-diffserv (which I would welcome you to review - Brian 
Carpenter is also currently reviewing it).

However, you are certainly right in thinking that L4S associated with 
the default PHB is by far and away the most important use-case for L4S. 
All the other possible schemes in l4s-diffserv are only possibilities - 
probably for corporate networks where proliferation of Diffserv models 
is currently most prevalent.

> I wouldn't say that it's determined solely by the transitions.
Correct. I didn't say that either.
> be useful to measure how many boxes out there actually do this
[BB] To measure this, you would need to have a box between the tunnel 
endpoints to mark the outer, before you could check what the behaviour 
of the decap was.

> actually set CE on congestion? MAMI results anyone?).
[BB] Brian Trammel re-ran ETHZ's ECN tests in Jan'19. Informally he told 
me yesterday that he found about 13 CE marks (i.e. still hardly any). 
But this might mean the links aren't loaded when he's looking. It's hard 
to apply enough load while doing a large-scale measurement - takes far 
too long per path.

Stuart Cheshire is helping me to try to get something meaningful out of 
the data Apple is continually gathering. The data Padma presented at 
MAPRG at IETF-98 was % of Apple devices tested that saw at least one CE 
mark in 12 hours.

The hard problem is, once we find something CE-marking, we're interested 
in knowing whether it's FQ (which protects flows from each other) or 
single queue (which doesn't). Greg White, Jake Holland & I devised a 
test for this between us. Tweak a congestion control so you have an 
aggressive one. Run it in parallel with another regular CC between the 
same two hosts. Then look for correlation of RTT movements. Like common 
bottleneck detection in RMCAT.

> the L4S receivers/senders cannot know that the CE wasn't set
Yes. That's a different point, but it's true.

> control evolution and I see this a very big disadvantage of this
The argument for using the square is to be compatible with the 
worst-case classic traffic, which is Reno. The worst-case will remain 
ossified for some many years yet. It doesn't mean that better CCs cannot 
develop within Classic. And to a certain extent, they still have to 
coexist with the worst case Classic... we'll see what the latest update 
on BBRv2 is in a few day's time.

But importantly, a square relationship between flow rates is not 
enforced by the network, it's encouraged for end-systems that have a 
default behaviour. But, if the network policy becomes wrong in future, 
end-systems can correct it.

> This would at least avoid the baked-in coupling law problem...
Er, no. 1:1 is just as much a baked-in policy, and it really is baked-in 
when the network is enforcing it.

That's why we developed the DualQ - we wanted to avoid the network 
enforcing rigid fairness at every instant. It's much less ossified when 
end-systems keep to it voluntarily.

Bob

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 24 Mar 2019 20:15:15 +0000 (UTC)

Hi Bob,

I note that all the non-dependent claims of US20170019343A1 (claims 1,14,22) seem to assume use of the proportional-integral controller (Note, I am not a lawyer, and especially not a patent lawyer). In Appendix B of draft-briscoe-tsvwg-aqm-dualq-coupled, an alternate algorithm 'Curvy RED' seems to replace PI, but it is noted that 'the Curvy RED algorithm has not been maintained to the same degree as the DualPI2 algorithm '.

Thanks,
Alex