Subject: [tsvwg] feedback and thoughts L4S / SCE

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 20 Nov 2020 14:33:42 +0100 (CET)

Hi,

I'm trying to catch up on the work has been done, but I thought I'd give 
some thoughts / feedback on what I see:

https://github.com/heistp/l4s-tests

"L4S transports experience intra-flow latency spikes at RFC3168 
bottlenecks, particularly with the widely deployed fq_codel"

FQ_CODEL isn't widely deployed. Yes, it's enabled in the linux kernel but 
very seldom at the bw chokepoint of the connection. I do not agree with 
the above statement at all. Thus any benchmarking vs FQ_ anything is 
invalid. What's mostly used today are typically simple FIFOs, sometimes 
diffserv enabled with multiple FIFOs.

This means the tunnel argument against any FQ or non-FQ AQM is invalid. I 
also doubt we'll see widely deployed FQ going forward because of the cost 
of doing this in hw (and most devices are hw accelerated).

"Deployments of fq_codel
The fq_codel qdisc has been in the Linux kernel since version 3.6 (late 
2012) and is now in widespread use in commercial routers (e.g. Ubiquiti 
EdgeMAX and UniFi products)"

This is misleading. Yes, EdgeMAX supports FQ_CODEL if you turn off 
hw_acceleration, bringing down the performance to 10% or less of what the 
hw acceleration yields. I do not agree that it's in "widespread" use.

I'd gladly take data to prove me wrong, that FQ_CODEL is in widespread use 
at Internet bw chokepoints. I'd be surprised if it was enabled on more 
than 1% of residential connections.

On to L4S.

Now, taking above statement and looking at what L4S is doing, for 1-2 
decades going forward L4S enabled transport will often encounter FIFOs, 
mostly without ECN capability at all. They need to be proven safe in this 
operational reality. From what I can see, this is still not the case?

Going forward:

In order to progress to a solution, we need to have participants agree on 
what the world looks like right now, what's doable in the next few years, 
and what's required to handle existing/near term reality. SCE proponents 
have to stop arguing that we'll have FQ_ anything near term, or that this 
is in wide use. It's not. L4S proponents need to prove that their 
proposals are safe in a world of FIFOs, and accept that this is going to 
be a reality for years to come. AQMs aren't deployed in a hurry, we're 
talking many years to deploy on the wider Internet.

We're spending the last bit in the IP header and we'd better get it right. 
In order to do that, I'd like to see realism and pragmatism in the 
reasoning.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Greg White <[email-masked]>
Date: Fri, 20 Nov 2020 19:38:43 +0000

Hi Mikael,

On your point about L4S in non-ECN FIFOs, I agree that these will dominate in the Internet for quite some time. But, I don’t believe there has been any evidence of lack of safety for L4S-compatible CCs in those bottlenecks.  It is, in fact, a requirement in the ECN-L4S-ID draft that CCs respond to loss in a Reno-safe way.
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-12#section-4.3

What has been argued is that the current TCP Prague prototype CC responds to loss *exactly* like Reno, and so is outcompeted by Cubic in these bottlenecks (see the pfifo results in https://github.com/heistp/l4s-tests#network-bias ).

If you attended Bob & Koen’s talk in ICCRG (https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-prague-congestion-control-00), you saw that they mention (slide 9) “faster than additive increase” as an aspect that is not currently included in the Prague CC.  They also mention (slide 15) “Integration with BBR/Cubic instead of Reno” as a possible area for others to contribute.

-Greg

From: tsvwg <[email-masked]> on behalf of Mikael Abrahamsson <swmike=[email-masked]>
Organization: People's Front Against WWW
Date: Friday, November 20, 2020 at 6:34 AM
To: "[email-masked]" <[email-masked]>
Subject: [tsvwg] feedback and thoughts L4S / SCE

Hi,

"Deployments of fq_codel

On to L4S.

Going forward:

reasoning.

--

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 20 Nov 2020 21:01:03 +0100

Hi Greg,

> On Nov 20, 2020, at 20:38, Greg White <[email-masked]> wrote:
> 
> Hi Mikael,
>  
> On your point about L4S in non-ECN FIFOs, I agree that these will dominate in the Internet for quite some time. But, I don’t believe there has been any evidence of lack of safety for L4S-compatible CCs in those bottlenecks.

	[SM] Correct, that is not s much a case for not being safe but rather for TCP Prague not delivering on its promises:

http://sce.dnsmgr.net/_archive/l4s-2020-11-17T164559-s1-rtt-update/l4s-s1-rttfair/l4s-s1-rttfair-ns-cubic-vs-prague-pfifo_1000_-50Mbit-160ms-10ms_tcp_delivery_with_rtt.svg

http://sce.dnsmgr.net/_archive/l4s-2020-11-17T164559-s1-rtt-update/l4s-s1-rttfair/l4s-s1-rttfair-ns-prague-vs-cubic-pfifo_1000_-50Mbit-160ms-10ms_tcp_delivery_with_rtt.svg

Cubic wiping out (but not "completely starving") TCP Prague in a FIFO with one flw @160ms RTT the other at 10ms. So TCP Prague in a non-ECN fifo gets ~1:4 of the available bandwidth. Let's not talk about latency, since without an L4S AQM TCP Prague is not a low latency transport protocol, to be generous.

That is less of a safety concern and more of a can L4S actually deiver on its promises issue.

Mind you Koen's proposed solution is to make TCP Prague automatically switch to a CUBIIC like CC response once RTTs are >= 80 ms, a clear sign of defeat if any. L4S is a short fast-lane, not more (and not less) in spite of being marketed as the "future" of the internet...

>  It is, in fact, a requirement in the ECN-L4S-ID draft that CCs respond to loss in a Reno-safe way.

	[SM] Sure and in the FIFO TCP Prague at least does no harm to CUBIC, but the ask by Lars was show do no harm AND deliver on its promises. For a change, with FiFO is not L4S being to aggressive.

> What has been argued is that the current TCP Prague prototype CC responds to loss *exactly* like Reno, and so is outcompeted by Cubic in these bottlenecks (see the pfifo results in https://github.c…

	[SM] Basing TCP Prague on DCTCP was a conscious choice of team L4S, with data indicating than basing on Linux' defsau;t TCP might have been a better approach, but it is what it is.

> If you attended Bob & Koen’s talk in ICCRG (https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-prague-congestion-control-00), you saw that they mention (slide 9) “faster than addit…

	[SM] So also "not tested" at all. Or put differently, not clear if actually achievable,  like RTT-independence.... (which generically is no achievable from a connections end-points)

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 22 Nov 2020 15:46:02 +0100

Hi Mikael,

just as a data point, OpenWrt switched to fq_codel as default qdisc on all interfaces some years ago. I have no numbers how many OpenWrt devices are in the wild, and how many commercial home router OSes are based on sufficiently modern OpenWrt, but that is some use. The OpenWrt project also has no official counter of running instances but for example Qualcomm's QCA Software Development Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one would naively assume. That said, the QDSK kernel seems to be quite old (at least Kernel major version 2 seems dead now), and I have no insight which features are back-ported to it.

> On Nov 20, 2020, at 14:33, Mikael Abrahamsson <swmike=[email-masked]> wrote:
> 
> 
> Hi,
> 
> I'm trying to catch up on the work has been done, but I thought I'd give some thoughts / feedback on what I see:
> 
> "L4S transports experience intra-flow latency spikes at RFC3168 bottlenecks, particularly with the widely deployed fq_codel"
> 
> FQ_CODEL isn't widely deployed.

	[SM] Given your lack of numbers about OpenWrt that is a pretty precise statement that is hard to prove or disprove. But for a change I actually agree that your intuition here, but I want to point out that it is a (likely) prediction and not a fact. It is important in this discussion to separate predictions from fact though, so let's be open about what we know and what we do not know.

> Yes, it's enabled in the linux kernel but very seldom at the bw chokepoint of the connection.

	[SM] Again a prediction, not a fact, as I said reasonably recent OpenWrt defaults to fq_codel on all interfaces (that support it) as well as on WiFi which with inceasing access rates more and more often becomes the true bottleneck (no numbers, as that is my prediction, based on my own experience).

> I do not agree with the above statement at all.

	[SM] Given our lack of hard numbers that is telling more about your  judgement than about reality.

> Thus any benchmarking vs FQ_ anything is invalid.

	[SM] And that is pretty harsh. Also unhelpful teminology, "irrelevant" I would accept, but invalid is simply the wrong quality to argue about. 

> What's mostly used today are typically simple FIFOs, sometimes diffserv enabled with multiple FIFOs.

	[SM] And here we are again at our lack of numbers... As before I share your intuition, but it is not a fact. 
"Fun" fact TCP Prague really is not at all suited to a FIFO world at all, as Pete demonstrated.

> 
> This means the tunnel argument against any FQ or non-FQ AQM is invalid.

	[SM] Again validity seems the wrong yardstick, but I take your point.

> I also doubt we'll see widely deployed FQ going forward because of the cost of doing this in hw (and most devices are hw accelerated).

	[SM] An often heard claim that suffers from the fact that that has simply never been tried at scale, so the cost argument reflects the status quo not necessarily actual economic "laws", no?

> 
> "Deployments of fq_codel
> The fq_codel qdisc has been in the Linux kernel since version 3.6 (late 2012) and is now in widespread use in commercial routers (e.g. Ubiquiti EdgeMAX and UniFi products)"
> 
> This is misleading. Yes, EdgeMAX supports FQ_CODEL if you turn off hw_acceleration, bringing down the performance to 10% or less of what the hw acceleration yields. I do not agree that it's in "widespread" use.

	[SM] Side-note, the same hardware acceleration also disallows quite a large number of features of the Linux kernel's networking features, and IMHO is a step in the wrong direction, allowing chep-ish devices to shine in stereotyped tests, but fail in real world use once the user/owner wants to step beyond the middle-of-the-road feature set supported by the accelerator. THere is a reason why hardware acceleration features have a hard time being accepted in the Linux kernel...
	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).

> 
> I'd gladly take data to prove me wrong, that FQ_CODEL is in widespread use at Internet bw chokepoints.

	[SM] As I am sure that Pete will take data disproving the opposite claim you make, but that leaves us none the wiser.
QUESTION: Where do you think most choke points live, at the ISP-end-user boundary, deeper in the ISP's network, or even at exchange points? My own limited experience points to the ISP-end-user edge, but that is purely anecdotal.

> I'd be surprised if it was enabled on more than 1% of residential connections.

	[SM] This is a good question. Anybody here can offer an sufficiently robust estimate of the total number of residential connections? 

> 
> On to L4S.
> 
> Now, taking above statement and looking at what L4S is doing, for 1-2 decades going forward L4S enabled transport will often encounter FIFOs, mostly without ECN capability at all. They need to be proven safe in this operational reality. From what I can see, this is still not the case?

	[SM] Why only bother about safety here, why not also expect actual delivering on its promises? ATM the data as presented by Pete indcates that L4S as currently designed and implemented will give robustly and reliably rate and latency advantages of L4S versus non-L4S traffic (while claiming only to give a latency edge), and will robustly and reliably give an advantage of short RTT L4S flows over longer RTT L4S (in significant excess over the known and expected RTT bias as seen in FIFOs and still significanly stronger than in other single queue AQMs like the non-L4S PI instance). in short all L4S currently does is establish a fast lane for short RTT flows (which accidentally is al that team L4S ever tested, so I assume that this is not accidental but a non verbalized design goal).

> 
> Going forward:
> 
> In order to progress to a solution, we need to have participants agree on what the world looks like right now, what's doable in the next few years, and what's required to handle existing/near term reality. SCE proponents have to stop arguing that we'll have FQ_ anything near term, or that this is in wide use. It's not. L4S proponents need to prove that their proposals are safe in a world of FIFOs,

	[SM] Addendum, we should also ecpect that L4S transports actually work reasonably well for actually transporting data in the existing mostly FIFO internet. L4S comes with known side-effects and wants a special treat (ECT(1)) so should actually be expected to deliver on it promises.

> and accept that this is going to be a reality for years to come. AQMs aren't deployed in a hurry, we're talking many years to deploy on the wider Internet.
> 
> We're spending the last bit in the IP header and we'd better get it right. In order to do that, I'd like to see realism and pragmatism in the reasoning.

	[SM] +1. A discussion that is closer tied to the existing data and that does not take "claims" in the drafts as facts would also help to make the discussion more relevant.

Regards
	Sebastian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 22 Nov 2020 15:56:46 +0100 (CET)

On Sun, 22 Nov 2020, Sebastian Moeller wrote:

> just as a data point, OpenWrt switched to fq_codel as default qdisc on all interfaces some years ago.

And how often is this fq_codel the slowest link in the chain?

> I have no numbers how many OpenWrt devices are in the wild, and how many 
> commercial home router OSes are based on sufficiently modern OpenWrt, 
> but that is some use. The OpenWrt project also has no official counter

Most of the commercial openwrt based devices will not use CPU based 
forwarding (required to make fq_codel work) but instead has hardware 
accelerator (which does not do fq_codel).

> of running instances but for example Qualcomm's QCA Software Development 
> Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one 
> would naively assume. That said, the QDSK kernel seems to be quite old 
> (at least Kernel major version 2 seems dead now), and I have no insight 
> which features are back-ported to it.

If you're using QCA hw-accelerated forwarding then you're not doing 
fq_codel.

> 	[SM] Again a prediction, not a fact, as I said reasonably recent 
> OpenWrt defaults to fq_codel on all interfaces (that support it) as well 
> as on WiFi which with inceasing access rates more and more often becomes 
> the true bottleneck (no numbers, as that is my prediction, based on my 
> own experience).

If you're using HW acceleration, then fq_codel isn't used also on wifi.

> 	[SM] Given our lack of hard numbers that is telling more about your  judgement than about reality.

Do you have numbers? How many ISP provided routers out there actually use 
fq_* ? ISP provided router is the most common in use.

> 	[SM] And that is pretty harsh. Also unhelpful teminology, "irrelevant" I would accept, but invalid is simply the wrong quality to argue about.

Fine.

> "Fun" fact TCP Prague really is not at all suited to a FIFO world at all, as Pete demonstrated.

Do you have better numbers?

> 	[SM] An often heard claim that suffers from the fact that that has 
> simply never been tried at scale, so the cost argument reflects the 
> status quo not necessarily actual economic "laws", no?

Do you know of a FQ_* implementation in hardware?

> 	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).

Thanks for ack:ing that.

> QUESTION: Where do you think most choke points live, at the ISP-end-user 
> boundary, deeper in the ISP's network, or even at exchange points? My 
> own limited experience points to the ISP-end-user edge, but that is 
> purely anecdotal.

The choke point is most often the customer last-mile access line or the 
wifi hop.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 22 Nov 2020 17:24:43 +0100

Hi Mikhael,

> On Nov 22, 2020, at 15:56, Mikael Abrahamsson <[email-masked]> wrote:
> 

	[SM] Same problem as before, we lack reliable numbers.

> 
>> I have no numbers how many OpenWrt devices are in the wild, and how many commercial home router OSes are based on sufficiently modern OpenWrt, but that is some use. The OpenWrt project also has no official counter
> 
> Most of the commercial openwrt based devices will not use CPU based forwarding (required to make fq_codel work) but instead has hardware accelerator (which does not do fq_codel).

	[SM] Well, there is a nss_wfq, and nss_codel, sure that is not the same as nss_fq_codel (see https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/drivers/net/ethernet/atheros/nss-qdisc). Let's call this a tie, sure, no fq_codel, but wfq (which to my knowledge is wegted fair queueing).

> 
>> of running instances but for example Qualcomm's QCA Software Development Kit (QSDK) is based on OpenWrt, so OpenWrt is more pervasive than one would naively assume. That said, the QDSK kernel seems to be quite old (at least Kernel major version 2 seems dead now), and I have no insight which features are back-ported to it.
> 
> If you're using QCA hw-accelerated forwarding then you're not doing fq_codel.

	[SM] Well, I would not and do not not use the accelerated path exactly do to its lack lack of the qdiscs that I would like to use, but I am not representative and will not claim to be so.

> If you're using HW acceleration, then fq_codel isn't used also on wifi.

	[SM] You are sure about that? As far as I know the wifi fq_codel instance lives in the shred 802.11linux infrastructure, is that not used by the QSDK?

> Do you have numbers?

	[SM] Nope, I do not and I admitted to that up front. Doesn't change the fact that you were extrapolating from non-existing numbers here, no?

> How many ISP provided routers out there actually use fq_* ? ISP provided router is the most common in use.

	[SM] Again, no numbers I can offer.

> Do you have better numbers?

	[SM] Sorry no, and as I said I share your intuition, but that tells more about my intuition than about reality ;)

> Do you know of a FQ_* implementation in hardware?

	[SM] No, looking at the literature, there are implementations for FPGAs though, but I am not sure how close to reality these are. But as I said, due to lack of demand (and lack of paying extra for a new feature) it is not clear to me whether the lack of FQ in commercial routers is driven by the task being technically infeasible or by it being economically unattractive.

>> 	Also "wide-spread" use is again a claim that should be based on numbers (which cuts both ways, so applies to Pete's claim as well).

	[SM] In case this is in doubt, I am trying to assess all solutions to low latency networking by the same yards stick and try to be consistent with my requirements (if not I try to explain a position that looks inconsistent at first).
	And, to repeat myself, and without hard numbers to back this up, I think your offered frequency estimates are quite reasonable.

> The choke point is most often the customer last-mile access line or the wifi hop.

	[SM] Good. That clearly indicates, that whether or not big-iron back-bone routers ever develop AQMs or not is not going to be the critical point for HFCC's feasibility! And then I would say SQM in OpenWrt demonstrates that FQ and compeent AQM at the ISP to end-user transition is feasible up to Gbps rates with gear below the $100 mark (a raspberry pi4B has enough CPU cycles for traffic shaping just below line rate, and it is just an USB3 ethernet dongle short of being a wired router)*. 
	Sure that currently is a) not a solution for most "normal" end-users, and b) is probably too expensive compared to what ISPs pay for the CPE they distribute (especially since it lacks adequate WiFi). But it does show that at that location FQ is not a pie-in-the-sky impossibility, nor prohibitively expensive.
	That it is IMHO possible and viable, does not change that I agree with you, that currently is is a small minority of links that are operated in that way. Introduction of HFCC would be a great possibility to change that ;)

Best Regards
	Sebastian

*) I do not claim to be fully objective here, I am involved in the SQM effort.

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 22 Nov 2020 18:41:14 +0200

> On 22 Nov, 2020, at 4:46 pm, Sebastian Moeller <[email-masked]> wrote:
> 

Well, for a start, take basically 100% of wireline free.fr subscribers, the largest wireline ISP in France, who for a number of years now has deployed fq_codel at its bottlenecks (including in CPE) as a matter of policy.  Is France 1% of the world?  No, but it's a darn good start, and free.fr is clearly not alone according to Jake Holland's data, which shows significant ECN activity on about half a dozen ASNs and an increasing trend overall.  I've also heard from reliable sources that Telenor is rolling out a similar deployment in Norway, and Preseem has helped to deploy many WISP networks that use fq_codel actively.

Let's be precise about this.  This does not imply that a higher than average proportion of free.fr, Telenor or WISP subscribers have ECN fully enabled in their endpoint hosts (as opposed to passively enabled as most servers do).  That means we can only actually see ECN activity on some small fraction of the deployment.  The corollary is that the deployment of AQM at bottlenecks is probably much wider than ECN data alone can show.  Take the latter as a *very* conservative lower bound.

Importantly, RFC-3168 AQMs provide the same rate of congestion signalling to Not-ECT traffic, via packet drops, as they do to ECT traffic using CE marks.  So all the interactions between TCP Prague and conventional CC algorithms apply just the same, and we have fresh data proving that.

Pete and I are preparing a presentation of concrete data to explain this situation more clearly.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Sun, 22 Nov 2020 20:26:39 +0100

Hi Michael, a few replies below...

> diffserv enabled with multiple FIFOs.

I do accept the criticism that the phrase "widely deployed" was only
qualified with a vendor list. So in response, I've contributed some ECN
data under a new thread, and I'll review my part of the writeup later
for any language that I think isn't supported well enough. Just asking
for some patience with that. :)

> of doing this in hw (and most devices are hw accelerated).

However, it's still used, at the edge, in backhauls, and built into
Linux WiFi drivers. The Preseem QoE platform is indirect evidence of
fq_codel use in backhauls, and we know there are ISPs that manage
queues in their backhauls with custom scripts using fq_codel. Instead
of delving into various products, I suspect that trying to gather ECN
stats at relevant places on the Internet might be more fruitful(?)

It's a known issue that any non-FQ RFC3168 bottleneck leads to
domination of non-L4S flows by competing L4S flows, tunnel or not. I'll
post another test scenario with several AQMs showing that once I write
it up.

> than 1% of residential connections.

Let's see if we can come up with more data.

>  SCE proponents 
> is in wide use. It's not.

We're working on SCE and know where we're headed at least, but it isn't
on the WG's table now.

> In order to do that, I'd like to see realism and pragmatism in the 

:)

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 28 Nov 2020 14:00:59 +0100

Hi Mikael,

As it turns out there is a nssfq_codel modul for qualcomm`s nss network acellerator cores:
see https://forum.openwrt.org/t/ipq806x-nss-drivers/12613/2160

Is that a sufficient response to put tge hypothesis to rest that acelleration techniques do not offer fq_codel?

Best Regards
        Sebastian

On 22 November 2020 15:56:46 CET, Mikael Abrahamsson <[email-masked]> wrote:
>> 	[SM] And that is pretty harsh. Also unhelpful teminology,
>"irrelevant" I would accept, but invalid is simply the wrong quality to
>argue about.
>
>Fine.
>
>> 	[SM] And here we are again at our lack of numbers... As before I
>share your intuition, but it is not a fact.
>>> I also doubt we'll see widely deployed FQ going forward because of
>the cost of doing this in hw (and most devices are hw accelerated).
>>
>> 	Also "wide-spread" use is again a claim that should be based on
>numbers (which cuts both ways, so applies to Pete's claim as well).
>
>Thanks for ack:ing that.
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 28 Nov 2020 15:20:07 +0100 (CET)

> Is that a sufficient response to put tge hypothesis to rest that acelleration techniques do not offer fq_codel?

This is great news. Do you have more examples?

I looked over the devices list with that SoC and most of them were 130EUR 
here (or more). Pretty high-end devices.

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 28 Nov 2020 17:50:34 +0200

> On 28 Nov, 2020, at 4:20 pm, Mikael Abrahamsson <swmike=[email-masked]> wrote:
> 
> I looked over the devices list with that SoC and most of them were 130EUR here (or more). Pretty high-end devices.

The SoC involved here is 5 years old.  Think about that for a moment.

Several of the devices on that list are no longer on the market, because they've been superseded.  So this is hardware that was moderately high-end (in a consumer CPE context) several years ago.  The only reason we're hearing about this nssfq_codel functionality right now is because the method of turning it on finally got reverse-engineered in the open-source world, no thanks to the hardware vendor.  I wish they would stop shooting themselves in the foot on that score.

I have no doubt that similar functionality is actually commonplace in the better CPE SoCs now, certainly in later Qualcomm products.  It's just not emphasised in the spec sheets because the marketing department doesn't understand it, and getting access to it from open source is like getting blood out of a stone.  But it is *there*.

Sebastian pointed out that this constitutes an existence proof for hardware-accelerated fq_codel functionality in reasonably-priced network-edge devices.  We also know that Approximate Fairness technology is available in ISP-grade hardware.  CPE vendors are actively promoting Airtime Fairness technology on the wifi side, which I happen to know comes along with fq_codel as a side effect.

At this point, any claim that fq_codel is not widely deployed in relevant contexts is laughable.

 - Jonathan Morton

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 28 Nov 2020 17:00:04 +0100 (CET)

> relevant contexts is laughable.

What's your definition on "widely deployed"?

How many percent of customer access lines and wifi access hops do you 
believe has fq_codel (or fq anything) enabled on them?

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 28 Nov 2020 18:12:52 +0200

> On 28 Nov, 2020, at 6:00 pm, Mikael Abrahamsson <[email-masked]> wrote:
> 
> How many percent of customer access lines and wifi access hops do you believe has fq_codel (or fq anything) enabled on them?

If we focus on wifi, Airtime Fairness and the fq_codel that comes with it is now integrated with open-source iwlwifi, mt76, ath9k, and ath10k drivers.  That's Intel, MediaTek, and Qualcomm Atheros wifi hardware basically sewn up from the AP end, where Airtime Fairness actually matters.

Take a random sampling of CPE devices, and see how many of them use the above - or it might be easier to count the ones that don't.  I'll wait.

 - Jonathan Morton

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 28 Nov 2020 17:24:12 +0100 (CET)

On Sat, 28 Nov 2020, Jonathan Morton wrote:

> Take a random sampling of CPE devices, and see how many of them use the 
> above - or it might be easier to count the ones that don't.  I'll wait.

My ubiquiti APs come with it, but it's default turned off.

It's not only important of the devices have the capability, it also needs 
to be default turned on so it's actually used (considering most people 
don't change config much).

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 28 Nov 2020 18:43:45 +0200

> On 28 Nov, 2020, at 6:24 pm, Mikael Abrahamsson <[email-masked]> wrote:
> 
> It's not only important of the devices have the capability, it also needs to be default turned on so it's actually used (considering most people don't change config much).

In that respect, I believe Ubiquiti are in a definite minority.  This is a feature that you have to actively disable in the stock kernel if you don't want it.  I've heard that Ubiquiti tend to use a rather old kernel for, you guessed it, reasons of hardware vendor intransigence regarding open-source support.

As a direct counterpoint, I'll point to TP-Link as a vendor who actively advertises Airtime Fairness as a feature on many of their current products, and not buried in the gritty details of cryptic standards-compliance nomenclature, but as a bullet point on the slick slideshow that consumers are presented with.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 28 Nov 2020 18:06:31 +0100

Hi Mikael,

I actually wanted to stay out of tis sub thread, but...

> On Nov 28, 2020, at 17:24, Mikael Abrahamsson <[email-masked]> wrote:
> 

	[SM] ... if end-user devices have such capabilities, the big ISPs that in all likelihood are the biggest buyers of such devices, should have little issues adding that to their requirements list for their purchases and making sure devices are configured properly, without driving up the costs unduly, no?. 
	The fact that that has not happened is in itself a rather clear indication that ISPs are really not that interested in tackling latency under load (and why should they, often unhappy users will switch to a higher/more profitable speed tier in a well-intended but not very effective attempt at improving a link's usefulness).
	Now, for end-users that is a different issue, and there are quite some, often on-line gamers interested in reducing their internet access link's latency, that are actually willing to buy and configure boutique devices, like the NetDUMA, or more down to earth and less hyped (but IMHO even more capable) evenroute's iqrouter. Incidentally, both are basically software distributions for relative standard existing router's from other manufacturers, and yet people are paying a premium to get those better behaving router OS, as latency under load performance of stock router OSes often is not "cutting it"...
	
	I note that L4S never reached out to those end-users actually caring for low latency... instead they are trying to convince the ISP market, without a narrative how this is going to imprive an ISP's bottom-line (the you need to do this to not fall behind the competition, is not a positive selling point IMHO).

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 30 Nov 2020 09:14:26 +0000

Hi

I am not sure what conclusions I should draw from this. 
Lossy AQMs is a non-issue as both DCTCP and Prague makes a fallback to loss
based recovery so that is settled.

That leaves us with the long discussion RFC3168 AQMs. Here the discussion
leaves me thinking that we are talking about a very limited actual
deployment in other words too much ado about very little. In addition, I get
the impression that these limited deployments are seen as monoliths that
cannot be modified and updated, which is a bit confusing too as fq_codel can
be configured for DCTCP..

So, my conclusion is that we should move on to WGLC, worries about possible
boxes on the network is something that can be quantified during the L4S
experiment.

/Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
> Sent: den 28 november 2020 18:07
> To: Mikael Abrahamsson <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [tsvwg] feedback and thoughts L4S / SCE
> 
> Hi Mikael,
> 
> > It's not only important of the devices have the capability, it also
needs to be
> default turned on so it's actually used (considering most people don't
change
> config much).
> 
> 	[SM] ... if end-user devices have such capabilities, the big ISPs
that in
> all likelihood are the biggest buyers of such devices, should have little
issues
> adding that to their requirements list for their purchases and making sure
> devices are configured properly, without driving up the costs unduly, no?.
> 	The fact that that has not happened is in itself a rather clear
indication
> that ISPs are really not that interested in tackling latency under load
(and why
> should they, often unhappy users will switch to a higher/more profitable
> speed tier in a well-intended but not very effective attempt at improving
a
> link's usefulness).
> 	Now, for end-users that is a different issue, and there are quite
some,
> often on-line gamers interested in reducing their internet access link's
> latency, that are actually willing to buy and configure boutique devices,
like
> the NetDUMA, or more down to earth and less hyped (but IMHO even more
> capable) evenroute's iqrouter. Incidentally, both are basically software
> distributions for relative standard existing router's from other
manufacturers,
> and yet people are paying a premium to get those better behaving router
OS,
> as latency under load performance of stock router OSes often is not
"cutting
> it"...
> 
> 	I note that L4S never reached out to those end-users actually caring
> for low latency... instead they are trying to convince the ISP market,
without a
> narrative how this is going to imprive an ISP's bottom-line (the you need
to do
> this to not fall behind the competition, is not a positive selling point
IMHO).

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 30 Nov 2020 10:51:42 +0100

Hi Ingemar,

I am not sure how your post relates to the one you seemingly replied to? 

> On Nov 30, 2020, at 10:14, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> 

	[SM] If you would look at what L4S actually delivers the conclusion should be that it simply is not ready, but I realize that in spite of hard supporting data your assessment ist different.

> based recovery so that is settled.

	[SM] Well, not really, both DCTCP and TCP Prague are atrocious with medium length RTTs and non-L4S AQMs (unless that AQM is flow fair), they are simply not ready for being used over the wider internet. In a sense that DCTCP is not suitable as an internet-wide protocol (without significant changes) is one of the results of the L4S experiments, as is the fact that TCP Prague does not yet? describe a complete set of required fixes (just look at TCP Pragues performance versus CUBIC in a purely drop based FIFO buffer). And Koen admitted as much when he proposed to change TCP Prague to behave like CUBIC if the RTT exceeds ~80ms. 
	I am still waiting for a demonstration of L4S robustly and reliably delivering on its promises over anything but the low hop-count short RTT paths that have been over-tested so far. 

> deployment in other words too much ado about very little.

	[SM] citation needed? The challenge is that we have no reliable numbers, one way or the other. Your assessment, says more about your leaning towards L4S than about the deployed reality in the internet. 

> In addition, I get
> cannot be modified and updated,

	[SM] So I wonder in what kind of blessed reality you seem to be living? In my experience updates for most consumer-grade networked equipment are pretty rare and often stop way before the useful live of such devices, assuming all of these can/will be updated in time is simply unrealistic. Any proposal that relies on that has the onus of convincing the reviewers why this time around this will be a successful approach. 

> which is a bit confusing too as fq_codel can

	[SM] The latter point might be true, but will fq_codel with your proposed configuration change still perform equally well with standard-compliant rfc3168 ECN flows? These are and will be for a considerable time be the dominant ECN users on the internet. 

> boxes on the network is something that can be quantified during the L4S

	[SM] Well, but then the experiment can not be "how to deploy L4S" maybe "can we deploy L4S" might be a worth while question, but it has become clear that nobody in team L4S is even willing to offer contingency plans what to do in case their hopes do not come true. How to re-claim ECT(1), what to do with deployed L4S dual-queue AQMs, what to do with endpoints that configured TCP Prague as default CC, all questions that should have be discussed in the L4S internet drafts IMHO, indicating that in addition to L4S' design and implementation not being ready, the drafts are equally half-baked (heck there isn't even an internet draft for L4S' reference CC implementation in TCP Prague at all, as far as I can see). Ready it ain't, so why rush to WGLC?

Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 30 Nov 2020 17:19:40 +0200

> On 30 Nov, 2020, at 11:51 am, Sebastian Moeller <[email-masked]> wrote:
> 

Bearing in mind that this is the Internet *Engineering* Task Force, when lacking reliable data the instinct should be to try and obtain it and to make pessimistic assumptions meanwhile, rather than to make optimistic assumptions.

For L4S, the appropriately pessimistic assumptions should, until clearly shown otherwise, be that RFC-3168 middlebox deployment is high but invisible, and that eliminating the majority of this deployment (in favour of L4S-aware middleboxes) would take several years.  The burden of proof is on L4S proponents to show otherwise, given the safety risks already demonstrated of deploying L4S into an RFC-3168 network.

At present the data clearly shows a non-zero active deployment rate of both RFC-3168 endpoints and RFC-3168 middleboxes.  Passive deployment (ie. not enabled by default) of RFC-3168 endpoints is much higher, constituting the overwhelming majority of current desktop and mobile operating systems.  Additionally, L4S clients communicating with existing RFC-3168 servers will generate RFC-3168 traffic from the server to the client, due to details of AccECN negotiation on the three-way handshake.  That much is (or should be) completely uncontroversial.

It is further believed that many RFC-3168 middlebox deployments are presently not easy to observe in traffic statistics, either because they are not always (or not often) bottlenecked at that point, or because the endpoints did not negotiate RFC-3168 capability for the observed traffic.

The former case (not deployed at the bottleneck) may result in AQM activity later, when the bottleneck shifts to the link where it is deployed, perhaps due to ISP-side upgrades or changing the location of wifi devices - or when the observed traffic moves to a closer endpoint in the network, such as a local CDN.  This last suggests that L4S traffic might see more AQM activity due to deployment on an advantageously-positioned CDN, than the existing vantage points from which data has so far been collected.  NB: even if the ISP network has been made L4S-aware prior to the CDN deployment, the end-user's network might not have been, and this is generally outside the ISP's control.

The latter case (non-negotiation of ECN) could still result in AQM activity that would only be observable as packet loss, and thus is difficult to distinguish from a dumb FIFO without deeper analysis.  That is the major source of uncertainty in the data.  This could become more visible if active (rather than passive) deployment of RFC-3168 endpoints increases, which would most likely be from changing the default setting in major operating systems.  Presently the default in both Windows and Linux is passive only.

I will remind readers that although TCP Prague responds appropriately with Multiplicative Decrease upon packet loss, that does not help it when encountering an RFC-3168 AQM that it shares with conventional AIMD traffic, even if the latter is Not-ECT.  This is because the RFC-3168 AQM will mark the L4S traffic at the same rate as it drops Not-ECT traffic and marks RFC-3168 traffic.  This therefore forces TCP Prague to rely on secondary protections such as FQ to avoid starving conventional traffic, but FQ is typically defeated by tunnel encapsulations.

All of the above will be repeated as open issues at any L4S WGLC, unless and until it is clearly shown to be false.  That, I think, should give the L4S team pause when calling for WGLC.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 30 Nov 2020 19:13:51 +0000

Here's another plea: If more people that those discussing amongst 
themselves feel there are other issues with letting this experiment 
progress, then they probably should let the list know - or directly 
contact the chairs.

Is there an important case where "RFC-3168 middlebox deployment is high, 
but invisible" (see below)?

Gorry

-- 
G. Fairhurst, School of Engineering