Subject: [tsvwg] Bi-directional asymmetric link test results, and Prague self-starvation

From: Pete Heist <[email-masked]>
Date: Sat, 9 May 2020 20:29:50 +0200

We received some valid criticism at the last interim meeting that we hadn’t published any bi-directional traffic tests, on asymmetric links in particular. In response to that, I’ve added scenarios 8 and 9 to our latest test batch.

The results raise more questions about classic bottleneck detection, and the performance of Prague under conditions we’ve not tested before. What we appear to see is, it’s possible for Prague flows to also starve other Prague flows, because the bottleneck detection heuristic can arrive at different conclusions at different times when there are competing flows, even without the presence of induced jitter.

One of our concerns about heuristic based classic bottleneck detection is that it will be turned off due to its unreliability. From there, the path to starvation of competing flows is pretty straightforward. As an example, if there is a Prague flow that hashes to the same fq_codel bucket as an RFC3168 ECN flow, starvation of the RFC3168 flow *will* occur. fq_codel has 1024 buckets by default. In high capacity backhaul links that use fq_codel (like my ISP has/does), hash collisions will happen reasonably often.

We hope the potential for starvation is understood and taken seriously. So, without further ado...

*** Scenario 8: Four flows each direction over typical cable modem link rates, 10/200Mbit, and 35/1000Mbit

* In the test below with TCP Prague and dualpi2, 10/200Mbit at 80ms RTT, we can see that one Prague flow (the violet trace) has detected the bottleneck as classic (which it’s not). It is then driven to starvation by the other Prague flows some time after t=40s, before those flows also decide it’s a classic bottleneck shortly before t=60s. They then proceed, in phase, as classic flows, underutilizing the link:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

Run to run, this behavior can differ, as we can see in the second run of the same test:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

* Here is the same test as above with CUBIC-SCE and Cake, followed by LFQ, SCE's FQ emulation using two queues:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-cake-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-lfq_cobalt-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

* In the test below with TCP Prague and dualpi2, 35/1000Mbit at 20ms RTT, we can also see flows that appear to detect the bottleneck as classic, and note some under-utilization in the box plot:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-35Mbit-1Gbit-20ms_box_totals.svg

* Here is the same test as above with CUBIC-SCE and cake, followed by LFQ. In the LFQ case as well as some other Twin-CodelAF plots not linked in this email, we’re aware of some throughput drops that are under investigation, however overall utilization is close to capacity:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-cake-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-cake-35Mbit-1Gbit-20ms_box_totals.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-lfq_cobalt-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-lfq_cobalt-35Mbit-1Gbit-20ms_box_totals.svg

* In the second run of scenario 8, we included some tests with pfifo and Linux’s default limit of 1000 packets, since those are still in common use on the Internet. Here is TCP Prague in a pfifo at 35/1000Mbit with 20ms RTT:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-pfifo_1000_-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-pfifo_1000_-35Mbit-1Gbit-20ms_box_totals.svg

* Here is the same test as above with CUBIC-SCE, which is about the same as plain CUBIC:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-pfifo_1000_-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-pfifo_1000_-35Mbit-1Gbit-20ms_box_totals.svg

*** Scenario 9: Four flows each direction over a 1Mbit uplink, 1/10Mbit and 1/50Mbit

* In the test below with TCP Prague and dualpi2, 1/10Mbit at 20ms RTT, we see large drop in utilization for some reason, accompanied by a significant increase in the ping time (reflecting the queue depth) over the RTT:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/l4s-s9-bidir-asym-1mbit/l4s-s9-bidir-asym-1mbit-ns-prague-dualpi2-1Mbit-10Mbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/l4s-s9-bidir-asym-1mbit/l4s-s9-bidir-asym-1mbit-ns-prague-dualpi2-1Mbit-10Mbit-20ms_box_totals.svg

* Here is the same test as above with CUBIC-SCE in LFQ:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s9-bidir-asym-1mbit/sce-s9-bidir-asym-1mbit-ns-cubic-sce-lfq_cobalt-1Mbit-10Mbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s9-bidir-asym-1mbit/sce-s9-bidir-asym-1mbit-ns-cubic-sce-lfq_cobalt-1Mbit-10Mbit-20ms_box_totals.svg

* We will also look further into why utilization is closer to capacity when using plain CUBIC and fq_codel:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s9-bidir-asym-1mbit/sce-s9-bidir-asym-1mbit-ns-cubic-fq_codel-1Mbit-10Mbit-20ms_box_totals.svg

*** Full results from both runs are here:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 9 May 2020 22:44:10 +0200

Hi Pete, hi list,

Thank you very much for this comprehensive set of tests that constitute a ton of work.

I have been claiming that L4S offer relatively little over the status quo in AQM for home networks and I want to point the lists attention at comparing the performance of different solutions for a traffic scenario that I consider important, bi-directional traffic on an asymmetric link at saturation (call it, the corona test). I will restrict myself to the 10/200 Mbps tests at 80ms RTT as these seem to be typical values for end-users accessing servers over the internet (as compared to a "highway to the nearest data center").

1) http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-cubic-dualpi2-10Mbit-200Mbit-80ms_all_scaled_delivery.svg
testing dualpi2's classic queue

2) http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-10Mbit-200Mbit-80ms_all_scaled_delivery.svg
testing  dualpi2's LL queue

3) http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-cake-10Mbit-200Mbit-80ms_all_scaled_delivery.svg
testing cake with cubic flows

Comparing 1) with 3) makes it clear that L4S' classic side AQM performance is underwhelming, both for aggregate download though-put as well as latency under load increases.

Comparing 2) with 3) demonstrates that L4S' LL queue with its own preferred reference transport protocol manages to keep latency under load increase in the same range as standard fq-cake can with cubic, and that interflow differences in 2) approach starvation levels (@team L4S have you ever tested that, like at all?). 

Looking at the same plots with an RTT of 20ms, dualpi2's classic queue still performs considerably worse, and dualpi2's LL queue looks at best equal to what cake can do with bog-standard CUBIC.

	In summary under these conditions and assuming the tests correct (which I have zero reason to doubt) L4S offers little to no advantage over the state of the art. I dare say it, "the emperor has no clothes".

	Now, I would wish that everybody here proposing that L4S considerable side-effects are acceptable, clarifies in their own mind what gains in L4S justify that trade-off. And since the trade-offs are real, please consider actual L4S performance and not descriptions from L4S' RFCs (unless demonstrated to be achievable in the real world these are really just promises).

Best Regards
	Sebastian

P.S.: I note again, that my claims do not involve SCE at all, this is really comparing L4S against existing tested AQM technology and against its own claims. At the current time I personally see zero rationale to permit L4S with any side-effects at all.

---

From: Dave Taht <[email-masked]>
Date: Sun, 10 May 2020 19:51:41 -0700

To clarify some things?

TCP Prague had no response to loss in the ll queue until the day after
the last videoconference? And: Most the papers written about it
assumed no
possibility of loss in that ll queue?

Anyway:

Before I lose a week of my life to try to delve into the latest
results posted, which at first glance appears to reveal some bugs
somewhere in the stack and/or tests, it would be great if the l4s team
and *especially* other observers here could also repeat them
independently and take a hard look at them? I don't want to get
involved until both sides agree they are accurate, really, there's
been enough heat on this list on stuff that was just bugs, to last a
century.

Participants here can mirror the entire dataset via:

wget -m -l 6 -e robots=no http://sce.dnsmgr.net/results # I think 6
levels is enough.

and then fly through them, generating comparison plots, with the
flent-gui. If you don't need the captures, you can exclude those with
a wget option.

SCEFolk: Thx so much for running this very core, basic test in the
bufferbloat out of 100+ test flent test suite! There's so many more in
there that are useful, and
I now realize the the endless unidirectional two-four flow tests I'd
got so tired of in this debate were necessary for debugging the code,
and I'd like to apologize publicly to you for losing my temper over it
last week. I had thought development of prague was much father along
than it was.

Moving forward...

Even better, it would be so great to have baseline, reference test
results on the rrul_be test against a couple common
cablemodem and cmts configurations, and the l4s side is better
situated to plunk the real, deployed numbers in.

I don't know what these are nowadays. Back when the bufferbloat.net
team members did the original work on (2011-2013) pie and fq_codel,
(laboring over packet captures mostly, in the stone knives and
bearskins era of AQM and FQ research), we used the following matrix
for emulating real world cablemodem and cmts behavior:

cablemodem up bfifo 64k (docsis 2.0), 256k (docsis 3.0)
cmts down bfifo 512k and 1Mbyte
against the 11/2, 100/10, 200/10, and 1gbit/35mbit services?

I've gone and verified that my realworld reference 100/10
implementation, still running the same code and hw, and the same cmts
we deployed against in 2014, behaves this way. From what I see in the
real world, the uplink bfifo has been cut to 48k or less on some
modems at least, and a goodly percentage, but far from all, of the
docsis 3.1 modems in the world still have at least 64k of buffering,
and a goodly percentage still do not have pie turned on. I have some
folk
turning in derived estimates as to the type and size of buffering over
the rest of the common cable service configurations.

There's dualpi results in the SCEteam's dataset already, so, just
getting this reference data would really help me in trying to connect
the dots from what
we did way back when to what's been happening of late.

Thx!