Subject: [tsvwg] [https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10] Question about guarantees between sharing of the two queues

From: Sebastian Moeller <[email-masked]>
Date: Sat, 9 Nov 2019 23:32:09 +0100

Sear All,

re-reading https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10 I notice

"Analytical study and implementation testing of the Coupled AQM have
   shown that Scalable and Classic flows competing under similar
   conditions run at roughly the same rate."

And later:

"4.1.1.  Avoiding Classic Starvation: Sacrifice L4S Throughput or Delay?
   Priority of L4S is required to be conditional to avoid total
   starvation of Classic by heavy L4S traffic.  This raises the question
   of whether to sacrifice L4S throughput or L4S delay (or some other
   policy) to mitigate starvation of Classic:

Sacrifice L4S throughput:   By using weighted round robin as the
      conditional priority scheduler, the L4S service can sacrifice some
      throughput during overload.  This can either be thought of as
      guaranteeing a minimum throughput service for Classic traffic, or
      as guaranteeing a maximum delay for a packet at the head of the
      Classic queue.
      The scheduling weight of the Classic queue should be small (e.g.
      1/16).  Then, in most traffic scenarios the scheduler will not
      interfere and it will not need to - the coupling mechanism and the
      end-systems will share out the capacity across both queues as if
      it were a single pool.  However, because the congestion coupling
      only applies in one direction (from C to L), if L4S traffic is
      over-aggressive or unresponsive, the scheduler weight for Classic
      traffic will at least be large enough to ensure it does not
      starve.
[...]
"

Recent testing has shown that such starvation is not a theoretical consideration, but for short RTTs rather seems the normal case. This brings me to me question:

Question: How to fix this obvious discrepancy between the claim of "run at roughly the same rate" and basically only implementing an anti-starvation backstop? Either the claim needs to be adjusted to Scalable traffic does not fully starve classic traffic, dualQ needs to be fixed and the "backstop" needs to be changed so that under saturating load, both traffic types truly get ~50% of the egress capacity.

 As it stands IMHO the observed sharing behavior seems insufficient to recommend using dualQ on the wider internet, see https://l4s.cablelabs.com/l4s-testing/key_plots/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.png for how  dualQ fails to properly share capacity between a single TCP Prague flow (1/p) and a single TCP Cubic flow (1/sqrt(p)) at a short RTT, unless approximately 9:1 is considered the proper share between L4S-style and normal TCP traffic.

Best Regards
	Sebastian

P.S.: I have brought up this point repeatedly, because I consider this to be important to address ASAP, and did not get any feed-back yet.

---

From: Dave Taht <[email-masked]>
Date: Sat, 09 Nov 2019 15:05:54 -0800

It is really wonderful to have these two groups using the
flent tool and producing comparable data on the same test suites in
a few cases now, and the testing so far found a couple bugs.

https://github.com/heistp/sce-l4s-bakeoff
https://l4s.cablelabs.com/

I do feel rather strongly that any major change to how the internet does
congestion control requires a maximum of publically repeatable
benchmarks using open, shared source code. [1]

I do keep hoping for a rrul test on a 15 to 20x1 asymmetric network
simulation like how most cablemodems and dsl boxes are configured, and
more of the rtt_fair_* tests, and what the heck - the tcp_square_wave
test, and any of a dozen others. tcp_nup with 32 flows... heck, hundreds
of flows... go to town! Burn some cycles!

A few packet caps would be nice to have also, to look
over relative mark rates in tcptrace and scetrace.

Anyway... l4steam....

In looking over the l4s.cablelabs site today though, I see the data is
only available as .png files, and that the associated flent.gz data
is not apparently being published.

.svgs help. Flent can also be run at a higher sample rate... but

There are a few things about having *.flent.gz files available that
would help us look at some things in more detail. Me, I tend to look at
comparison cdf plots more than anything else, and secondly bar plots,
and lacking that it's hard to make a distinction betwen a whole bunch of
tiny lines at the bottom about overall queue depths and latency
distributions, among other things, without squinting, and doing tricks
in photoshop to make the two or more datasets "line up".

(there's dozens of plot types in the flent tool, and comparison plots
are a snap)

By default (when run without -x) flent captures very little metadata
about the system it is run on (IP addresses, a couple sysctls, and
qdiscs) but it's helpful to have. One example that would be in that
metadata, is that I'm unsure if the ns3 data is using an IW4 or IW10?

It sounds like you are rate limiting with htb? (to what quantum?)

Another example, in more "native" environments running at a simulated
line rate, BQL is quite important to have in the simulation
also. there's been a couple papers published on BQL's benefits vs a raw
txring, thus far, there's a good plot of what it does in fig 6 of:

http://sci-hub.tw/10.1109/LANMAN.2019.8847054

Lastly...

So far as I know ns(X) does not correctly simulate GSO/TSO even when
run in DCE mode, but I could be out of date on that. TBF (and cake)
do break apart superpackets, htb (+ anything, like fq_codel or dualq)
do not.

I also have completely forgotten at this point, to what extent the
ns3 model of fq_codel matches the deployed reality. It was such a long
time ago.... as best as I recall ns3 didn't even have ecn support then!

I'm taking apart
https://gitlab.com/tomhend/modules/l4s-evaluation/blob/master/README.md
as I write.

thx for the data!

[1] If anyone hasn't read my related rant yet, it's here:
https://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sat, 9 Nov 2019 19:31:23 -0800 (PST)

Dave,
	Thank you for the links, some short responses inline.

Regards,
Rod

> of flows... go to town! Burn some cycles!

I believe Pete has some work scheduled for the (near?) future
in this area.

> over relative mark rates in tcptrace and scetrace.
I do believe all you seek for Petes data is there,
https://github.com/heistp/sce-l4s-bakeoff/#test-output
specifically items:
*.flent.gz- standard Flent output file, may be used to view original data or re-generate plots
*.tcpdump_*.pcap.bz2- pcap file compressed with bzip2 -9

> ..svgs help. Flent can also be run at a higher sample rate... but
> 

And apparently is still missing some ECN (RFC3168) support as your
following link contains some patches:
	6.  cd back one level to the top-level ns-3 directory,
	and patch the ns-3.30.1 release with a patch provided
	in the l4s-evaluation/patches/ directory. This patch
	adds support for ECE handling and DCTCP in the base
	TCP code, and adds ECN handling to CoDel and FqCoDel models.

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
> 

-- 
Rod Grimes                                                 [email-masked]

---

From: Dave Taht <[email-masked]>
Date: Sat, 09 Nov 2019 20:50:02 -0800

> 	Thank you for the links, some short responses inline.

I did a bit more data review of the l4s data today.

https://l4s.cablelabs.com/results/6-dctcp-20191018-214743/6-dctcp_25_50Mbps_0.95_0ms.pdf

has an interesting growth line in the middle series of charts.

Two anecdotes: 1) arguably the internet is so messed up because everyone
has optimized for speedtest's 20s test run rather than a longer
period. 2) I lost much hair to this maddening one: http://blog.cerowrt.org/post/disabling_channel_scans/

It would be good to have a test that also ran for more than a minute,
say 5, or 10, so that a global trend be seen.

> I believe Pete has some work scheduled for the (near?) future

I will sleep so much better about both proposals once a rrul test
is run, and then from there accecn can be better evaluated.

thx everybody for your efforts on this!

> *.tcpdump_*.pcap.bz2- pcap file compressed with bzip2 -9

Groovy. Don't stop doing that. ;)

>> It sounds like you are rate limiting with htb? (to what quantum?)

?

>> line rate, BQL is quite important to have in the simulation

?

>> run in DCE mode, but I could be out of date on that. TBF (and cake)

?

>> time ago.... as best as I recall ns3 didn't even have ecn support then!

I looked over the ns3 fq_codel implementation, and (by mark one eyeball)
it appears largely correct - nowadays we increase the codel count in the
bulk dropper, but unless there's a specific flooding test (?) in the
suite, we're not going to hit that. Might be able to fix that once I get
used to writing OOP again.

More important to verify is the actual endpoint setup in the experiment
to make sure a real 5 tuple is in use....

The ns3 codel code matches the linux implementation, also.

I'd still have to spin up the test suite to actually trust it, and
I was (2012) pretty sure the cubic implementation differed
significantly from linuxes. 

> 	TCP code, and adds ECN handling to CoDel and FqCoDel models.

Well, I am *happy* to see it was the current release of ns3 + a little
bit. That's a relief.

That said - oy - um, er - this being the first ever use of ECN in
ns3 requires quite a bit of validation. From a quick look at this patch,
simply off the top of my head - it doesn't follow rfc3168's
recommendation in section

6.1.5.  Retransmitted TCP packets

"This document specifies ECN-capable TCP implementations MUST NOT set
   either ECT codepoint (ECT(0) or ECT(1)) in the IP header for
   retransmitted data packets, and that the TCP data receiver SHOULD
   ignore the ECN field on arriving data packets that are outside of the
   receiver's current window.  "...

I think (need to look harder)

... as much as some folk here would like to obsolete that, linux does
actually follow this portion of the spec (I don't know about other OSes)
, and it has long been one of my concerns regarding l4s (and cake's)
interactions with loss and flipping between queues in general.

---

From: "Tilmans, Olivier (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 10 Nov 2019 23:59:20 +0000

Hi Dave,

> is not apparently being published.

If you follow the "full testbed results" link,
you end up on this page which links to the raw data generated with Pete's
tests in its 2nd sentence/§:
https://drive.google.com/open?id=1tbRBWkvxyEeDzT5u3EVynupXvEjG8MMb

Best,
Olivier

---

From: Dave Taht <[email-masked]>
Date: Sun, 10 Nov 2019 16:17:59 -0800

Tom Henderson <[email-masked]> writes:

> Dave,
> I missed replying earlier to a few of your points below (inline).
>
> On 11/9/19 3:05 PM, Dave Taht wrote:
>
>>
> in the ns-3 simulations posted, BQL is enabled on all links.

Cool. If only the dsl and cable worlds had adopted this! it allows for
much smarter handling of packet delivery higher in the stack at the cost
of one interrupt's worth of standing queue. Without BQL we wouldn't be
scaling linux past 10GigE today.

I keep hoping *switches* will start doing bql, also.

Note:

The AQL work (long in google wifi and a few other places) - "airtime
queue limits" - is finally entering linux mainline. This makes for a
similar savings to BQL at interrupt time on media with variable "line
rate" encodings, extremely useful for wifi and lte and possibly for
cable and ethernet over powerline devices.

Some details on how well it works are in the google document here:

http://flent-newark.bufferbloat.net/~d/Airtime%20based%20queue%20limit%20for%20FQ_CoDel%20in%20wireless%20interface.pdf

Toke's current patchset: https://patchwork.kernel.org/cover/11206223/

> Correct, we do not have an ns-3 model for GSO/TSO.  Is it needed (in
> the simulation) if BQL is enabled with small device queues?

I don't know. Are you seeing GRO/GSO/TSO superpackets in the path on
this simulation? It isn't on for a variety of pseudo devices,
particularly in older releases of linux.

The 4.4 kernel (released 2016-01-10) you are leveraging predates some
major new network subsystem features, like better pacing, sch_etx, and
the switch to EDF scheduling (in linux 5.1). I'm sure there are dozens
more things that might matter, which is why we test. :) I imagine neal
is more up on all the changes since 4.4 that might matter.

In the field:

We started running into problems with GRO starting in 2014, where gige
line rate bursts of IW10,20,30,42 or more packets hit certain brands of
home router hardware, and were then released as a single superpacket. The
resulting load spike (stepping down, to, say, 5mbit) was quite noticible
when doing concurrent voip applications, and also codel tended to be late
in recogizing stuff past the "burp", it didn't matter how much FQ we had
in certain scenarios when dealing with superpackets.

back in 2015, we'd had to put in commit:
a5d28090405038ca1f40c13f38d6d4285456efee
to get GSO even remotely right.

Anyway...

After much gnashing of teeth and pulling of hair while trying to
selectively disable or split gso on various pieces of consumer hw,
in the end we made gso-splitting the default in cake, rather than
try to turn off GRO everywhere it was needed, or revise the htb shaper
so fq_codel could be used more effectively with GSO in place. (as I said
"tbf" does splitting, htb - which is what most sqm systems use - does not).

fq_codel_fast (which has sce support) also
splits GSO. I'd filed a bug on the L4S github site requesting that they
also implement GSO splitting and use memory, not packet limits, a while back.

Given how fine grained either SCE or L4S need to be on their signalling,
my assumption (needing testing!), is that GSO/GRO/TSO need to be
disabled to get an expected result, and with GSO/GRO/TSO on, especially
at lower rates, the results will be "interesting", and thus in the field
with the final delivered codebase(s), it needs to be handled properly by
the qdisc either way it goes.

Last year GSO went always on in linux kernels, which generally means
that locally sourced tcp packets are always 2 packets in size or larger.
the popular sch_fq scheduler has a 2 packet quantum also. This is
great, if you are targetting 40gige....

commit 0a6b2a1dc2a2105f178255fe495eb914b09cb37a
Author: Eric Dumazet <[email-masked]>
Date:   Mon Feb 19 11:56:47 2018 -0800

    tcp: switch to GSO being always on
    
    Oleksandr Natalenko reported performance issues with BBR without FQ
    packet scheduler that were root caused to lack of SG and GSO/TSO on
    his configuration.
    
    In this mode, TCP internal pacing has to setup a high resolution timer
    for each MSS sent.
    
    We could implement in TCP a strategy similar to the one adopted
    in commit fefa569a9d4b ("net_sched: sch_fq: account for schedule/timers drifts")
    or decide to finally switch TCP stack to a GSO only mode.
    
    This has many benefits :
    
    1) Most TCP developments are done with TSO in mind.
    2) Less high-resolution timers needs to be armed for TCP-pacing
    3) GSO can benefit of xmit_more hint
    4) Receiver GRO is more effective (as if TSO was used for real on sender)
       -> Lower ACK traffic
    5) Write queues have less overhead (one skb holds about 64KB of payload)
    6) SACK coalescing just works.
    7) rtx rb-tree contains less packets, SACK is cheaper.
        This patch implements the minimum patch, but we can remove some legacy
    code as follow ups.
    
    Tested:
    
    On 40Gbit link, one netperf -t TCP_STREAM
    
    BBR+fq:
    sg on:  26 Gbits/sec
    sg off: 15.7 Gbits/sec   (was 2.3 Gbit before patch)
    
    BBR+pfifo_fast:
    sg on:  24.2 Gbits/sec
    sg off: 14.9 Gbits/sec  (was 0.66 Gbit before patch !!! )
    
    BBR+fq_codel:
    sg on:  24.4 Gbits/sec
    sg off: 15 Gbits/sec  (was 0.66 Gbit before patch !!! )
    
    Signed-off-by: Eric Dumazet <[email-masked]>
    Reported-by: Oleksandr Natalenko <[email-masked]>
    Signed-off-by: David S. Miller <[email-masked]>

> - Tom

---

From: Dave Taht <[email-masked]>
Date: Sun, 10 Nov 2019 16:49:07 -0800

"Tilmans, Olivier (Nokia - BE/Antwerp)"

> https://drive.google.com/open?id=1tbRBWkvxyEeDzT5u3EVynupXvEjG8MMb

Thanks! I missed that! I'm watching google take ages to attempt to zip
up the directory, and delighted to be able to use flent natively on the
result.

(if you can't tell, I haven't been paying much attention to this stuff,
 I popped up this weekend to catch up)

---

From: Tom Henderson <[email-masked]>
Date: Sun, 10 Nov 2019 19:13:35 -0800

Reposting an earlier message

> -------- Forwarded Message --------
> Subject: Re: [tsvwg] sce vs l4s comparison plots?
> Date: Sun, 10 Nov 2019 14:27:58 -0800
> From: Tom Henderson <[email-masked]>
> To: Dave Taht <[email-masked]>, tsvwg IETF list <[email-masked]>, [email-masked]
>
>

Dave, I missed replying earlier to a few of your points below (inline).

- Tom

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 11 Nov 2019 05:21:45 +0200

> On 11 Nov, 2019, at 5:13 am, Tom Henderson <[email-masked]> wrote:
> 

GSO could form superpackets on egress even with BQL, if the latter chooses a queue depth of more than one MTU.

Also, GSO often forms superpackets on receive, so that when subsequently forwarded to an egress interface the queue has to handle them as such; receive buffers are not affected by BQL at all.

 - Jonathan Morton

---

From: Tom Henderson <[email-masked]>
Date: Sun, 10 Nov 2019 19:31:12 -0800

Dave, responses inline below.
> particularly in older releases of linux.

No, neither mode is modeling offload.  My question was about whether,
if BQL is being modeled, and we assume traditional MTU (offload is
off, or offload is on and the packets are split), the resulting pacing
due to BQL is sufficient.  If not, some pointers to recommended ways
to model it would be welcome.

> is more up on all the changes since 4.4 that might matter.

Understood; we are looking for a volunteer to help us with getting
newer kernels enabled in DCE.

- Tom

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 11 Nov 2019 11:41:05 +0000 (UTC)

(see below)

    On Monday, November 11, 2019, 12:19:26 AM GMT, Dave Taht <[email-masked]> wrote:  

[AB] at the CO end (ie downstream) my understanding (which may be out of date) 
is that DSL ICs usually punt queuing to a switch IC. There's a standard (G999.1) which existssolely to enable this (it provides per-subscriber backpressure over ethernet). So, switches doingBQL is exactly what would be required for dsl to do BQL (in the downstream direction).
(Given BQL is a workaround, switches would want to implement a direct solution if they do anything).
Alex

---

From: Dave Taht <[email-masked]>
Date: Mon, 11 Nov 2019 10:25:08 -0800

"[email-masked]" <[email-masked]> writes:

> is that DSL ICs usually punt queuing to a switch IC. There's a
> standard (G999.1) which exists
> solely to enable this (it provides per-subscriber backpressure over
> ethernet). So, switches doing
> BQL is exactly what would be required for dsl to do BQL (in the
> downstream direction).

Given that a lot of stuff has moved to a purer software solution,
slicing, sdn, etc, including switching, I can imagine this would
move more into software, also - if folk were still producing new
dsl gear.

One thing that has always been unclear to me is where
dsl subscriber software rate limits are enforced. 

> solution if they do anything).

I'd hope so!

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Tue, 12 Nov 2019 12:19:52 +0100

> dsl subscriber software rate limits are enforced.

Generally either by forcing the DSL modem to sync up at a rate no higher
than that which the customer has bought; or (more often these days I
think) on a completely separate box that does shaping (or policing) in
the network.

I've seen small ISPs build such boxes using Linux and HTB - and Jesper
and I have been helping a few doing it right (scaling across CPUs, and
using FQ-CoDel on the leaves).

-Toke

---

From: Steven Blake <[email-masked]>
Date: Tue, 12 Nov 2019 10:33:33 -0500

> dsl subscriber software rate limits are enforced. 

In the BRAS. Historically these have been built with quite
sophisticated hierarchical QoS schedulers implemented in custom
silicon. The downstream aggregation switches had tiny buffers and no
meaningful QoS features.

Regards,

// Steve

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Tue, 12 Nov 2019 17:24:42 +0100

Steven Blake <[email-masked]> writes:

> In the BRAS.

Heh. In Danish, 'bras' literally means 'junk'. I'm sure that's just a
coincidence :)

-Toke