Subject: [aqm] some comments I'd made on the cablelabs study privately

From: Dave Taht <[email-masked]>
Date: Sun, 9 Feb 2014 18:34:55 -0800

Since that study and test design were highly influential on the AQM
requirements draft, I am going to publish now here what my comments
were at the time, with a couple updates here and there....

I am not sure if any of the ns2 code from the last round made it out
to the public. ?

* Executive summary

  The Cablelabs AQM paper was the best simulation study of the effects
of the new bufferbloat-fighting AQMs verses the common Internet
traffic types in the home that has been created to date...

  However, it is only a study of half the edge network. It assumes
throughout that there are no excessive latencies to be had from the
CMTS side. Field measurements show latencies in excess of 1.8 seconds
from the CMTS side. 300ms on Verizon gpon. Buffer sizes in the range
of 64k to 512k on DSL in general, with some RED, SFQ, and SQF actually
deployed there.

  So... while the focus has been on what is perceived as the larger
problem, the cable modems themselves, downstream behavior was not
studied and the entire simulation set to "reasonable" values for ns2
modelers... and not seen in the real world.

  In the real world (RW) Flows are almost always bidirectional. What
happens on the downstream side affects the upstream side and vice
versa, as per Van Jacobson's "fountain" analogy. Correctly
compensating for bidirectional TCP dynamics is incredibly important.

  The second largest problem with original cablelabs study is that it
only analyzed traffic at one specific (although common) setting for
cable operators, 20Mbits down, and 5 Mbits up. A common, lower setting
should be analyzed, as well as more premier services. Some tweaking of
codel derived technologies (flows and quantum), and of pie (alpha and
beta) are indicated both at lower and higher bandwidths for optimum
results.  Additionally the effects of classification, notably of
background traffic, has not been explored.

  There are numerous other difficulties in the simulations and models
that need to be understood in order to make good decisions moving
forward. This document goes into more detail on those later.

All the AQMs tested performed vastly better than standard FIFO drop
tail as well as buffercontrol. They all require minimal configuration
to work. With some configuration they can be made to work better.

* Recomendations I'd made at the time

** Study be repeated using at least two more bandwidth settings
** More exact emulation of current CMTS behavior
   Based on real world measurements
** Addition of more traffic types, notably VPN and Videoconferencing
** Improvements to the VOIP, web models
** Continued attempts at getting real world and simulated benchmarks
to "line up".

My approach has been to follow the simulation work and try to devise
real world benchmarks that are similar, and feed back the results into
the ongoing simulation process. There are multiple limitations in this
method, too, notably getting repeatable results, and doing large scale
tests on customer equipment, both of which are subject to heisenbugs.

* Issues in the cablelabs study
** Downstream behavior
   Tests with actual cablemodems in actual configurations shows a
significant amount of buffering on the downstream. At 20Mbits, DS
buffering well in excess of 1 second has been observed. The effect of
excessive buffering on this side has not been explored in these tests.

Certain behaviors - TCP's "burstiness" as it opens it's window to
account for what it is thinking as a long path - reflect interestingly
on congestion avoidance, on the downstream, and the effects on the
upstream side of the pair are interesting too.

I note that my own RW statistics were often very skewed by some very
bad ack behavior on TSO offloads that has been a bug in Linux for
years and recently fixed.

** Web model
*** The web model does not emulate DNS lookups.
      Caching DNS forwarders are typically located on a gateway box
(not sure about cablemodems ??), and the ISP locates a full DNS server
nearby, (within 10ms RTT). DNS traffic is particularly sensitive to
delay, loss and head of line blocking, and slowed DNS traffic stalls
subsequent tcp connections on sharded web traffic in particular.

*** The web model does no caching
    A fairly large percentage (not high enough) of websites make use
of various forms of caching, ranging from marking whole objects as
cachable for a certain amount of time, or using the etags method to
provide checksum-like value for an if-modified get request. Use of the
former method eliminates a RTT entirely, the latter works inside of a
http 1.1 pipeline well.

*** The web model does not use https
    Establishing a secure http connection requires additional round trips.

*** The web model doesn't emulate tons of tabs
    Web users, already highly interactive, now tend to have tons of
tabs, all on individual web sites, many of which are doing some sort
of polling or interaction in the background against the remote web
server. These benchmarks do not emulate this highly common behavior.

** TCP Cubic in ns2 is not the same as modern TCP cubic in Linux

Showing that these new AQMs work correctly with modern TCPs is essential.

** VOIP model
   The VOIP model measures *one-way* egress delays only, and does not
track the burstiness of packet loss, or the burstiness of jitter. It
uses an archaic codec emulation, as well. Something like opus should
be looked at.

** The gaming model
   Treats gaming as equivalent to VOIP, with the same problems as the
VOIP model. Gaming traffic is bidirectional, even more so than voip.

Additionally, some of the traces I have access to use a
15ms, rather than 20ms period, to emitted packets. (I have collected a
vast amount of gaming traffic data that I have not yet had time to
analyze)

** Videoconferencing

was not stuidied.

** Bittorrent model
   The torrent model is flawed in multiple ways. Notably it only tests
phase III of the typical torrent cycle, not the download or down/up
phases...

**** The download saturation problem

    Torrent can and WILL saturate a downlink and not respond to
congestion indicators until over 100ms of delay is observed. Most
clients do have a ratelimit set for download. It is often turned off
after-hours.

**** The upload saturation problem shown in the study

    Bittorrent clients have evolved to where, out of the box, there is
a very low rate limit set, typically in the range of 50-150KBytes/sec.
This makes bittorrent uploads a non-problem for most people.

Still, benchmarking each of these phases would be worthwhile.
Torrent can be fixed.

*** LEDBAT != Bittorrent

    LEDBAT as defined and uTP as deployed remain significantly
different. No implementation of bittorrent I've looked at (utorrent
and transmission) behaves anything like the LEDBAT modules I have.

*** TCP-LEDBAT kernel module is buggy
    In my RW tests this Linux congestion control module never gets out
of SSTHRESH and into congestion avoidance. The behavior LOOKs like
bittorrent (look! It's scavaging), when in reality, it's merely stuck
at a low rate.

   However, it is possible this module works correctly under older
linuxes under ns2.

   The RW behavior of bittorrent under RED and SFQ was explored in
[[YIXI2012]] and under those two AQM systems, formerly scavaging flows
are reprioritized to have roughly the same weight as non-scavaging
flows until 100ms of delay is incurred.

   The simulated LEDBAT results do show that extraordinarily high
numbers of persistent high rate flows interact badly with fq_codel, at
this 20/4 bandwidth setting. Using another TCP that is not buggy will
probably be even worse. However, bittorrent is a very special case.
Dozens of full rate flows are extremely rare in the real world.

(RW benchmarks show even fairly large numbers of flows (50+) show
fq_codel still doing quite well)

** IP address Hashing

   ns2 does not have support for a full 5 tuple including the protocol
(e.g. TCP, UDP, ARP, etc). This makes hashing multiple protocols
together problematic and I'm unsure if this was compensated for
correctly in all the models.

** Fq_codel and PIE configuration with 10ms target

   While I buy into (kathie doesn't) the idea that the delay target
variable needs to be greater than the cable MAC media acquisition time
(in this model), it does not need to be set to 10ms. 6.X should be
sufficient to avoid MAC acquisition artifacts. There is a quadratic
response time to delay in TCP, so the doubling of the default target
to aim for of 5ms to 10ms results in much fuller queues. (Lest I be
mis-interpreted, the quadratic behavior is end to end and I honestly
don't know what differences we'd see between a 6.5ms target and a 10ms
target)

PIE has similar constraints, (but apparently ran fine on the cable MAC)

so a target delay of 6.X or 7 being tried
for both would be interesting.

**I note that (2014) pie has now grown a target of 20ms in the Linux
implementation, and a closer tie with the htb rate shaper in the
as-yet-unpublished cablelabs model. **

At very, very low bandwidths (<4mbit) we have found it desireable to
increase the  fq_codel target to account for a single MTU at that
bandwidth.

The fq_codel interval estimation window in the cablelabs
testing was set to 150ms, rather than the default 100ms. I note that
most experimental variants of codel fiddle with the "drop resumption"
portion of the algorithm which is very sensitive to the interval. It's
an area of research...

The estimation window in pie has grown to 10k bytes.

* rate limiters have a cost.

At higher rates, more fq_codel queues are of use, and the RW tests
point to the rate limiter being the principal CPU hog and source of
problems, the drop/mark/scheduler algorithm hardly enter into it.

The PIE codel and fq_codel algorithms barely show upon a trace....

** Back to back packet drop

Gaming and VOIP traffic tolerate single, random packet drop with
aplomb. It's bursty packet loss and sudden delays that cause audible
artifacts, and gaming misbehavior. You can hear 3 packets lost in a
row, (which is no different from a sudden delay of 60ms on a stream)
So in addition to the packet loss figure, a good measurement would be
a bursty packet loss, and a bursty delay graph.

Hopefully in the aqm cases it follows a random distribution, but...

(Similarly many tcps respond to bursty packet loss in a drastic
fashion, but do not react too much to bursty delay)

Another problem with VOIP is "creeping delay", where a voip queue
builds and builds and then delivers or drops a full boatload of
packets to catch up. I have experienced this on multiple wifi based
voip sessions where I ended up with seconds of delay on the line over
time...

** ns2 issues
    Many of the problems in this test series are due to using an obsolete
and undermaintained network model system (ns2). Alternatives exist
that are better (ns3, minitel, etc) in multiple aspects. While ns2 can
certainly be improved the cost/benefit ratio of using ns3 seems
better. (There is no harm in using other technologies as a cross-check
too.)

*** Ns2 doesn't support many modern networking features, like ECN.
*** ns2 doesn't have TOS values (so it can't do diffserv)
*** ns2 doesn't have port numbers
   fq_codel uses a 5 tuple of port, protocol, src and destination ips

* Statistical notes

  Analyzing network queue, and network traffic behavior does not lend
itself to many means of statistical analysis. In particular, throwing
out the upper or lower percentiles of most results is a bad practice -
with real time systems, it's the outliers that are interesting and
important.

  This paper uses CDF plots throughout, which is a fine way to measure
the full range of results, however the use of a log scale is difficult
on the unpracticed eye.

  In it's summary form, the paper uses a method of averaging together
the results of each separate subtest, and the weighting is unclear.
This in particular gives the bittorrent result a lot more weight than
one
would expect.

NOTE: RW = real world

* Overall recommendations...
** Continue developing better models!
** Find a cable vendor willing to do the exploratory work
    It took me 24 hours to port the code from ns2 to Linux.
Analysis of several other operating systems indicate that a week in
conjunction with a local expert would be enough to make this code work
on most other network OSes.
** Better models of media acquisition,  (cable, wifi, lte),
aggregation, or scheduling characteristics (wifi, lte, token ring)
dissimilar to ethernet.

* Nits

** fq_pie

Given the probalistic dropping technique in PIE, it too can benefit
from flow queuing, fair queueing or weighted fair queuing tecniques
inserted before it in the chain, with only one PIE queue needed to do
so.

(This is unlike fq_codel, which due to the isochronous nature of the
timestamping has to have one codel queue per fq in order to work,
presently)

The prospect of fq + pie seems quite promising.

* Packet Trains

* Next steps in the RW cerowrt tests

** pie and fq_pie improvements
   Cerowrt gained pie support back in august or so and has tracked
revisions 1-4. Codel has had some tiny tweaks as well.

** Several new codel and fq_codel models

   based on the cablelabs testing, it is evident that number of queues
could be dependent on the available bandwidth, which can be made
automatic, there are also a few tweaks that can be made to pie and
codel (ecn handling, notably)

   Also it seems possible to improve fq_codel's behavior by adopting
slightly different strategies for nearly empty queues, under load.
Like (original) PIE, favoring dropping big packets more, particularly
when a queue is at 1 and the system is experiencing large delays,
seems plausible to handle the bittorrent problem while not affecting
most other traffic types.

** classification

  Under test are several shapers that use limited amounts of
classification. In the RW, in the campground test site, Over 52% of
all packets are marked CS1 on egress. Applications are obviously
trying to deprioritize themselves, it seems logical to try to support
that.

* Side notes

It should be clear that these AQMs and packet schedulers can apply not
only to edge networks, but anywhere there is a fast to slow transition
on a network. This includes within a machine itself!

It may well be easier to apply these technogies to load balancers,
high end Linux based routers, and vm root hardware, piecemeal, far
faster than they can be deployed en-mass across the customer edge
network. Linux based servers can also benefit, today. Deploying these
AQM technologies on any scale will help gain needed operational
experience with them before they have to be burned into hard to update
edge customer network hardware.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 10 Feb 2014 07:13:59 +0100 (CET)

> background traffic, has not been explored.

Just as a data point, currently Comhem in Sweden offers a DOCSIS based 
cable-Internet service of up to 500/50 (500 down, 50 up).

What would be of great interest would be to test this kind of service in 
several modes, including having 10 of those on the same CMTS port and 
congesting both up and downstream of the cable system itself (because 
there isn't 500 megabit/s total upstream capacity to support 10 
subscribers all trying to use 50 megabit/s per user. Same thing 
downstream. Would be very interesting to see how these users interacted 
with each other combined with LEDBAT and having TCP flows that didn't 
incur too much latency for the individual user, whilst assuring that each 
subscribed line got its fair share of bandwidth from the system.

> Still, benchmarking each of these phases would be worthwhile.

Well, users still would like to set the upload rate to something close to 
what they have purchased, and still have it yield to other traffic that 
happens to need the upstream bw, so further study would be great.

> Another problem with VOIP is "creeping delay", where a voip queue builds 
> and builds and then delivers or drops a full boatload of packets to 
> catch up. I have experienced this on multiple wifi based voip sessions 
> where I ended up with seconds of delay on the line over time...

This was a big problem 5+ years ago, where I think the PDV (jitter) buffer 
just grew, and never shrunk again even if network conditions improved. I 
haven't experienced this lately, so for instance Skype must have fixed 
this.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Greg White <[email-masked]>
Date: Mon, 10 Feb 2014 19:47:50 +0000

Dave, 

Thanks for posting the helpful critique, and I hope that it helps shape
the eval-guidelines draft.

I think simulations are always a "work-in-progress" of sorts - there is
always something you can do better next time.  By their nature,
simulations are simplifications of reality, and one of the challenges is
to spend time developing fidelity where it is needed and not where it is
not (we've tried to be very conscious of that, but it is always good to
get another set of eyes on the problem), another is to try to avoid
letting the combinatorics get out of hand.  And, you are right that
simulations can have bugs (but as you've noted, so does the RW).

Some of the items in your list we've already addressed since publishing
that white paper last spring, some we may address in future work, and some
we may need to agree to disagree on the importance of.  Also, we are in
the process of getting a bunch of our tools made public (in ns-2.36) so
that the research community can potentially tackle some areas that we're
not able to get to.

Once we have real systems to experiment with, we'll be doing some RW
experiments too (both to validate implementations, and to validate the
approach).  I hope we can leverage some of the work you've been doing in
your testbed (e.g. RRUL) when we get to that point.

I responded to a couple of Mikael's points below.

-Greg

>subscribed line got its fair share of bandwidth from the system.

In our recent simulation work (unpublished), we've been using the
following configurations: 20/5 (the original cfg), 100/20, 1000/200.  One
finding for us was that the fq benefits diminished considerably as the
data rates went up.

Also, we model shared link congestion by modulating the capacity available
to our tested user. This was described in our April 2013 paper. Since one
user's queuing latency doesn't impact another user, the interaction
between users is largely limited to how much capacity is available to our
tested user at any point in time.  Our model was, admittedly, a simple
one, but I'm not aware of any better models at the moment.  We did collect
data on some real systems to validate the model, but had difficulty
finding a real system that exhibited significant congestion (I'm sure
several hands will go up to volunteer their own broadband connection as
being the poster child for this ;-).

>>builds 
>_______________________________________________
>aqm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/aqm

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 11 Feb 2014 09:43:45 +0100 (CET)

On Mon, 10 Feb 2014, Greg White wrote:

> Also, we model shared link congestion by modulating the capacity 
> available to our tested user. This was described in our April 2013 
> paper. Since one user's queuing latency doesn't impact another user, the 
> interaction between users is largely limited to how much capacity is 
> available to our tested user at any point in time.  Our model was,

Could you please elaborate on this. If we're 10 users all using our 50 
megs up, now 5 of them stop sending, since now the upstream system 
capacity per actually sending user has doubled, how can the queueing delay 
not be affected? What happens when those 5 that now went silent started 
sending again?

What I'm after is to measure the system as a whole, the CPE queueing 
algorithm together with the cable "system" interaction and how this would 
interact with TCP streams.

The same use-case would be valid for PON.

---

From: Greg White <[email-masked]>
Date: Tue, 11 Feb 2014 16:17:54 +0000

>not be affected? What happens when those 5 that now went silent started

It is just as you think it is.  When 10 users are placing equal,
saturating load on the link, they will each get (on average) 10% of the
link rate.  When there are 5 users, they will each get (on average) 20% of
the link rate.  This is enforced by the CMTS. So when the active users
undergoes a step change from 10 to 5, each of the 5 remaining active users
will see a doubling of their link rate. The reverse will happen when the
active users returns to 10.  Step changes in link rate will result in step
changes in queuing latency.

This we model.

What I was getting at is that the above is true for user #1 regardless of
what applications and what AQM user #2 is using.  So we don't need to
simulate user #2 (it is sufficient to model him) if we're interested in
user #1's performance.

-Greg

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 12 Feb 2014 09:15:21 +0100 (CET)

On Tue, 11 Feb 2014, Greg White wrote:

> What I was getting at is that the above is true for user #1 regardless 
> of what applications and what AQM user #2 is using.  So we don't need to 

Are you sure? If AQM user #2 has 500 ms worth of packets in his upstream 
buffer (FIFO) vs 10 ms worth of packets with FQ_CODEL, wouldn't that make 
him adapt faster to increased upstream capacity? If nothing else, he has 
plenty of packets buffered that can be emptied out, whereas with FQ_CODEL 
you do not have this huge standing buffer with packets ready to send?

---

From: Greg White <[email-masked]>
Date: Wed, 12 Feb 2014 19:35:11 +0000

What you've said is true, and we will see the first order effect of that
in our model.  At that rate increase transition, user 1 will have ~20% of
the link rate *available* to him.  If he cannot capitalize on it because
he has a small queue, then that capacity can be used by user 2 (who has a
large queue). But, still, this effect on user 1 is only caused by user 1
himself in this case, user 2 has no impact.  Comparatively speaking, it is
true that user 2 would temporarily see a larger amount of free capacity on
the link (> 20% available). This we don't try to model.  I may be wrong,
but I think this is a small nuance of a synthetic scenario.

Certainly, more complex models can be built to look into nuances of
specific situations, and perhaps this is an area for other researchers to
explore. But, at that point, I think you have to move away from simple
synthetic usage models like we've been discussing in this thread (5 users
simultaneously go immediately dark or simultaneously immediately saturate
their link).  TCP sessions (and other traffic loads) come and go across
the user base asynchronously, so the available capacity fluctuates over
time in a more random manner than we've used in this example. Similarly
within each modem the traffic loads and TCP sessions will come and go and
be in various states of congestion avoidance (hence differing amounts of
queue), so the availability of a 500ms or 10ms of queued traffic isn't
guaranteed in the FIFO vs FQ_CODEL cases.

-Greg

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 13 Feb 2014 06:37:16 +0100 (CET)

On Wed, 12 Feb 2014, Greg White wrote:

> specific situations, and perhaps this is an area for other researchers 
> to explore. But, at that point, I think you have to move away from 
> simple synthetic usage models like we've been discussing in this thread 
> (5 users simultaneously go immediately dark or simultaneously 
> immediately saturate their link).  TCP sessions (and other traffic 
> loads) come and go across the user base asynchronously, so the available 
> capacity fluctuates over time in a more random manner than we've used in 
> this example. Similarly within each modem the traffic loads and TCP 
> sessions will come and go and be in various states of congestion 
> avoidance (hence differing amounts of queue), so the availability of a 
> 500ms or 10ms of queued traffic isn't guaranteed in the FIFO vs FQ_CODEL 
> cases.

How quickly does the CMTS react to customer demand, is this done 
continously (basically down to millisecond)? I don't know how cable works, 
I just know how LTE works for instance, where it actually takes a little 
while to adapt to changes in customer rate requirements. If cable does 
this is more or less round robin fairness between users then I agree with 
you that the current model is sufficient.

---

From: Greg White <[email-masked]>
Date: Thu, 13 Feb 2014 15:43:11 +0000

>you that the current model is sufficient.

It is done continuously.  There is about a 6ms lag between the CM sending
a request for a transmission opportunity (e.g. "I currently have 3742
bytes in queue ready to transmit") and the actual transmission opportunity
occurring (e.g. "Ok you can transmit 1768 bytes starting at time X"), but
essentially* every data transmission is handled via this request-grant
mechanism, i.e. the CMTS doesn't grant a tx opportunity for a CM unless it
is explicitly requested for.

*there are some special cases (not relevant here) that differ from this