Subject: [aqm] Minutes of the AQM WG session

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 23 Jul 2015 06:56:28 +0000

Hi,

Thanks to David Schinazi for serving as the notes taker;

I've uploaded the minutes https://www.ietf.org/proceedings/93/minutes/minutes-93-aqm

If you made a statement during the session, please review that the notes capture the essence of what you wanted to convey!

Also, one name is missing (and I'm unable to replay the meetecho recording currently) in the minutes, if you know who made that comment please inform us chairs.

Thanks,
  Richard (aqm chair)

---

From: Dave Taht <[email-masked]>
Date: Thu, 23 Jul 2015 13:23:57 +0200

1) in hard delay targets, I am credited with what matt mathis said
(not that I disagree).

2) In the dual queue thing I had noted that distinguishing between the
two queues based solely on the presence of ecn capability and then
using dctcp was non-sensical as plenty of other things like cubic
would also end up with ecn enabled.

3) and for the record, fq_codel as it exists today works reasonably
well when hammered with dctcp + ecn, cubic + ecn, and and any other
stuff without markings with the defaults. It may not give the desired
reduction in individual queue length - and cubic vs dctcp will do
badly against each other on a hash collision - but it is, at least,
"mostly safe" were some sysadmin finger foo things and push dctcp (or
some other non traditional cc) out on the world internet against
fq_codel'd ecn-enabled systems.

4) If you can't tell, it *really bothers me* when someone reports a
bug in dctcp - at ietf - rather than through proper channels -
particularly when it leads to evil behavior on loss, and yet I have no
patches submitted nor means to reproduce it.  PLEASE GET A PATCH OUT
to the netdev maintainers, it will be immediately put into the next
release of linux AND backported to the linux stable releases.

What I see is dctcp_clamp_alpha_on_loss not defaulting to on, which
based on the comments, should default to on. Is there somewhere else
this is busted?

static void dctcp_state(struct sock *sk, u8 new_state)

{

        if (dctcp_clamp_alpha_on_loss && new_state == TCP_CA_Loss) {

                struct dctcp *ca = inet_csk_ca(sk);

                /* If this extension is enabled, we clamp dctcp_alpha to

                 * max on packet loss; the motivation is that dctcp_alpha

                 * is an indicator to the extend of congestion and packet

                 * loss is an indicator of extreme congestion; setting

                 * this in practice turned out to be beneficial, and

                 * effectively assumes total congestion which reduces the

                 * window by half.

                 */

5) Missing from the preso was the actual dual queue length, just a
comparison (cool demo tho!) - you get what queue length using curvy
red?

6) cake, of course, gets to 100s of flows without hash collisions.

7) As I missed the tcp-prague discussion, is the proposal to reserve
ect(1) - the former nonce bit - for dctcp? or some other diffserv or
flowheader marking?

On Thu, Jul 23, 2015 at 8:56 AM, Scheffenegger, Richard <[email-masked]> wrote:
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

-- 
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 23 Jul 2015 11:49:20 +0000

Thank you Dave,

I'll update the minutes.

I believe the authors of DualAQM assume that all ECN traffic will have the less drastic reaction as a default (with the hope of deploying this before wide-spread use of legacy 3168 style ECN marking takes hold).

Also, thank you for your comments regarding FQ_codel and the reported linux issue (which, as you note, may be due to an incorrect config setting in the latest releases).

Regarding the identifier to differentiate legacy ECN (cc reaction on ECN per RTT = cc reaction to loss in RTT) and the differentiated response (proportional to the # of CEs per RTT) is an open question.

Perhaps we can start to gather the views of the group on this list already; There have been other uses for ECT(1) been proposed over time, and it has also been pointed out that repurposing a DiffServ Codepoint may not be the easierst to do from a procedural view.

Also, I would like to encourage members of the aqm wg to review the various versions of http://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn (the technical content did change in each version substancially). Disclaimer, I'm one of the authors of that draft.

Best regards,
  Richard

> -----Original Message-----
> From: Dave Taht [mailto:[email-masked]]
> Sent: Donnerstag, 23. Juli 2015 13:24
> To: Scheffenegger, Richard
> Cc: [email-masked]
> Subject: Re: [aqm] Minutes of the AQM WG session
> 
> 1) in hard delay targets, I am credited with what matt mathis said (not
> that I disagree).
> 
> 2) In the dual queue thing I had noted that distinguishing between the two
> queues based solely on the presence of ecn capability and then using dctcp
> was non-sensical as plenty of other things like cubic would also end up
> with ecn enabled.
> 
> 3) and for the record, fq_codel as it exists today works reasonably well
> when hammered with dctcp + ecn, cubic + ecn, and and any other stuff
> without markings with the defaults. It may not give the desired reduction
> in individual queue length - and cubic vs dctcp will do badly against each
> other on a hash collision - but it is, at least, "mostly safe" were some
> sysadmin finger foo things and push dctcp (or some other non traditional
> cc) out on the world internet against fq_codel'd ecn-enabled systems.
> 
> 4) If you can't tell, it *really bothers me* when someone reports a bug in
> dctcp - at ietf - rather than through proper channels - particularly when
> it leads to evil behavior on loss, and yet I have no patches submitted nor
> means to reproduce it.  PLEASE GET A PATCH OUT to the netdev maintainers,
> it will be immediately put into the next release of linux AND backported
> to the linux stable releases.
> 
> What I see is dctcp_clamp_alpha_on_loss not defaulting to on, which based
> on the comments, should default to on. Is there somewhere else this is
> busted?
> 
> comparison (cool demo tho!) - you get what queue length using curvy red?
> 
> > I've uploaded the minutes
> > https://www.ietf.org/proceedings/93/minutes/minutes-93-aqm
> >
> > If you made a statement during the session, please review that the notes
> capture the essence of what you wanted to convey!
> >
> > Also, one name is missing (and I'm unable to replay the meetecho
> recording currently) in the minutes, if you know who made that comment
> please inform us chairs.
> >
> > Thanks,
> >   Richard (aqm chair)
> >

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Tue, 28 Jul 2015 08:57:45 +0000

Hi Dave,

2) Yes, we used only ECN to distinguish between classic and L4S TCP. As Richard already mentioned, we hope indeed that the final deployment could be that simple too. The benefits of using ECN combined with 1/p congestion controllers is so big that we hope nobody will want anything less ;-). Also, (see CurvyRED and PI^2 in my backup slides) AQMs that think twice to drop "at-the-output" are much simpler to implement and understand (due to linear relation between p and number of flows), and are immediately compatible with 1/p congestion controllers (by skipping the square-at-the-output). It might be useful to think about a curvy or squared CoDel/Cake (note the intentionally missing "fq_") as well... (to improve dctcp support as mentioned in your topic 3). As far as I understand the new option in the *FQ_*codel is disabling codel completely and applies also a sojourn time based step function for ECN (which makes sense for the fq_ version, where every flow has its own queue & AQM, and RR-scheduling is used for fairness).

4) After I presented in the ICCRG, Daniel Borkmann suggested trying the dctcp_clamp_alpha_on_loss option, but it didn't work. The problem is probably that the alpha, once set, is moving quickly to a lower value before it is applied (at the end of the RTT). So it's a matter of keeping the state in another variable and reset that one, once the alpha=1 is set just before it is applied to reduce the window. Daniel Borkmann, Florian Westfall and Glenn Judd are aware of the dctcp_clamp_alpha_on_loss issue.

5) Actually Curvy RED is used to control the classic queue and couple it (non-curved) to the DCTCP traffic (for the throughput fairness). The size of the L4S queue is normally 0, only initial windows and other flow variations above the bottleneck links are shortly queued in the L4S queue. If no classic traffic is available, our experiments used 5 packets (= 1.6ms using the DCTCP step function marking threshold).

6) Would be good, as hashed FQ gives a surprisingly (for us at least) high number of collisions.

7) Nothing decided yet (see topic 2).

Regards,
Koen.

> From: aqm [mailto:[email-masked]] On Behalf Of Dave Taht
> Sent: donderdag 23 juli 2015 13:24
> > If you made a statement during the session, please review that the
> notes capture the essence of what you wanted to convey!
> >

---

From: Dave Taht <[email-masked]>
Date: Tue, 28 Jul 2015 11:37:25 +0200

> 2) Yes, we used only ECN to distinguish between classic and L4S TCP. As Richard already mentioned, we hope indeed that the final deployment could be that simple too. The benefits of using ECN combi…

I have not yet had time to push the ce_threshold idea into toke's
testbed. You can infer from the commit message where it is being tried
in production, and the commit message has the only public data so far
on how it works:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=80ba92fa1a92dea128283f69f55b02242e213650

and I believe it entered the linux mainline as of linux 4.1

As for thinking about much of this, curvyred, etc, more deeply I
prefer to have patches and data to play with first.

> 4) After I presented in the ICCRG, Daniel Borkmann suggested trying the dctcp_clamp_alpha_on_loss option, but it didn't work. The problem is probably that the alpha, once set, is moving quickly to…

Good, then I shall cease to worry about it. Still bothersome.

> 5) Actually Curvy RED is used to control the classic queue and couple it (non-curved) to the DCTCP traffic (for the throughput fairness). The size of the L4S queue is normally 0, only initial windo…

The demo showed the dctcp queue having < 5ms delay, and the "classic"
queue having 42 ms delay.

When switched to fq_codel it showed 20ms delay for both queues. This
struck me as overlarge. A common problem is that perhaps you had
ethernet offloads still on for GRO/GSO/TSO on the aqm hop? Codel's
(and pie and qfq!) original deployment was designed and tested with
offloads off, mostly, and we strenously encouraged experimenters to
turn it off - so much so that we did not do enough testing with it on.
Recently we were blindsided by the armada 385 chip (used by a whole
bunch of new high speed home routers) doing very aggressive GRO - on
everything - I had never seen it crack 24k before, it routinely
cracked 64k and that led to some headaches.

Codel had the maxpacket concept in it which was designed to operate
correctly at a single MTU, not 64K!

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a5d28090405038ca1f40c13f38d6d4285456efee

I do not know how to fix it in red or pie at the moment. pie's
estimator is byte based, but there may be other issues. So it would be
my hope that your demo would run better against fq_codel with offloads
off and/or this patch applied. ?

Cake has "peeling", which turns the up to 64k superpackets back into
appropriately-sized packets for the afforded workload. So has TBF, in
a more limited fashion. HTB does not, presently. Probably we will add
peeling features to htb at some point.

> 6) Would be good, as hashed FQ gives a surprisingly (for us at least) high number of collisions.

set associative hashing in cake *is* turning in pretty great and
interesting results, so far. and computationally cheap. set
associativity, of course, is a very common thing in conventional cpu
cache design, it is surprising I have not seen it in queue design as
well before now (anyone got any prior cites?)

I don't know what a "surprisingly" high number of collisions is, the
math is straightforward. Admittedly most of the workloads were tons of
short flows against a very few number of big flows, and only recently
(with cake) did we start trying to tackle torrent-like flows again.

> 7) Nothing decided yet (see topic 2).

l.

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 28 Jul 2015 12:44:45 +0300

> I don't know what a "surprisingly" high number of collisions is...

Surprising to people unfamiliar with the birthday paradox, perhaps. The set
associative hash produces results that are much more intuitively reasonable.

- Jonathan Morton

---

From: John Leslie <[email-masked]>
Date: Tue, 28 Jul 2015 10:50:36 -0400

Scheffenegger, Richard <[email-masked]> wrote:
> 
> Regarding the identifier to differentiate legacy ECN (cc reaction on
> ECN per RTT = cc reaction to loss in RTT) and the differentiated
> response (proportional to the # of CEs per RTT) is an open question.

   Wide open!

> Perhaps we can start to gather the views of the group on this list
> already; There have been other uses for ECT(1) been proposed over
> time, and it has also been pointed out that repurposing a DiffServ
> Codepoint may not be the easierst to do from a procedural view.

   Nor (IMHO) is a repurposed DiffServ Codepoint likely to work well.

   OTOH, I think the time is ripe to deprecate ECN Nonce; and reserve
ECT(1) for (currently) Experimental uses.

   We could pretty easily combine that with an IP Option to define the
particular experiment, and even record data about forwarding node action.
(Admittedly, this is more difficult in IPv6, so I suggest initially
considering IPv4; and then adding an IPv6 kludge, probably hop-by-hop.)

   IMHO, the only other uses for ECT(1) that have approached "traction"
relate to differentiation of forwarder action for ECN marking.

> Also, I would like to encourage members of the aqm wg to review the various
> versions of http://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn
> (the technical content did change in each version substantially).
> Disclaimer, I'm one of the authors of that draft.

   I believe I've reviewed them all; but is it truly reasonable to expect
all WG members to review more than the latest one?

   Fundamentally, the latest version calls for three counters, to be
masked to low-order bits reporting changes in the counters. There are
three possibilities on _where_ these bits will be reported. (I'd rather
not bike-shed the virtues of each yet...)

   draft-briscoe-aqm-dualq-coupled does not appear to have been posted
yet: I'll be only too happy to discuss it once it is...

   Fundamentally, it proposes to identify a separate low-latency queue
which will report via ECN a near-zero-latency condition (vs. a zero
congestion in this second queue), while the default queue operates by
a legacy AQM mecanism, probably signaled by packet drop.

   This strikes me as Experimental: there are a number of issues, such
as how this low-latency queue overflows and what happens when it does,
which need to be resolved before we could even consider Standards Track.
At first blush, this looks like a good match for Experimental use of
ECT(1).

--
John Leslie <[email-masked]>

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Fri, 31 Jul 2015 10:31:05 +0000

Surprisingly high for a an experiment of all combinations from 0 to 10 flows of each (=120 combinations of up to max 20 flows together). Indeed using birthday paradox, collision counting we calculated that on average each experiment (the 120 combinations) has 6.5 collisions. The one in the presentation had (by accident) the least (5), but others had more than 10.
We did also experiments with an extra 20MBps UDP flow, and in the first run, one of the collisions was in combination 1Cubic-1Dctcp. So with only 3 flows, the DCTCP and UDP flow got in the same queue, and DCTCP completely starved… For large amounts, statistics apply; for small amounts, Murphy ;-).

Koen.

From: aqm [mailto:[email-masked]] On Behalf Of Jonathan Morton
Sent: dinsdag 28 juli 2015 11:45
To: Dave Taht
Cc: Scheffenegger, Richard; De Schepper, Koen (Koen); [email-masked]
Subject: Re: [aqm] Minutes of the AQM WG session

> I don't know what a "surprisingly" high number of collisions is...

Surprising to people unfamiliar with the birthday paradox, perhaps. The set associative hash produces results that are much more intuitively reasonable.

- Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Fri, 31 Jul 2015 12:45:31 +0200

On Fri, Jul 31, 2015 at 12:31 PM, De Schepper, Koen (Koen)
<[email-masked]> wrote:
> Surprisingly high for a an experiment of all combinations from 0 to 10 flows
> of each (=120 combinations of up to max 20 flows together). Indeed using
> birthday paradox, collision counting we calculated that on average each
> experiment (the 120 combinations) has 6.5 collisions. The one in the
> presentation had (by accident) the least (5), but others had more than 10.
>
> We did also experiments with an extra 20MBps UDP flow, and in the first run,
> one of the collisions was in combination 1Cubic-1Dctcp. So with only 3
> flows, the DCTCP and UDP flow got in the same queue, and DCTCP completely
> starved… For large amounts, statistics apply; for small amounts, Murphy ;-).

Was this the linux version or the ns2 version? The ns2 version had no
direct support for accessing the packet headers and thus you had to
configure the flows carefully and simulate that. Secondly in a more
real world, additional entropy in the form of vastly different
destaddrs would be around.

If this was the linux version... interesting!

I had complained about the quality of the hash before, notably on the
lack of entropy in the proto field, and the xor in ipv6 (enough so
that an entire new hashing architecture has been arriving in newer
versions of linux with support for a much broader range of hash
targets, and support for mpls, and macaddrs and so on)

However, it is a shame to be unlucky, no matter the algorithm. 8 way
set associativity is working pretty well elsewhere.