Subject: [aqm] AQM BoF - RTGAREA slides

From: Wesley Eddy <[email-masked]>
Date: Mon, 29 Jul 2013 08:42:35 -0400

As BoF chairs, we were asked to talk to the RTGAREA meeting
at this week's IETF about the AQM BoF and what we're doing.

The slides I'm planning to discuss there are:
http://www.ietf.org/proceedings/87/slides/slides-87-aqm-3.pptx

-- 
Wes Eddy
MTI Systems

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 29 Jul 2013 17:50:44 +0100

Wes,

#4 "Bufferbloat exists in routers, lower-layer switches, and other 
middleboxes "
...and hosts (altho not of such interest to RTG, this is also 
educating them on what we're doing)

Gap: They wlil want to know whether we are talking about big iron 
routers, big iron switches, access nodes, home hubs, etc. So we need 
to say the problem is mainly in access and edge equipment but you see 
it everywhere. (They will want to know whether we are treading on 
what they perceive as their territory.)

#5 "separation into multiple queues" implies particular solutions. I 
suggest you just say "traffic isolation".

#9 The statements about RED are over-negative. It's not that it isn't 
an effective solution to bufferbloat if correctly configured (as all 
the tests so far have shown). It's that it's hard to configure.
The charter reads to me is as much about advice on configuring 
existing stuff like RED, as it is about designing new stuff. We don't 
want to give the message "the only solution is to rip out everything 
you've already bought and buy this new stuff".

HTH

Bob

At 13:42 29/07/2013, Wesley Eddy wrote:
>_______________________________________________
>aqm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/aqm

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Spencer Dawkins <[email-masked]>
Date: Mon, 29 Jul 2013 12:22:25 -0500

On 7/29/2013 11:50 AM, Bob Briscoe wrote:
> it everywhere. (They will want to know whether we are treading on what 
> they perceive as their territory.)

Wes,

I liked the slides you've prepared. As a data point on #4, I had a very 
similar conversation with a co-worker for breakfast this morning. I'm 
pretty sure you'll get this as a question if you don't put it on your 
slides.

Thanks,

Spencer

---

From: Gorry <[email-masked]>
Date: Mon, 29 Jul 2013 19:36:54 +0200

+1 on RED - I can't see why would be creating an impression the BOF is about obsoleting RED, I hope it is actually about deploying AQM and understanding how to make this work better and provide good tools - I had the feeling that the  text I read was a little negative,

Gorry

On 29 Jul 2013, at 12:50 PM, Bob Briscoe <[email-masked]> wrote:

> Wes,
> 
> #4 "Bufferbloat exists in routers, lower-layer switches, and other middleboxes "
> ...and hosts (altho not of such interest to RTG, this is also educating them on what we're doing)
> 
> Gap: They wlil want to know whether we are talking about big iron routers, big iron switches, access nodes, home hubs, etc. So we need to say the problem is mainly in access and edge equipment but you see it everywhere. (They will want to know whether we are treading on what they perceive as their territory.)
> 
> #5 "separation into multiple queues" implies particular solutions. I suggest you just say "traffic isolation".
> 
> #9 The statements about RED are over-negative. It's not that it isn't an effective solution to bufferbloat if correctly configured (as all the tests so far have shown). It's that it's hard to configure.
> The charter reads to me is as much about advice on configuring existing stuff like RED, as it is about designing new stuff. We don't want to give the message "the only solution is to rip out everything you've already bought and buy this new stuff".
> 
> HTH
> 
> 
> Bob
> 
> 
>

---

From: Dave Taht <[email-masked]>
Date: Mon, 29 Jul 2013 11:41:50 -0700

On Mon, Jul 29, 2013 at 9:50 AM, Bob Briscoe <[email-masked]> wrote:
> Gap: They wlil want to know whether we are talking about big iron routers,
> big iron switches, access nodes, home hubs, etc. So we need to say the
> problem is mainly in access and edge equipment but you see it everywhere.
> (They will want to know whether we are treading on what they perceive as
> their territory.)
>
> #9 The statements about RED are over-negative. It's not that it isn't an
> effective solution to bufferbloat if correctly configured (as all the tests
> so far have shown). It's that it's hard to configure.

It's not just that it's hard to configure. It's *impossible* to
configure for links of wildly varying bandwidth such as wifi and other
forms of wireless.

RED was designed for fixed bandwidth p2p links, like what you got in
the good ole days with a T1 line.

Other forms of shared links (cable for example) also have varying link
rates that are hard to respond to in RED.

That said, I've got no problem with this wg generating clear
guidelines on when and where and HOW to configure RED and other
smarter queue management algorithms.

> The charter reads to me is as much about advice on configuring existing
> stuff like RED, as it is about designing new stuff. We don't want to give
> the message "the only solution is to rip out everything you've already
> bought and buy this new stuff".

Seeing RED deployed more where it could matter (like interprovider
interconnects and other fixed rate lines) would be a goodness.

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: David Collier-Brown <[email-masked]>
Date: Mon, 29 Jul 2013 15:39:59 -0400

> they perceive as their territory.)

It will occur wherever a slower-bandwidth link connects to a
higher-bandwidth one, and will be most visible where the differences are
substantial. A typical example might be at a home router, or at the
interface between a fast machine-room backbone and a slow wide-area
connection.

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain
(416) 223-8968

---

From: Michael Welzl <[email-masked]>
Date: Tue, 30 Jul 2013 00:15:04 +0200

Hi,

I like the slides too, but I think that slide #5 should contain:
- counteract synchronization

This is in fact a goal of AQM that I had always considered to be one of the key motivators behind RED (without that goal, it could probably have just been ED instead  :-)  ). Somewhat strangely, this goal hasn't been mentioned much recently.

Is it just a simulation artifact? I don't think so. Yes we see it a lot in simulations:
http://www.icir.org/floyd/papers/phase.pdf
but it also appears in reality:
http://caia.swin.edu.au/pubs/ATNAC06/Jaym.pdf
and it would surprise me greatly if it wasn't a much more significant problem in more controlled environments, in particular data centers.

Cheers,
Michael

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 29 Jul 2013 17:32:10 -0500

The first time I heard Van talk about ECN (and, I'm sure, something like
RED), he said randomness was really important ...

Spencer

On Tuesday, July 30, 2013, Michael Welzl wrote:

> This is in fact a goal of AQM that I had always considered to be one of
> the key motivators behind RED (without that goal, it could probably have
> just been ED instead  :-)  ). Somewhat strangely, this goal hasn't been
> mentioned much recently.
>
> and it would surprise me greatly if it wasn't a much more significant
> problem in more controlled environments, in particular data centers.
>
> Cheers,
> Michael
>
>
> On Jul 29, 2013, at 2:42 PM, Wesley Eddy <[email-masked]<javascript:;>>
> wrote:
>
> > [email-masked] <javascript:;>
> [email-masked] <javascript:;>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 30 Jul 2013 05:52:02 +0000

> Seeing RED deployed more where it could matter (like 
> interprovider interconnects and other fixed rate lines) would 
> be a goodness.

I am just curious: Is there any data that shows *any* benefit of RED (or whatever other AQM flavor) on a 10G or 100G link?

Michael

---

From: Stephen Hemminger <[email-masked]>
Date: Tue, 30 Jul 2013 10:54:10 -0700

On Tue, 30 Jul 2013 05:52:02 +0000

> I am just curious: Is there any data that shows *any* benefit of RED (or whatever other AQM flavor) on a 10G or 100G link?

The problem is when one link is slower than the data sources, so yes.
When doing 100G to 10G bridging or routing, the same issues occur
but with different timing.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 30 Jul 2013 21:42:04 +0000

> interprovider 
> > I am just curious: Is there any data that shows *any* 
> benefit of RED (or whatever other AQM flavor) on a 10G or 100G link?
> 
> When doing 100G to 10G bridging or routing, the same issues 
> occur but with different timing.

Hand-waving is simple. I was wondering about real data showing a problem at a 10G or 100G link (and, please, no ns-2, we are in the I*E*TF).

Michael

---

From: Dave Taht <[email-masked]>
Date: Tue, 30 Jul 2013 15:12:50 -0700

> Hand-waving is simple. I was wondering about real data showing a problem at a 10G or 100G link (and, please, no ns-2, we are in the I*E*TF).

I don't get much chance to play at speeds higher than 100Mbits. If
someone wants to donate a few 10GigE devices to the yurtlab and
hardware sufficient to drive them to saturation, I won't mind....

 I did gather a TON of packet headers from "the gathering" gaming
event earlier this year (if anyone wants a look let me know offline).

On the first day they experienced a problem when they attempted to
have 3 10GigE fiber ports in action. Slides 20 and 21 here show the
sort of latencies that were being experienced by that line before and
after it was fixed... They were pumping 10-16Gbit through their
connection 24/7...

ftp://ftp.gathering.org/TG/2013/Tech/TG13-Dave-Taht-preso.pdf

I infer from these plots that there is at least 50ms worth of
buffering hiding in the cisco line card at or near the fiber error
correction layer. It didn't appear that RED could have been enabled,
or it would have helped, but does suggest there be big buffers down
there...

They ended up just disabling the 3rd line to get to the results shown
on slide 21. Steinar or Marius (cc'd) might be able to add more to
this convo.

A lot of good results came out of that gathering (strong
recommendation: more researchers should go next year!) - notably some
big performance improvements to htb and the tcp stack - and a better
awareness of how to have 5000 wifi devices sharing the same space.

http://technet.gathering.org/2013/03/30/wireless-part-2/

-- 
Dave Täht

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 30 Jul 2013 22:39:39 +0000

> >> When doing 100G to 10G bridging or routing, the same 
> issues occur but 
> >> with different timing.
> >
> > Hand-waving is simple. I was wondering about real data 
> showing a problem at a 10G or 100G link (and, please, no 
> ns-2, we are in the I*E*TF).
> >
> > Michael
> 
> I don't get much chance to play at speeds higher than 
> 100Mbits. If someone wants to donate a few 10GigE devices to 
> the yurtlab and hardware sufficient to drive them to 
> saturation, I won't mind....
> 
>  I did gather a TON of packet headers from "the gathering" 
> gaming event earlier this year (if anyone wants a look let me 
> know offline).
> 
> On the first day they experienced a problem when they 
> attempted to have 3 10GigE fiber ports in action. Slides 20 
> and 21 here show the sort of latencies that were being 
> experienced by that line before and after it was fixed... 
> They were pumping 10-16Gbit through their connection 24/7...
> 

Thanks for sharing the data. But I am not sure if I can parse slide 20 and 21. Why is y-Axis of the order of tens of Mbit/s on a multiple 10G link?

The original phrase I refer to was that "RED could matter at interprovider connects", and I am looking for real data for that scenario.

Michael

---

From: Dave Taht <[email-masked]>
Date: Tue, 30 Jul 2013 16:01:56 -0700

> Thanks for sharing the data. But I am not sure if I can parse slide 20 and 21. Why is y-Axis of the order of tens of Mbit/s on a multiple 10G link?

'cause I was sharing it with 5000+ other users! is part of the
explanation, the long tcp ramp up period and normal packet loss from
going across the atlantic from norway to the west coast of the USA
accounts for some of the rest, bugs for some more, and it's noisy data
all along the path...

The only relevant bit in those slides to this discussion was the
simultaneous latency measurement in the bottommost plot  (Aside from
the interesting period to the tcp sawtooths over that very long
link...) which showed some dark buffering somewhere deep in the line
card.

I've talked about the structure of the rrul test before elsewhere and
on multiple presentations, but it's normal purpose is to load up a
link to the best ability of the host and server, and measure latency
simultaneously.

Here, by accident, it showed that the lines had a problem long before
it was diagnosed and fixed...

> The original phrase I refer to was that "RED could matter at interprovider connects", and I am looking for real data for that scenario.

I'd like some too! There are no doubt some members here on nanog,
which might be a good place to get some.

I ran this test multiple times later over the course of the event, and
at no point did the line saturate 20Gbit so latencies stayed flat. So
in order to see what happens at 10Gbit, some live testing with hosts
sufficient to drive it and routers or switches capable of handling it,
would be required.

Ivy bridge hardware can run at these rates fairly easily.

-- 
Dave Täht

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 31 Jul 2013 06:29:41 +0100

Michael,

The tech report behind the later papers on Data Center TCP (DCTCP) 
contains data from 19,000 measurements on what the delay was before 
and after having deployed DCTCP throughout their production Bing data 
centres on 1Gbps and 10Gbps links.
<http://research.microsoft.com/pubs/121386/dctcp-public.pdf>

Before (no AQM, 14ms of buffer):
* 90% of the time RTT <1ms
* 10% of the time RTT 1-14ms

After:
* with DCTCP: 99% of the time RTT < 0.08ms
* with RED: 99% of the time RTT < 0.2ms

Around Fig 16 is where they compare DCTCP & RED at 10Gb/s. They make 
the point that it was hard to config RED, because it gave only 78% 
utilisation if set at the recommended values for this scenario. But 
they tuned it up to give 92% before doing the above comparison of delay.

I know this isn't inter-provider interconnects, but it fits the 
figures you asked for. And it confirms my point that, if you've 
already bought your switch, you can get RED to work to reduce delay 
without having to throw it away and buy another one.

Bob

At 06:52 30/07/2013, Scharf, Michael (Michael) wrote:
>I am just curious: Is there any data that shows *any* benefit of RED 
>(or whatever other AQM flavor) on a 10G or 100G link?
>
>Michael

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 31 Jul 2013 06:32:08 +0100

Dave,

If you're implementing or turning on an AQM, you 
know whether you have a variable rate link or not.

My point was that we need to give useful advice 
to different interest groups, not just push new 
solutions without saying old ones still have sound applicability areas.

Bob

At 19:41 29/07/2013, Dave Taht wrote:
> > ...and hosts (altho not of such interest to 
> RTG, this is also educating them
> > on what we're doing)
> >
> > #5 "separation into multiple queues" implies 
> particular solutions. I suggest
> > you just say "traffic isolation".
> >
>Fixing bufferbloat with cerowrt: 
>http://www.teklibre.com/cerowrt/subscribe.html