Subject: [iccrg] AQM deployment status?

From: Naeem Khademi <[email-masked]>
Date: Tue, 24 Sep 2013 19:21:14 +0200

Hi

Sorry if I'm asking a redundant question here, but any citeable information
on AQM's deployment on the Internet? RED, ARED, BLUE, (FQ_)CoDel's
deployment either at the edge or core? I'm curious to know how much of it
has made it into real deployment.

Regards,
Naeem

---

From: "LOCHIN Emmanuel" <[email-masked]>
Date: Tue, 24 Sep 2013 19:24:46 +0200

Le Mardi 24 Septembre 2013 19:21 CEST, Naeem Khademi <[email-masked]> a écrit: 
 +1

Emmanuel

---

From: "Akhtar, Shahid (Shahid)" <[email-masked]>
Date: Wed, 25 Sep 2013 14:20:37 +0000

Hi Naeem,

Most routers/switches/access equipment support RFC 2309, which is a description of RED. It is not fully clear how often RED is actually turned on in today's networks, the below research looked at RED deployment on access networks and concluded that only a fraction of the access lines use RED (section 4.3.2).

http://research.microsoft.com/en-us/um/people/ssaroiu/publications/imc/2007/imc2007-dischinger.pdf

I am not aware of any more recent version of the same tests.

Best regards,

-Shahid.

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Naeem Khademi
Sent: Tuesday, September 24, 2013 12:21 PM
To: [email-masked]; bloat; [email-masked]
Subject: [aqm] AQM deployment status?

Hi

Sorry if I'm asking a redundant question here, but any citeable information on AQM's deployment on the Internet? RED, ARED, BLUE, (FQ_)CoDel's deployment either at the edge or core? I'm curious to know how much of it has made it into real deployment.

Regards,
Naeem

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 25 Sep 2013 14:31:21 +0000

Hi,

On Sep 25, 2013, at 15:20, "Akhtar, Shahid (Shahid)" <[email-masked]> wrote:
> Most routers/switches/access equipment support RFC 2309, which is a description of RED.

I've heard that statement from many different people. I wonder if it is actually true. Is there any hard data on this?

(From experience with the - small - selection of datacenter gear I have access to, almost none has usable RED/ECN support. Even in cases where the silicon is claimed to support it, the firmware doesn't allow configuration.)

Lars

---

From: "Akhtar, Shahid (Shahid)" <[email-masked]>
Date: Wed, 25 Sep 2013 18:51:21 +0000

Hi Steinar and Lars,

Please see below examples of support for RED/WRED from switches (from ALU and Cisco websites, search for RED or WRED in document): 

ALU 7450:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&ved=0CDMQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2F7450ESS_HS_MDA_ds.pdf&ei=pS5DUrIZiI70BLyYgYgK&usg=AFQjCNGS8uB-0Ele3TpDIRqbL4p1_kd2DQ&sig2=v-AgGJMpcZjrpjP2oZIHrA&bvm=bv.53077864,d.eWU

ALU Omniswitch:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CDkQFjAC&url=http%3A%2F%2Fwww3.alcatel-lucent.com%2Fwps%2FDocumentStreamerServlet%3FLMSG_CABINET%3DDocs_and_Resource_Ctr%26LMSG_CONTENT_FILE%3DData_Sheets%2Fent_OS6855_datasheet_en.pdf&ei=9i1DUsXlOo788QTU94HQAQ&usg=AFQjCNFSXSEpkFjSihK1a0pkSHhItU8sZQ&sig2=3fitEZLZZI7Dmf0H2T3QbQ&bvm=bv.53077864,d.eWU

Cisco Catalyst 6000/6500:  

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper09186a0080131086.html

Cisco Catalyst 4500:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps4324/white_paper_c11-539588.html

Best regards,

-Shahid.

-----Original Message-----
From: Steinar H. Gunderson [mailto:[email-masked]] 
Sent: Wednesday, September 25, 2013 9:35 AM
To: Eggert, Lars
Cc: Akhtar, Shahid (Shahid); bloat; [email-masked]; [email-masked]
Subject: Re: [Bloat] [iccrg] [aqm] AQM deployment status?

On Wed, Sep 25, 2013 at 02:31:21PM +0000, Eggert, Lars wrote:
> I've heard that statement from many different people. I wonder if it 
> is actually true. Is there any hard data on this?

I don't have hard data, but my soft data is that generally, devices branded as switches (including L3 switches) do not support it. Routers might.

/* Steinar */
--
Homepage: http://www.sesse.net/

---

From: Naeem Khademi <[email-masked]>
Date: Wed, 25 Sep 2013 21:19:39 +0200

Thanks Shahid

Although interesting to know that (W)RED has made it into some hardware (in
your RE to Lars' point), my question is more about "deployment" at the edge
or the core, whether it's being used or not?

Cheers,
Naeem

On Wed, Sep 25, 2013 at 8:51 PM, Akhtar, Shahid (Shahid) <
[email-masked]> wrote:

> Hi Steinar and Lars,
>
> Please see below examples of support for RED/WRED from switches (from ALU
> and Cisco websites, search for RED or WRED in document):
>
> ALU 7450:
>
>
> I don't have hard data, but my soft data is that generally, devices
> branded as switches (including L3 switches) do not support it. Routers
> might.
>
> /* Steinar */
> --
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 25 Sep 2013 21:24:56 +0200 (CEST)

> Please see below examples of support for RED/WRED from switches (from ALU and Cisco websites, search for RED or WRED in document):

I'd venture to claim that putting RED on a device with a few milliseconds 
worth of buffer depth is pretty much useless (for instance the Cisco 6704 
or 6724 linecards). So most datacenter equipment doesn't have this, or if 
they do, it's not that usable even if they had (who cares about drop 
probability when taildrop is being done at 4-5 ms buffer depth?).

For higher end platforms, for instance all cisco CPU based routers (for 
some value of "all") can be configured with RED, fair-queue or similar, 
but they come with FIFO as default. This has been the same way since at 
least the mid 90ties as far as I know, long way back to cisco 1600 device 
etc.

Higher end Cisco equipment such as ASR9k, 12000, CRS etc, all support 
WRED, and here it makes sense since they all have ~50ms worth of buffering 
or more. They also come with FIFO as default setting.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 26 Sep 2013 09:39:45 +0000

HI Mikael,

[chair-hat off]

> probability when taildrop is being done at 4-5 ms buffer depth?).

I'd state that people operating datacenters with request-response type data exchange via TCP do care a lot about the microscopic drop distribution. Typically, a tail-drop queue has the unfortunate property of losing the more critical packets of such an exchange, leading to excessive "transaction" (higher level protocol) delays due to lengthy TCP loss recovery.

Agreed, that is not the regular "internet" use case, but the same gear is being deployed there...

Richard Scheffenegger

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Mikael Abrahamsson
> Sent: Mittwoch, 25. September 2013 21:25
> To: [email-masked]
> Cc: [email-masked]; bloat
> Subject: Re: [iccrg] [aqm] [Bloat] AQM deployment status?
> 
> > Please see below examples of support for RED/WRED from switches (from
> ALU and Cisco websites, search for RED or WRED in document):
> 
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 27 Sep 2013 04:59:11 +0200 (CEST)

On Thu, 26 Sep 2013, Scheffenegger, Richard wrote:

> I'd state that people operating datacenters with request-response type 
> data exchange via TCP do care a lot about the microscopic drop 
> distribution. Typically, a tail-drop queue has the unfortunate property 
> of losing the more critical packets of such an exchange, leading to 
> excessive "transaction" (higher level protocol) delays due to lengthy 
> TCP loss recovery.

Do you know of simulations or similar that investigates this? I would like 
to understand why having RED on a 3ms buffer depth device makes a 
difference and how much difference it makes.

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 27 Sep 2013 09:06:16 +0000

On Sep 27, 2013, at 4:59, Mikael Abrahamsson <[email-masked]> wrote:
> Do you know of simulations or similar that investigates this? I would like to understand why having RED on a 3ms buffer depth device makes a difference and how much difference it makes.

Because saving 3ms on a path with microsecond latency is a big deal?

Lars

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 27 Sep 2013 13:53:21 +0000

Hi Mikael,

the papers around DCTCP have probably the information you are looking for (map/reduce type workloads, leading to incast issues - momentary filling of queues; loosing the final segments of a TCP session [1], which is likely with drop tail queue management policy, is well known to result in timely retransmission timeout-type recoveries. Tweaking OS time granularities down to recover in few milliseconds on paths that usually have a few dozen microseconds delay is often not good enough, but even that is not for cautious.

Apparently, this is seen important and valuable enough to motivate the unilateral deployment of DCTCP with Windows 8 server, even though the modification in the switches is then still necessary.

Richard Scheffenegger

[1] Google has presented the liklihood of packet loss per segment vs. burst size; Figure 3 in this paper http://plus.url.google.com/url?sa=z&n=1380288903097&url=http%3A%2F%2Fstatic.googleusercontent.com%2Fexternal_content%2Funtrusted_dlcp%2Fresearch.google.com%2Fen%2F%2Fpubs%2Farchive%2F41217.pdf&usg=lEiaJfELSWQoGNsDNsbQgbSO9RQ.

> From: Mikael Abrahamsson [mailto:[email-masked]]
> Sent: Freitag, 27. September 2013 04:59
> To: Scheffenegger, Richard
> Cc: [email-masked]; [email-masked]; bloat
> Subject: RE: [iccrg] [aqm] [Bloat] AQM deployment status?
> 
> > distribution. Typically, a tail-drop queue has the unfortunate
> > property of losing the more critical packets of such an exchange,
> > leading to excessive "transaction" (higher level protocol) delays due
> > to lengthy TCP loss recovery.
>

---

From: "Akhtar, Shahid (Shahid)" <[email-masked]>
Date: Fri, 27 Sep 2013 16:49:31 +0000

Naeem,

If you look into the document I initially sent

They show that 26% of DSL lines have RED actually deployed - and tuned on!

They did not detect any RED on the cable side.

-Shahid.
________________________________
From: Naeem Khademi [mailto:[email-masked]]
Sent: Wednesday, September 25, 2013 2:20 PM
To: Akhtar, Shahid (Shahid)
Cc: Steinar H. Gunderson; Eggert, Lars; [email-masked]; [email-masked]; bloat
Subject: Re: [aqm] [Bloat] [iccrg] AQM deployment status?

Thanks Shahid

Although interesting to know that (W)RED has made it into some hardware (in your RE to Lars' point), my question is more about "deployment" at the edge or the core, whether it's being used or not?

Cheers,
Naeem

On Wed, Sep 25, 2013 at 8:51 PM, Akhtar, Shahid (Shahid) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Steinar and Lars,

ALU 7450:

ALU Omniswitch:

Cisco Catalyst 4500:

Best regards,

-Shahid.

-----Original Message-----
From: Steinar H. Gunderson [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Wednesday, September 25, 2013 9:35 AM
To: Eggert, Lars
Cc: Akhtar, Shahid (Shahid); bloat; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Bloat] [iccrg] [aqm] AQM deployment status?

/* Steinar */
--
Homepage: http://www.sesse.net/
_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 28 Sep 2013 08:59:16 +0200 (CEST)

On Fri, 27 Sep 2013, Scheffenegger, Richard wrote:

> Hi Mikael,
>
> the papers around DCTCP have probably the information you are looking 
> for (map/reduce type workloads, leading to incast issues - momentary 
> filling of queues; loosing the final segments of a TCP session [1], 
> which is likely with drop tail queue management policy, is well known to 
> result in timely retransmission timeout-type recoveries. Tweaking OS 
> time granularities down to recover in few milliseconds on paths that 
> usually have a few dozen microseconds delay is often not good enough, 
> but even that is not for cautious.

Ok, so this makes sense within a data center with ECN, but I fail to see 
any substantial upside deploying this in an environment where RTT is in 
the tens of ms because TCP won't have time to react.

I seem to remember reading about enhancement to TCP where packets are 
actually sent out with spacing between packets instead of in line-rate 
bursts. If everybody started doing that then I would imagine there would 
be less tail-drop in large RTT environments with large TCP windows?

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sat, 28 Sep 2013 09:01:33 +0200 (CEST)

> Because saving 3ms on a path with microsecond latency is a big deal?

Yes, of course. I am not a datacenter guy so this is not a problem I face, 
because most of packets in my environment travel tens or hundreds of 
milliseconds.

So in datacenter one wants to start marking ECN on packets very soon into 
buffer depth, hoping sender will get feedback and throttle back the speed 
way before one gets taildrops?

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 28 Sep 2013 07:04:59 +0000

On Sep 28, 2013, at 9:01, Mikael Abrahamsson <[email-masked]>
 wrote:
> So in datacenter one wants to start marking ECN on packets very soon into buffer depth, hoping sender will get feedback and throttle back the speed way before one gets taildrops?

Yep. There have been a bunch of papers on datacenter TCP variants recently (look through the last 2-3 years of SIGCOMM papers, all online.)

Lars

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 29 Sep 2013 02:16:28 +0100

Mikael,

The shallow marking theshold is not the most significant feature of 
the AQM in DCTCP. More importantly, it does not delay congestion 
signals by averaging them over a period equivalent to a worst-case 
RTT (about 100ms), like all other AQMs do (RED, PIE, CoDel etc).

The shallow marking threshold certainly keeps standing queuing delay 
low. However, that's only under long-running constant conditions. 
During dynamics, not waiting a few hundred msec to respond to a 
change in the queue is what keeps the queuing delay predictably low. 
Dynamics are the norm, not constant conditions.

For instance, the AQM in DCTCP signals to a flow in slow-start as 
soon as it crosses the threshold. So a flow with 20ms RTT will get 
the signal in 20ms, and a flow with 3ms RTT will get the signal in 
3ms (e.g. to or from a CDN cache). Whereas all other AQMs will delay 
all signals for of the order of 100ms, even if the flow has a much 
shorter RTT (because all these other AQMs don't know the RTT of each 
flow, so they use a nominal worst-case RTT).

The source can use the signal as soon as it arrives (which it does at 
the end of slow-start), or it can smooth the signal itself (which it 
does once it's in congestion avoidance phase). But it can smooth the 
signal over its own RTT, rather than guessing a worst-case RTT.

CoDel taught us that the best line-rate auto-tuning an AQM can do is 
to use service time. DCTCP teaches us that the best RTT auto-tuning 
an AQM can do is /not/ to try to guess the RTT in the first place. 
Instead it is best to defer anything to do with RTT to the end-system.

We should learn from both lessons.

Bob

At 08:04 28/09/2013, Eggert, Lars wrote:
>Content-Language: en-US
>Content-Type: multipart/signed;
>         boundary="Apple-Mail=_19C08185-C4A7-46F5-925E-BA32C00EB99A";
>         protocol="application/pgp-signature"; micalg=pgp-sha1
>
> > So in datacenter one wants to start marking ECN on packets very 
> soon into buffer depth, hoping sender will get feedback and 
> throttle back the speed way before one gets taildrops?
>
>Yep. There have been a bunch of papers on datacenter TCP variants 
>recently (look through the last 2-3 years of SIGCOMM papers, all online.)
>
>Lars
>
>

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 29 Sep 2013 11:37:56 +0200 (CEST)

On Sun, 29 Sep 2013, Bob Briscoe wrote:

> low. However, that's only under long-running constant conditions. During 
> dynamics, not waiting a few hundred msec to respond to a change in the 
> queue is what keeps the queuing delay predictably low. Dynamics are the 
> norm, not constant conditions.

Well, my original question was in the context of a 3-5ms tail drop queue 
(such as are frequently available in lower end switches). My understanding 
from earlier experience is that TCP will severely saw-tooth under these 
conditions and only way I could see marking as being valuable was if the 
RTT was less than buffer depth (or at least very low).

There was a discussion earlier on bloat-l regarding what the impact of a 
10ms CoDel queuing scheme will have on 200ms RTT existing non-ECN TCP 
performance. Deep queues were invented to handle this specific use-case.

One way would absolutely be for TCP to not send a lot of packets 
back-to-back but instead pace them out at the rate that actually makes the 
TCP rate be exact in a millisecond resolution instead of as it is today, 
perhaps tens or hundreds of milliseconds. I believe some implementations 
already do this.