Subject: Re: [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 Mar 2013 16:50:51 +0000

Folks. I posted the email I sent yesterday as a draft, for discussion. I welcome comments, and if substantive comments are made, suggested text.

On Mar 13, 2013, at 12:48 PM, <[email-masked]>
 wrote:

> 
> A new version of I-D, draft-baker-tsvwg-aqm-recommendation-00.txt
> has been successfully submitted by Fred Baker and posted to the
> IETF repository.
> 
> Filename:	 draft-baker-tsvwg-aqm-recommendation
> Revision:	 00
> Title:		 IETF Recommendations Regarding Active Queue Management
> Creation date:	 2013-03-13
> Group:		 Individual Submission
> Number of pages: 7
> URL:             http://www.ietf.org/internet-drafts/draft-baker-tsvwg-aqm-recommendation-00.txt
> Status:          http://datatracker.ietf.org/doc/draft-baker-tsvwg-aqm-recommendation
> Htmlized:        http://tools.ietf.org/html/draft-baker-tsvwg-aqm-recommendation-00
> 
> 
> Abstract:
>   Fifteen years after the IAB issued its recommendations regarding
>   congestion control in RFC 2309, a major issue in the community is the
>   issue that RFC addresses: Buffer bloat.  It may be time to update the
>   recommendation.
> 
> 
> 
> 
> The IETF Secretariat
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 13 Mar 2013 18:25:22 +0100 (CET)

On Wed, 13 Mar 2013, Fred Baker (fred) wrote:

> Folks. I posted the email I sent yesterday as a draft, for discussion. I 
> welcome comments, and if substantive comments are made, suggested text.

I voiced my opinion on the tsv-area meeting via jabber that AQM especially 
on low/medium bw links (up to 10-20 megabits/s) is needed and bufferbloat 
is a serious problem. As an employee of an operator, I would like to see 
documents in the line of RFC6204, ie basic requirements, for (in this 
case) AQM that I can use in procurement documents. Preferrably these would 
be implemented by kernel vendors (linux kernel and others) so the actual 
device vendor only needs to turn it on and perhaps set a few parameters 
(like communicate the link speed to the queue handler).

Coming back to the document at hand, reading 
draft-baker-tsvwg-aqm-recommendation-00.txt, I notice a few things. Am I 
misinterpreting text such as:

"Hence, network communication to the host regarding the moderation of
    its traffic flow SHOULD include both ECN and loss, with ECN being
    triggered earlier than loss."

... that it recommendeds that ECN traffic will be marked before non-ECN 
traffic is dropped? I believe desired behaviour is that ECN traffic is 
marked and non-ECN traffic is dropped at the exact same queue depth (or 
whatever mechanism causes packets to be dropped/marked).

Also, the documents seem to handle both host stack implementations and 
device queue management. Am I correct? Perhaps it would be preferable to 
have one document handling queueing (including the queue on the host 
towards its network interface (wifi/fixed/cellular etc), and another 
document talking about "TCP congestion control mechanisms"? Or this the 
document at hand trying to be an umbrella document for other documents?

Anyhow, I feel this (AQM/congestion) area of discussion is extremely 
important and it solves a real life problem that impacts a lot of people 
on the Internet as it works today. Personally I have routers with quite 
advanced AQM on my home connection (100/100 megabit/s) because I feel that 
FIFO even with WRED isn't good enough. However I understand that 
"fair-queue" uses a lot of CPU at these speeds, so it might not be 
feasable to do too advanced stuff at those speeds. However, at ADSL type 
accesses, for the uplink interface there is definitely a huge upside to 
have AQM.

When evaluating AQM the test scenarios should be quite advanced (I liked 
what I heard in ICCRG meeting yesterday) including VoIP, hundreds of TCP 
sessions using bittorrent, gaming (could be over TCP) etc. The ideal 
solution would be something that automatically allow at least per-user 
fairness in the upstream and something that perhaps sent ACKs and other 
smaller packets ahead of the queue of larger data packets.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: [email-masked]
Date: Wed, 13 Mar 2013 18:28:13 -0000

Fred,

Comments below:

Section 2, pt 2
"Deployed AQM SHOULD use ECN as well as loss, and set thresholds
 to mark traffic earlier than it is lost."
- This is not clear, I agree SHOULD use ECN for ECT traffic, of course.
- I'm not sure about threshold question that sets ECN drop before ECN loss
 - I like the idea for various reasons (I'm not expanding that here), but
this isn't what I understand as the current recommended TCP ECN reaction -
which reacts to CE in the same way as loss?

We need to be careful that we don't suggest not using ECN can gain advantage.

Section 2 pt 3
- Again I agree, but not sure we can say this as a BCP requirement? I
think we should think about how best to present this.

Section 2 pt 4
- Agree and we also now have tunnel technologies considering ECN support,
so also these?

Section 2 pt 5
- Nice, but not not possible - so TCP without ECN  *IS* going to cause
loss and delay if it shares the same congested queue. The idea of defining
guidance on what to expect here is also good, and maybe a significant step
to getting a better understanding.

I note that RFC 2309 does recommend RED but importantly it did not
motivate it in the way that now makes AQM an imperative. It also largely
pre-dated ECN and certainly the experience in ECN implementation.

Gorry

>> URL:
>> http://www.ietf.org/internet-drafts/draft-baker-tsvwg-aqm-recommendation-00.txt
>> Status:
>> http://datatracker.ietf.org/doc/draft-baker-tsvwg-aqm-recommendation
>> Htmlized:
>> http://tools.ietf.org/html/draft-baker-tsvwg-aqm-recommendation-00
>>
>>
>> Abstract:

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 Mar 2013 21:36:27 +0000

On Mar 13, 2013, at 1:25 PM, Mikael Abrahamsson <[email-masked]>
 wrote:

> I voiced my opinion on the tsv-area meeting via jabber that AQM especially on low/medium bw links (up to 10-20 megabits/s) is needed and bufferbloat is a serious problem. As an employee of an operator, I would like to see documents in the line of RFC6204, ie basic requirements, for (in this case) AQM that I can use in procurement documents. Preferrably these would be implemented by kernel vendors (linux kernel and others) so the actual device vendor only needs to turn it on and perhaps set a few parameters (like communicate the link speed to the queue handler).

> Coming back to the document at hand, reading draft-baker-tsvwg-aqm-recommendation-00.txt, I notice a few things. Am I misinterpreting text such as:
> 
> ... that it recommendeds that ECN traffic will be marked before non-ECN traffic is dropped? I believe desired behaviour is that ECN traffic is marked and non-ECN traffic is dropped at the exact same queue depth (or whatever mechanism causes packets to be dropped/marked).

Hmm. I probably didn't word that all that well.

My point is that at some level of congestion stress, we should mark ECN-capable traffic and drop non-ECN-capable traffic. At some higher level of stress, we should drop from all streams, ECN-capable or not. In general, I'd like to believe that ECN is effective in managing end-to-end throughput.

> Also, the documents seem to handle both host stack implementations and device queue management. Am I correct? Perhaps it would be preferable to have one document handling queueing (including the queue on the host towards its network interface (wifi/fixed/cellular etc), and another document talking about "TCP congestion control mechanisms"? Or this the document at hand trying to be an umbrella document for other documents?

One could argue for the separation. To my mind, though, they are firmly linked. Besides self-defense, the reason the network marks/drops traffic is to signal to TCP/SCTP. if that's the case, it seems like it's useful to say what that signal means to TCP/SCTP, and how we hope it might work.

> Anyhow, I feel this (AQM/congestion) area of discussion is extremely important and it solves a real life problem that impacts a lot of people on the Internet as it works today. Personally I have routers with quite advanced AQM on my home connection (100/100 megabit/s) because I feel that FIFO even with WRED isn't good enough. However I understand that "fair-queue" uses a lot of CPU at these speeds, so it might not be feasable to do too advanced stuff at those speeds. However, at ADSL type accesses, for the uplink interface there is definitely a huge upside to have AQM.

That depends on how FQ is implemented. The implementation I did in 1994 probably would have been high overhead. It has been minted by others, and I believe rewritten to be a calendar queue implementation. For that, I would not expect it was that much higher. I'll go take a look, though.

> When evaluating AQM the test scenarios should be quite advanced (I liked what I heard in ICCRG meeting yesterday) including VoIP, hundreds of TCP sessions using bittorrent, gaming (could be over TCP) etc. The ideal solution would be something that automatically allow at least per-user fairness in the upstream and something that perhaps sent ACKs and other smaller packets ahead of the queue of larger data packets.
> 
> --

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 Mar 2013 22:10:09 +0000

> - I'm not sure about threshold question that sets ECN drop before ECN loss

As I just said to Mikael, I don't think I worded that one well. I'm envisaging two thresholds, testing two stress levels. It a lower one, one marks ECN-capable traffic and drops non-ECN-capable traffic; at the higher one, one drops from all traffic. What "threshold" means in a given algorithm is algorithm-dependent, however.

> which reacts to CE in the same way as loss?

Well, 3168 says that TCP should respond in the same way it responds to loss. By that, what they mean is that it reduces cwnd, and if in slow-start, exit slow-start. The other thing it does to respond to loss is to retransmit a message. There is obviously no need to retransmit, but it should indeed play with cwnd in some way.

How it responds will be specific to the congestion control algorithm, though. With newreno, it might set cwnd to 1 or to the initial value, or perhaps halve it. With CUBIC, IIRC, it reduces it by a smaller fraction. With CalTech FAST, which is a delay-based model and tunes toward the knee, I could imagine that instead of literally reducing cwnd (if you're at the knee, why would you do that?) you might recalculate the current mean RTT. 

> We need to be careful that we don't suggest not using ECN can gain advantage.

Did I say that?

> think we should think about how best to present this.

2.3.  AQM algorithms deployed SHOULD NOT require operational tuning 

There are algorithms that don't require tuning. I'm suggesting we use one.

> - Agree and we also now have tunnel technologies considering ECN support,

2.4.  AQM algorithms deployed SHOULD be effective on all common
      Internet traffic 

We want to follow the rules for remarking traffic; I believe that's RFC 2983.

> to getting a better understanding.

     2.5.  TCP and SCTP congestion control algorithms SHOULD
           maximize their use of available bandwidth without
           incurring loss or undue round trip delay 

I'm not quite as pessimistic about the possibility here. We actually do have congestion control algorithms that tune to a point approximating the knee, and at least one that will co-exist with a loss-based model effectively (CAIA CDG). Note that I'm not dying to get to the knee, although that would be nice; the point is to not hit the cliff. Once the window advances to/beyond the knee, we are using the available bandwidth in its entirety; increasing cwnd after that point merely increases delay and the probability of loss, not throughput rate.

> pre-dated ECN and certainly the experience in ECN implementation.

Hmm. OK, then maybe we do need to re-do 2309. That was my first instinct when Wesley and I first chatted about this in mail, and I thought I would try simply building on it. I'll take a gander at that.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 14 Mar 2013 04:09:48 +0100 (CET)

On Wed, 13 Mar 2013, Fred Baker (fred) wrote:

> That depends on how FQ is implemented. The implementation I did in 1994 
> probably would have been high overhead. It has been minted by others, 
> and I believe rewritten to be a calendar queue implementation. For that, 
> I would not expect it was that much higher. I'll go take a look, though.

To be concrete, turning on fair-queue (and some other stuff to assure bw 
for certain tcp/port combinations) on a Cisco 7200 style device seems to 
give 2-3x CPU usage for the same traffic. This is with just 30-100 flows. 
This is in latest code available (15.2).

So since in my market we have gig speeds (we have low cost residential 
CPEs capable of moving ~600 megabits/s in a single tcp session), CPU 
impact of these queueing mechanisms is important.

And yes, I feel doing AQM at 100/100 speeds is still important, but here 
the limiting factor for speed might actually be the CPU of the CPE, so the 
queueing mechanism should preferrably be able to prioritize and gracefully 
handle the case where the links are not full but instead the CPE CPU is 
insufficient and that's what causing congestion/drops.

---

From: Andrew McGregor <[email-masked]>
Date: Wed, 13 Mar 2013 23:59:45 -0400

A Netgear WNDR3800 (680MHz MIPS) can handle fq_codel on two flat out
802.11n interfaces at the same time and still be very lightly loaded.  CPU
is not a problem because fq_codel is a tiny fraction of the usage of the
firewall, connection tracking and 802.11 AP code, all of which are
necessary (in IPv4 the NAT also uses more CPU than fq_codel).

On Wed, Mar 13, 2013 at 11:09 PM, Mikael Abrahamsson <[email-masked]>wrote:

>> probably would have been high overhead. It has been minted by others, and I
>> believe rewritten to be a calendar queue implementation. For that, I would
>> not expect it was that much higher. I'll go take a look, though.
>>
>
> CPEs capable of moving ~600 megabits/s in a single tcp session), CPU impact
> of these queueing mechanisms is important.
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 14 Mar 2013 06:07:37 +0100 (CET)

On Wed, 13 Mar 2013, Andrew McGregor wrote:

> 802.11n interfaces at the same time and still be very lightly loaded. 
> CPU is not a problem because fq_codel is a tiny fraction of the usage of 
> the firewall, connection tracking and 802.11 AP code, all of which are 

I'll have to look into what devices we're using and what CPUs they're 
using. The WNDR3800 seems to be a fairly high end device when I look at 
the price point.

However, the other part of your reply sounds hopeful. I read 
draft-nichols-tsvwg-codel-01 and I now realise that yes, it shouldn't be 
too cpu intensive as the hashing into bins sounds efficient on a 
per-packet level.

Would having multiple bin "trees" increase CPU usage a lot? I'm thinking 
of the use case where there are 10 computers in the home, of which one is 
doing 100 upload TCP sessions (bittorrent). If there was some mechanism 
that could identify a source IP address that caused most of the congestion 
and put them in a separate bin "tree" (basically each source IP would get 
its own set of bins, up to a maximum of perhaps 16 trees) and then these 
sets of bins would be emptied in a round-robin fashion (MDRR perhaps?). Or 
perhaps there would just be a "high volume speaker" tree and a "low volume 
speaker" tree, and it would be enough to just MDRR between these two 
trees. I realise we're now into "flow" territory which is exactly what 
Codel doesn't need to bother with currently, but just checking...

I saw in ICCRG that some testing was done with LEDBAT which I would 
imagine would help for the bittorrent case, but what about trying to 
achieve some of that without needing LEDBAT by using the above technique?

---

From: Andrew McGregor <[email-masked]>
Date: Thu, 14 Mar 2013 09:38:09 -0400

fq_codel achieves nearly all of what you're suggesting already, except that
it works on a flow basis rather than on a host basis.  You could add a DRR
layer hashing on IP address only to get host-level balancing.  An efficient
implementation of that would again add only a very little overhead.

fq_codel actually has a 'high volume' and 'low volume' flow rotation, which
are called 'old' and 'new' in the code, and commonly 'thick' and 'thin'
flows in mailing list discussions.  Flows arrive in the thin flow list, get
demoted to the thick flow list when they build queue, and are removed from
either list when they drain completely.  When a flow reappears it will be
introduced back in to the thin flow list, but it will keep its codel state.
 Flows are served from the thin list until it is empty, and then from the
thick list, in round-robin order, so it is very much like DRR.

On Thu, Mar 14, 2013 at 1:07 AM, Mikael Abrahamsson <[email-masked]>wrote:

> using. The WNDR3800 seems to be a fairly high end device when I look at the
> price point.
>
> too cpu intensive as the hashing into bins sounds efficient on a per-packet
> level.
>
> speaker" tree, and it would be enough to just MDRR between these two trees.
> I realise we're now into "flow" territory which is exactly what Codel
> doesn't need to bother with currently, but just checking...
>