Subject: [aqm] Text for aqm-recommendation on independent ECN config

From: Bob Briscoe <[email-masked]>
Date: Thu, 5 Dec 2013 18:13:05 +0000

Fred, Gorry, all,

I promised to suggest text for draft-ietf-aqm-recommendation about 
allowing the AQM's behaviour to be independent for ECN and non-ECN 
packets. In the process, I realised we can't talk about independent 
AQMs for ECN without also including Diffserv.

This gets messy, because I believe a good AQM for BE traffic with and 
without ECN, should remove much if not all the need for Diffserv. But 
we can't ignore Diffserv.

_________________________________________________________________________________________
{In Section 4: add another bullet between recommendations 2 & 3:}

3{New}. It SHOULD be possible to make different instances of an AQM 
algorithm apply to different subsets of packets that share the same 
queue. It SHOULD be possible to classify packets into these subsets 
at least by ECN codepoint [RFC3168] and Diffserv codepoint [RFC2474] 
(or the equivalent of these fields at lower layers).

{Then a new section to expand on this before the current Section 4.3.}
4.3{New}. Independent AQM Instances for ECN and Diffserv

The recommendation to provide a separate instance of the AQM for ECN 
packets goes beyond the assumptions of RFC 3168, which assumed that 
only one instance of an AQM will handle both ECN-capable and 
non-ECN-capable packets.

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 5 Dec 2013 19:49:59 +0000

Thanks.

Gorry, Matt Mathis also said he would be putting together some notes. 

On Dec 5, 2013, at 10:13 AM, Bob Briscoe <[email-masked]>
 wrote:

> Fred, Gorry, all,
> 
> I promised to suggest text for draft-ietf-aqm-recommendation about allowing the AQM's behaviour to be independent for ECN and non-ECN packets. In the process, I realised we can't talk about independent AQMs for ECN without also including Diffserv.
> 
> This gets messy, because I believe a good AQM for BE traffic with and without ECN, should remove much if not all the need for Diffserv. But we can't ignore Diffserv.
> 
> 3{New}. It SHOULD be possible to make different instances of an AQM algorithm apply to different subsets of packets that share the same queue. It SHOULD be possible to classify packets into these subsets at least by ECN codepoint [RFC3168] and Diffserv codepoint [RFC2474] (or the equivalent of these fields at lower layers).
> 
> The recommendation to provide a separate instance of the AQM for ECN packets goes beyond the assumptions of RFC 3168, which assumed that only one instance of an AQM will handle both ECN-capable and non-ECN-capable packets.
> 
> 
> 
> Bob
> 
> 
> 

------------------------------------------------------
8 issues in virtual infrastructure
http://dcrocker.net/#fallacies

---

From: Jim Gettys <[email-masked]>
Date: Thu, 5 Dec 2013 14:50:23 -0500

On Thu, Dec 5, 2013 at 1:13 PM, Bob Briscoe <[email-masked]> wrote:

> allowing the AQM's behaviour to be independent for ECN and non-ECN packets.
> In the process, I realised we can't talk about independent AQMs for ECN
> without also including Diffserv.
>
> without ECN, should remove much if not all the need for Diffserv. But we
> can't ignore Diffserv.
>

I agree in principle with what Bob is trying to say here (and is very much
what I've been saying in my blog entry of last summer).

Once you have things under control, the need for Diffserv diminishes
dramatically (but does not go away).

But as Bob notes, there is still a good use for Diffserv: suitably marked
traffic may want to contend for access to the channel differently: your
marked VOIP packets may want to change the priority with which you request
channel access, so that you get more timely access to the medium. This
conserves transmit opportunities, which is often the scarcest resource in
many systems (e.g. 802.11, DOCSIS, etc.). This can be the difference
between your VOIP working well, and not working well, on a busy 802.11
network as well as using the channel as efficiently as possible.

Similarly, if you have packets you know are background, it's helpful to
know that to ensure that they never contend for access to the medium but
will always defer to other traffic, and just scavenge available space in
other transmit opportunities where possible.

I'm a bit loathe though to tie the behavior to queues, however; in
particular, best effort traffic may want to be sent in the same aggregate
as higher (or lower) priority traffic, if there is remaining space in the
aggregate.

In short, the mental model we've had that there is a one-to-one model of
hardware and software queues (not to mention flows in a given software
queue) is often incorrect (or at least seriously sub-optimal) in today's
systems (even if the hardware queues "work" properly, which it appears they
do not in 802.11).

So I'm not sure Bob's new section 3 here is how to best to state this (or
to deal with the terminology problem).  Certainly, it may be the same
instance of an AQM algorithm, rather than different instances, for example.
 And "

It SHOULD be possible" is more a pious wish than anything else.  But I
agree in spirit with what Bob's trying to say.
                               - Jim

> algorithm apply to different subsets of packets that share the same queue.
> 
> 
> It SHOULD be possible to classify packets into these subsets at least by
> ECN codepoint [RFC3168] and Diffserv codepoint [RFC2474] (or the equivalent
> of these fields at lower layers).
>
> packets goes beyond the assumptions of RFC 3168, which assumed that only
> one instance of an AQM will handle both ECN-capable and non-ECN-capable
> packets.
>
>
>
> Bob
>
>
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: [email-masked]
Date: Fri, 6 Dec 2013 10:46:21 -0000

OK, I'll check through the AQM thread before the end of next week and
summarise what I think has been commented and what we have agreed to
change.

I think the latest discussion on requirements that change diffserv/ECN may
need a strong consensus - so it would be good to understand this.

Gorry

>> packets. In the process, I realised we can't talk about independent AQMs
>> for ECN without also including Diffserv.
>>
>> queue. It SHOULD be possible to classify packets into these subsets at
>> least by ECN codepoint [RFC3168] and Diffserv codepoint [RFC2474] (or
>> the equivalent of these fields at lower layers).
>>

---

From: "Robinson, Dave C (Dave)" <[email-masked]>
Date: Fri, 6 Dec 2013 15:22:41 +0000

Jim et al,

I agree with your comments on Diffserv. I'd go further and characterise Diffserv as a business tool whereas ECN and general AQMs as network optimisation tools. By that I mean that Diffserv should be used by an application to specify a business policy be it low latency for VoIP / gaming or preferred forwarding for your video service. AQMs can guess at the types of traffic. For example an isochronous flow of short packets could well be VoIP which would benefit from expedited forwarding. But it has to be an operator policy to decide between two VoIP services.

Same argument for bulk traffic from different video services. One can be a partner traffic from your internal CDN nodes which you would like to accelerate through your network.

ECN is a good tools to use to tell sources the the network is becoming overloaded and is kinder than dropping packets. But it cannot know the business policy. Just help manage congestion.

Regards
      Dave

Sent from my iPad

On 5 Dec 2013, at 19:50, Jim Gettys <[email-masked]<mailto:[email-masked]>> wrote:

On Thu, Dec 5, 2013 at 1:13 PM, Bob Briscoe <[email-masked]<mailto:[email-masked]>> wrote:
Fred, Gorry, all,

I agree in principle with what Bob is trying to say here (and is very much what I've been saying in my blog entry of last summer).

Once you have things under control, the need for Diffserv diminishes dramatically (but does not go away).

But as Bob notes, there is still a good use for Diffserv: suitably marked traffic may want to contend for access to the channel differently: your marked VOIP packets may want to change the priority with which you request channel access, so that you get more timely access to the medium. This conserves transmit opportunities, which is often the scarcest resource in many systems (e.g. 802.11, DOCSIS, etc.). This can be the difference between your VOIP working well, and not working well, on a busy 802.11 network as well as using the channel as efficiently as possible.

Similarly, if you have packets you know are background, it's helpful to know that to ensure that they never contend for access to the medium but will always defer to other traffic, and just scavenge available space in other transmit opportunities where possible.

I'm a bit loathe though to tie the behavior to queues, however; in particular, best effort traffic may want to be sent in the same aggregate as higher (or lower) priority traffic, if there is remaining space in the aggregate.

In short, the mental model we've had that there is a one-to-one model of hardware and software queues (not to mention flows in a given software queue) is often incorrect (or at least seriously sub-optimal) in today's systems (even if the hardware queues "work" properly, which it appears they do not in 802.11).

So I'm not sure Bob's new section 3 here is how to best to state this (or to deal with the terminology problem).  Certainly, it may be the same instance of an AQM algorithm, rather than different instances, for example.  And "
It SHOULD be possible" is more a pious wish than anything else.  But I agree in spirit with what Bob's trying to say.
                               - Jim

3{New}. It SHOULD be possible to make different instances of an AQM algorithm apply to different subsets of packets that share the same queue.
It SHOULD be possible to classify packets into these subsets at least by ECN codepoint [RFC3168] and Diffserv codepoint [RFC2474] (or the equivalent of these fields at lower layers).

Bob

________________________________________________________________
Bob Briscoe,                                                  BT
_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 6 Dec 2013 15:51:11 +0000

Generally agree. That said, the Conex work has suggested that ECN marking or thresholds might operate differently depending on the DSCP. As a manufacturer/implementor, my comment is that all things are possible to him who doesn't have to do it. However, I "get" that for elastic traffic "congestion experienced" might be defined as "delay, or projected delay, exceeded some threshold", while for a class of real-time traffic it might be defined as "octet throughput rate within this class exceeds some threshold".

On Dec 6, 2013, at 7:22 AM, "Robinson, Dave C (Dave)" <[email-masked]> wrote:

> On 5 Dec 2013, at 19:50, Jim Gettys <[email-masked]> wrote:
> 
>> 
>> 
>> 
>> So I'm not sure Bob's new section 3 here is how to best to state this (or to deal with the terminology problem).  Certainly, it may be the same instance of an AQM algorithm, rather than different instances, for example.  And " It SHOULD be possible" is more a pious wish than anything else.  But I agree in spirit with what Bob's trying to say.
>>                                - Jim
>> 
>>

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 06 Dec 2013 15:57:59 +0000

... and RFC 4774 describes how ECN interpretation can change based on DSCP.

Gorry

On 06/12/2013 15:51, Fred Baker (fred) wrote:
> Generally agree. That said, the Conex work has suggested that ECN
> marking or thresholds might operate differently depending on the DSCP.
> As a manufacturer/implementor, my comment is that all things are
> possible to him who doesn't have to do it. However, I "get" that for
> elastic traffic "congestion experienced" might be defined as "delay, or
> projected delay, exceeded some threshold", while for a class of
> real-time traffic it might be defined as "octet throughput rate within
> this class exceeds some threshold".
>
> On Dec 6, 2013, at 7:22 AM, "Robinson, Dave C (Dave)"
> <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>> Jim et al,
>>
>> I agree with your comments on Diffserv. I'd go further and
>> characterise Diffserv as a business tool whereas ECN and general AQMs
>> as network optimisation tools. By that I mean that Diffserv should be
>> used by an application to specify a business policy be it low latency
>> for VoIP / gaming or preferred forwarding for your video service. AQMs
>> can guess at the types of traffic. For example an isochronous flow of
>> short packets could well be VoIP which would benefit from expedited
>> forwarding. But it has to be an operator policy to decide between two
>> VoIP services.
>>
>> Same argument for bulk traffic from different video services. One can
>> be a partner traffic from your internal CDN nodes which you would like
>> to accelerate through your network.
>>
>> ECN is a good tools to use to tell sources the the network is becoming
>> overloaded and is kinder than dropping packets. But it cannot know the
>> business policy. Just help manage congestion.
>>
>> Regards
>>       Dave
>>
>> Sent from my iPad
>>
>>>     I promised to suggest text for draft-ietf-aqm-recommendation
>>>     about allowing the AQM's behaviour to be independent for ECN and
>>>     non-ECN packets. In the process, I realised we can't talk about
>>>     independent AQMs for ECN without also including Diffserv.
>>>
>>>     This gets messy, because I believe a good AQM for BE traffic with
>>>     and without ECN, should remove much if not all the need for
>>>     Diffserv. But we can't ignore Diffserv.
>>>
>>>
>>> I agree in principle with what Bob is trying to say here (and is very
>>> much what I've been saying in my blog entry of last summer).
>>>
>>> But as Bob notes, there is still a good use for Diffserv: suitably
>>> marked traffic may want to contend for access to the channel
>>> differently: your marked VOIP packets may want to change the priority
>>> with which you request channel access, so that you get more timely
>>> access to the medium. This conserves transmit opportunities, which is
>>> often the scarcest resource in many systems (e.g. 802.11, DOCSIS,
>>> etc.). This can be the difference between your VOIP working well, and
>>> not working well, on a busy 802.11 network as well as using the
>>> channel as efficiently as possible.
>>>
>>> Similarly, if you have packets you know are background, it's helpful
>>> to know that to ensure that they never contend for access to the
>>> medium but will always defer to other traffic, and just scavenge
>>> available space in other transmit opportunities where possible.
>>>
>>> particular, best effort traffic may want to be sent in the same
>>> aggregate as higher (or lower) priority traffic, if there is
>>> remaining space in the aggregate.
>>>
>>> In short, the mental model we've had that there is a one-to-one model
>>> of hardware and software queues (not to mention flows in a given
>>> software queue) is often incorrect (or at least seriously
>>> sub-optimal) in today's systems (even if the hardware queues "work"
>>> properly, which it appears they do not in 802.11).
>>>
>>> So I'm not sure Bob's new section 3 here is how to best to state this
>>> (or to deal with the terminology problem).  Certainly, it may be the
>>> same instance of an AQM algorithm, rather than different instances,
>>> for example.  And "
>>> It SHOULD be possible" is more a pious wish than anything else.  But
>>> I agree in spirit with what Bob's trying to say.
>>>                                - Jim
>>>
>>>
>>>     _____________________________________________________________________________________________
>>>     3{New}. It SHOULD be possible to make different instances of an
>>>     AQM algorithm apply to different subsets of packets that share
>>>     the same queue.
>>>     It SHOULD be possible to classify packets into these subsets at
>>>     least by ECN codepoint [RFC3168] and Diffserv codepoint [RFC2474]
>>>     The recommendation to provide a separate instance of the AQM for
>>>     ECN packets goes beyond the assumptions of RFC 3168, which
>>>     assumed that only one instance of an AQM will handle both
>>>     ECN-capable and non-ECN-capable packets.
>>>
>>>
>>>
>>>     Bob
>>>
>>>
>>>
>>>     ____________________________________________________________________
>>>     Bob Briscoe,                                                  BT
>>>     _________________________________________________
>>>     aqm mailing list
>>>     [email-masked] <mailto:[email-masked]>
>>>     https://www.ietf.org/mailman/__listinfo/aqm
>>>     <https://www.ietf.org/mailman/listinfo/aqm>
>>>
>>>
>
>

---

From: David Collier-Brown <[email-masked]>
Date: Fri, 06 Dec 2013 11:41:57 -0500

> business policy. Just help manage congestion.

1) it's also a non-business policy issue: I may want to provide low
latency to a friend or child playing a game at my house, or provide
poorer time/throughput guarantees to a web or ftp download. Yes, this
indeed a niggle (;-))

2) I'd be happier providing a metric to the network's management than an
implementation. 
Let's assume I'm doing VOIP or something else small and low-latency to a
known endpoint, and want to say for this socket, (I want = <100
milliseconds, I'm inducing  load =  0.03 mbits/sec).  The implementation
can turn that into a DIFFSERV option, if it's supported, or something
else if not.   This, IMHO, deserves consideration, but not as part of
this discussion.  Thus "off topic".

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain
(416) 223-8968

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Dec 2013 03:04:26 +0000

Jim,

I'm just checking we're not talking past each other. I'll repeat two 
quotes from each of us, then comment.

>On Thu, Dec 5, 2013 at 1:13 PM, Bob Briscoe 
><<mailto:[email-masked]>[email-masked]> wrote:
>>(or the equivalent of these fields at lower layers),

At 19:50 05/12/2013, Jim Gettys wrote:
>"Certainly, it may be the same instance of an AQM algorithm, rather 
>than different instances, for example."

That's true of course, but the case with one AQM handling all packets 
within a queue is the norm. I want to check you're happy with the converse:
1) A set-up more like WRED which was based on Dave Clark's RIO (RED 
with in and out of contract). So we can have WPIE, WCoDel etc where 
the differentiation between aggregates is provided by different AQM 
instances in the same queue, not by different queues with different 
scheduling priorities.
2) Extending this so that AQM differentiation can be between 
ECN-capable and Not-ECN-capable aggregates, not just between Diffserv 
classes (an example being CoDel with a lower 'interval' for 
ECN-capable packets).

I presented the evaluations of this last idea in tsvwg on the final 
Friday of the Vancouver IETF - I don't think you were there. 
<http://www.ietf.org/proceedings/88/slides/slides-88-tsvwg-20.pdf>
This is my primary motivation for this wordsmithing - I'm trying 
allow us to move towards zero signalling delays in CoDel, PIE and RED 
(currently defaults of 200ms, 100ms and 512packets respectively, 
which are not good for dynamics).

Bob

At 19:50 05/12/2013, Jim Gettys wrote:

>On Thu, Dec 5, 2013 at 1:13 PM, Bob Briscoe 
><<mailto:[email-masked]>[email-masked]> wrote:
>Fred, Gorry, all,
>
>I agree in principle with what Bob is trying to say here (and is 
>very much what I've been saying in my blog entry of last summer).
>
>differently: your marked VOIP packets may want to change the 
>priority with which you request channel access, so that you get more 
>timely access to the medium. This conserves transmit opportunities, 
>which is often the scarcest resource in many systems (e.g. 802.11, 
>DOCSIS, etc.). This can be the difference between your VOIP working 
>well, and not working well, on a busy 802.11 network as well as 
>using the channel as efficiently as possible.
>
>In short, the mental model we've had that there is a one-to-one 
>model of hardware and software queues (not to mention flows in a 
>given software queue) is often incorrect (or at least seriously 
>So I'm not sure Bob's new section 3 here is how to best to state 
>this (or to deal with the terminology problem).  Certainly, it may 
>be the same instance of an AQM algorithm, rather than different 
>instances, for example.  And "
><mailto:[email-masked]>[email-masked]

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 11 Dec 2013 04:06:47 +0000

On Dec 10, 2013, at 7:04 PM, Bob Briscoe <[email-masked]>
 wrote:

> This is my primary motivation for this wordsmithing - I'm trying allow us to move towards zero signalling delays in CoDel, PIE and RED (currently defaults of 200ms, 100ms and 512packets respectively, which are not good for dynamics).

Violent agreement. The target, I should think, is zero statistical delay.

---

From: Jim Gettys <[email-masked]>
Date: Wed, 11 Dec 2013 11:55:51 -0500

On Tue, Dec 10, 2013 at 10:04 PM, Bob Briscoe <[email-masked]> wrote:

> of these fields at lower layers),
>
>
> "Certainly, it may be the same instance of an AQM algorithm, rather than
> different instances, for example."
>
>
> 1) A set-up more like WRED which was based on Dave Clark's RIO (RED with
> in and out of contract). So we can have WPIE, WCoDel etc where the
> differentiation between aggregates is provided by different AQM instances
> in the same queue, not by different queues with different scheduling
> priorities.
> 2) Extending this so that AQM differentiation can be between ECN-capable
> and Not-ECN-capable aggregates, not just between Diffserv classes (an
> example being CoDel with a lower 'interval' for ECN-capable packets).
>
> I presented the evaluations of this last idea in tsvwg on the final Friday
> of the Vancouver IETF - I don't think you were there. <http://www.ietf.org/proceedings/88/slides/slides-88-tsvwg-20.pdf>
>

Yes, unfortunately I had to leave before the Friday session.

> This is my primary motivation for this wordsmithing - I'm trying allow us
> to move towards zero signalling delays in CoDel, PIE and RED (currently
> defaults of 200ms, 100ms and 512packets respectively, which are not good
> for dynamics).
>

Certainly signalling delays are very important: this is why I'm favorably
inclined to "head mark/drop", as it signals TCP as quickly as possible,
keeping the response of the TCP feedback loop as tight as possible (and
part of why I like CoDel so much for the highly variable bandwidth problem
we face at the edge of the net).

It's *really* important than when the bandwidth drops suddenly that
everyone gets told to slow down quickly (exactly how quickly probably
depends on the propagation change characteristics of the medium), or
packets can pile up in a big way.

How quickly the mark/drop algorithm can figure out that signalling is
appropriate is the *other* piece of getting good dynamics.  Here I don't
doubt that something may be discovered that is better than CoDel in the
slightest.

> Bob

My basic issue is one of terminology: people have talked about "best
effort" queues.  In reality, this is a "class" of service, rather than a
single queue, and when you get into the mental model of BE being a single
queue, (rather than a set of queues) it can lead one astray quickly and
easily.

It's really easy to fall into the idea of a single software queue mapping
to some single hardware supported queue, and that's a cognitive mistake, as
aggregating MACs are showing us; transmit ops are often the scarcest
resource...

Diffserv marking has the potential to give a "hint" to distinguish how
particular flows should be handled (scheduled) in a service class, and as
my previous example shows, that hint may be very useful in channel access
decisions (e.g. voip on 802.11).

But fq_codel teaches the lesson that packet scheduling combined with
keeping TCP sane is a key improvement over handling either problem apart...
In particular, the first packets of new flows/reappearing flows are vastly
more "important" than other packets in terms of the latency cost to users
of that service. Each flow has in essence its own queue in this service
class, and we're using information from that to help schedule the packets
in ways that minimize latency to the user.

So in this case, a single algorithm is acting over a bunch of flows in a
single class of service, and both scheduling packets among the flows, and
signalling TCP flows appropriately when they should "slow down".

So I think you and I are on close to the same page (but have been burned
badly in the past by terminology issues getting in the way).  On HTTP/1.1
we wasted probably > 2 years talking past each other because we didn't have
clear and concise terminology that we all understood the same way.

And I don't claim I have the right terminology for all this stuff, either
(even in this mail).

Which is why I was loathe to suggest exact text...
                           - Jim

> instance of an AQM algorithm, rather than different instances, for
> example.  And "

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 11 Dec 2013 19:21:14 +0000

Jim,

At 16:55 11/12/2013, Jim Gettys wrote:

>On Tue, Dec 10, 2013 at 10:04 PM, Bob Briscoe 
>>>AQM algorithm apply to different subsets of packets that share the same queue.
>That's true of course, but the case with one AQM handling all 
>packets within a queue is the norm. I want to check you're happy 
>with the converse:
>ECN-capable and Not-ECN-capable aggregates, not just between 
>Diffserv classes (an example being CoDel with a lower 'interval' for 
>ECN-capable packets).
>
>Friday of the Vancouver IETF - I don't think you were there. < 
>http://www.ietf.org/proceedings/88/slides/slides-88-tsvwg-20.pdf >
>
>
>allow us to move towards zero signalling delays in CoDel, PIE and 
>RED (currently defaults of 200ms, 100ms and 512packets respectively, 
>Certainly signalling delays are very important: this is why I'm 
>favorably inclined to "head mark/drop", as it signals TCP as quickly 
>as possible, keeping the response of the TCP feedback loop as tight 
>as possible (and part of why I like CoDel so much for the highly 
>variable bandwidth problem we face at the edge of the net).
>
>everyone gets told to slow down quickly (exactly how quickly 
>probably depends on the propagation change characteristics of the 
>medium), or packets can pile up in a big way.
>
>How quickly the mark/drop algorithm can figure out that signalling 
>is appropriate is the *other* piece of getting good dynamics.  Here 
>I don't doubt that something may be discovered that is better than 
>CoDel in the slightest.

It takes a CoDel instance (within an fq structure) 200ms from its 
queue first passing 'threshold' before it will ever drop the first 
packet (unless the queue hits taildrop before that). So if the RTT is 
20ms, that's 220ms signalling delay. In fq_codel this creates 
considerable self-delay for short flows or r-t apps, which kill their 
own latency before they get any loss signal to tell them to slow 
down. Even for elastic flows, with congestion signals delayed by so 
much, they risk hitting themselves with a huge train of overshoot 
loss. This would be the same for fq_pie, except the number is 100ms + RTT.

Yes, the e2e transport could measure delay growth, but it doesn't 
know whether the delay is coming from a queue that is isolated from 
others or not. So it doesn't want to slow down too quickly in 
response to delay growth in case it gets screwed by other traffic. 
Ie. using delay growth as a signal entails considerable signalling 
delay due to all the uncertainty.

The proposal you missed in tsvwg was to define ECN as an immediate 
signal from the network, 'interval'=0 in CoDel terms, so the host 
always gets congestion signals as fast as possible, and if it needs 
bursts of signals smoothed out, it can do that itself.

The suggested wording ensures all AQM implementations will allow 
operators, vendors and users to configure such a mechanism. But I've 
generalised it from ECN to Diffserv too (because the implementation 
would be no different).

>effort" queues.  In reality, this is a "class" of service, rather 
>than a single queue, and when you get into the mental model of BE 
>being a single queue, (rather than a set of queues) it can lead one 
>astray quickly and easily.

Yeah, I know this. I suspected we were talking past each other.

I need you to allow the other case into your mind for this 
conversation. The wording is specifically about the case where 
"different subsets of packets ... share the same queue".

We can talk about an fq structure for this another time, but it's a 
really complicated way of doing it. Given simple looks like it could 
work, why get complicated already?

>It's really easy to fall into the idea of a single software queue 
>mapping to some single hardware supported queue, and that's a 
>cognitive mistake, as aggregating MACs are showing us; transmit ops 
>are often the scarcest resource...

It's only a cognitive mistake if one is not aware of all the options. 
I'm fully aware of all the options.

To be specific, a queue into a wireless medium should be configured 
so it holds some 'good queue' in reserve for transmit ops, but the 
queues on top of this that TCP self-inflicts even briefly are not 
'good queues' even if they are isolated from other flows by fq - VJ 
was wrong to generalise the phrase 'good queue' to all bursts of 
queue - it is only necessary to hold back from signalling and allow a 
burst of queue if the only possible signal is a drop. With ECN, you 
don't have this dilemma. This is the key to rapid dynamics.

>Diffserv marking has the potential to give a "hint" to distinguish 
>how particular flows should be handled (scheduled) in a service 
>class, and as my previous example shows, that hint may be very 
>useful in channel access decisions (e.g. voip on 802.11).
>
>keeping TCP sane is a key improvement over handling either problem 
>apart... In particular, the first packets of new flows/reappearing 
>flows are vastly more "important" than other packets in terms of the 
>latency cost to users of that service. Each flow has in essence its 
>own queue in this service class, and we're using information from 
>that to help schedule the packets in ways that minimize latency to the user.

I know all this. Please can we keep to the conversation about how to 
avoid the 200ms signalling delay that fq_codel inflicts on each flow 
(and the similar signalling delays that other AQMs inflict).

>So in this case, a single algorithm is acting over a bunch of flows 
>in a single class of service, and both scheduling packets among the 
>flows, and signalling TCP flows appropriately when they should "slow down".

Yup, I know this.

>So I think you and I are on close to the same page (but have been 
>burned badly in the past by terminology issues getting in the 
>way).  On HTTP/1.1 we wasted probably > 2 years talking past each 
>other because we didn't have clear and concise terminology that we 
>all understood the same way.

As I thought, we are talking past each other. We need to be able to 
have a conversation that is not always "Hmm, that's sounds like it 
might be interesting. Can I tell you about fq_codel now?"

Bob

>And I don't claim I have the right terminology for all this stuff, 
>either (even in this mail).
>
>>Once you have things under control, the need for Diffserv 
>>diminishes dramatically (but does not go away).
>>
>>priority with which you request channel access, so that you get 
>>more timely access to the medium. This conserves transmit 
>>opportunities, which is often the scarcest resource in many systems 
>>(e.g. 802.11, DOCSIS, etc.). This can be the difference between 
>>your VOIP working well, and not working well, on a busy 802.11 
>>Similarly, if you have packets you know are background, it's 
>>helpful to know that to ensure that they never contend for access 
>>to the medium but will always defer to other traffic, and just 
>>scavenge available space in other transmit opportunities where possible.
>>
>>It SHOULD be possible" is more a pious wish than anything 
>>else.  But I agree in spirit with what Bob's trying to say.
>>                                - Jim
>>
>>ECN packets goes beyond the assumptions of RFC 3168, which assumed 
>>that only one instance of an AQM will handle both ECN-capable and 
>>non-ECN-capable packets.
>>
>>
>>Bob
>>
>>

---

From: Dave Taht <[email-masked]>
Date: Wed, 11 Dec 2013 12:00:49 -0800

On Wed, Dec 11, 2013 at 11:21 AM, Bob Briscoe <[email-masked]> wrote:
> I'm just checking we're not talking past each other. I'll repeat two quotes
> from each of us, then comment.
>
> It SHOULD be possible to classify packets into these subsets at least by ECN
> codepoint [RFC3168] and Diffserv codepoint [RFC2474] (or the equivalent of
> these fields at lower layers),
>
>
> That's true of course, but the case with one AQM handling all packets within
> a queue is the norm. I want to check you're happy with the converse:
> 1) A set-up more like WRED which was based on Dave Clark's RIO (RED with in
> and out of contract). So we can have WPIE, WCoDel etc where the
> differentiation between aggregates is provided by different AQM instances in
> the same queue, not by different queues with different scheduling
> priorities.
> 2) Extending this so that AQM differentiation can be between ECN-capable and
> Not-ECN-capable aggregates, not just between Diffserv classes (an example
> being CoDel with a lower 'interval' for ECN-capable packets).
>
> of the Vancouver IETF - I don't think you were there. <
> This is my primary motivation for this wordsmithing - I'm trying allow us to
> move towards zero signalling delays in CoDel, PIE and RED (currently
> defaults of 200ms, 100ms and 512packets respectively, which are not good for
> dynamics).
>
>
> keeping the response of the TCP feedback loop as tight as possible (and part
> of why I like CoDel so much for the highly variable bandwidth problem we
> face at the edge of the net).
>
> It's *really* important than when the bandwidth drops suddenly that everyone
> gets told to slow down quickly (exactly how quickly probably depends on the
> propagation change characteristics of the medium), or packets can pile up in
> a big way.
>
> It takes a CoDel instance (within an fq structure) 200ms from its queue
> first passing 'threshold' before it will ever drop the first packet (unless
> the queue hits taildrop before that). So if the RTT is 20ms, that's 220ms
> signalling delay. In fq_codel this creates considerable self-delay for short
> flows or r-t apps, which kill their own latency before they get any loss
> signal to tell them to slow down. Even for elastic flows, with congestion
> signals delayed by so much, they risk hitting themselves with a huge train
> of overshoot loss. This would be the same for fq_pie, except the number is
> 100ms + RTT.

Things have so consistently expressed things this way that I began to
doubt the reality myself. It seems like a large number of folk on this list
don't get it either, so I am going to try an explain in a new way.

Tackling codel first:

the first phase of codel has effectively a "training" period where a link going
from unloaded to loaded for the first time ever - the very first drop with the
default interval will happen in 200ms, yes. IF it stays loaded and  over the
target delay after the first drop/mark, it will then tune to ever
smaller intervals
to approximate an ideal drop rate until the latency on the link drops below the
target. At which point the algorithm saves that rate, and stops doing anything
until the next time the target delay is exceeded.

Some keep asserting that that is all there is to codel, saying things like
 "there is a linear increase in drop probability" using the invsqrt mechanism,

*which is true during the training phase*.

After that approximation of ideal drop/mark rate is obtained, the algorithm
goes quiescent until the next time the target delay is consistently exceeded,
at which point it schedules the next drop at a little more than the stored
previous drop rate. It then continually seeks around that point up and down.

If the delay drops below the target in this phase, the algorithm stops
again and decreases the drop rate again, as it's too high. If the delay
stays above target after the drop for the current value of the interval,
the drop rate increases.

This is an interesting solution to kleinrocks formulation of "power", where
he once said an average of one packet should be in the queue, codel aims
to never have less than one packet in the queue.

And the switch into and out of drop mode going above target is entirely
dependent on the characteristics over time of the flows on the system,
completely nonlinear, and where codel spends 99.999% of it's time on a
loaded link.

As debussy said: "Music is the *space* between the notes".

I wish i had a name for this second "seeking" phase that makes as much sense as
"congestion avoidance".

So asserting that you'll have a 200ms interval on a codled link always is just
blatantly incorrect. On first boot, yes. On a busy network, never again.[1]

In the event of a link going completely idle, and staying
idle, there is hysteresis built into the code so it will retain that
drop rate for a few
hundred milliseconds (it's 8*interval in some versions of the code,
4 in others), before resetting count to 1 and the resulting estimation
window to interval.

It is certainly possible to come up with a codel variant that more
closely matches the preconditions of the network, and the speeds on
the link.

In the data center case there there are two knobs to set the target
delay and interval to the desired time width of that network. I keep
hoping someone in a datacenter with resources to play will merely set
them, and report the results. I'd suggest target 500us and interval
10ms as numbers most modern x86 server hardware can reliably
accomplish without invoking other scheduling delays from cpu
schedulers, etc. Even lower values are possible, but
these two are fairly close to a data center width, and I hope someone
will try them. [2]

PLEASE? I simply don't have data center resources, yet the experiment
is EASY if you do. (note: ECN is on by default in fq_codel and off by
default in codel.)

:whew: hopefully that explains that.

NEXT, is tackling what has been said about the behavior of fq_codel in this
environment, but to avoid cluttering up the issue, I'll stop here at
getting codel
described right, today. And go back to sigcomm for a while

[1] on fq_codel the values are cached, too, but… well… I will try to explain
later. yes, more work is needed in this area.

[2] It might make sense to modify the hysterisis in the data center case

> Yes, the e2e transport could measure delay growth, but it doesn't know
> whether the delay is coming from a queue that is isolated from others or
> not. So it doesn't want to slow down too quickly in response to delay growth
> in case it gets screwed by other traffic. Ie. using delay growth as a signal
> entails considerable signalling delay due to all the uncertainty.
>
> The proposal you missed in tsvwg was to define ECN as an immediate signal
> from the network, 'interval'=0 in CoDel terms, so the host always gets
> congestion signals as fast as possible, and if it needs bursts of signals
> smoothed out, it can do that itself.
>
> The suggested wording ensures all AQM implementations will allow operators,
> vendors and users to configure such a mechanism. But I've generalised it
> from ECN to Diffserv too (because the implementation would be no different).
>
>
>
> My basic issue is one of terminology: people have talked about "best effort"
> queues.  In reality, this is a "class" of service, rather than a single
> queue, and when you get into the mental model of BE being a single queue,
> (rather than a set of queues) it can lead one astray quickly and easily.
>
>
> I need you to allow the other case into your mind for this conversation. The
> wording is specifically about the case where "different subsets of packets
> ... share the same queue".
>
> We can talk about an fq structure for this another time, but it's a really
> complicated way of doing it. Given simple looks like it could work, why get
> complicated already?
>
>
> It's really easy to fall into the idea of a single software queue mapping to
> some single hardware supported queue, and that's a cognitive mistake, as
> It's only a cognitive mistake if one is not aware of all the options. I'm
> fully aware of all the options.
>
> To be specific, a queue into a wireless medium should be configured so it
> holds some 'good queue' in reserve for transmit ops, but the queues on top
> of this that TCP self-inflicts even briefly are not 'good queues' even if
> they are isolated from other flows by fq - VJ was wrong to generalise the
> phrase 'good queue' to all bursts of queue - it is only necessary to hold
> back from signalling and allow a burst of queue if the only possible signal
> is a drop. With ECN, you don't have this dilemma. This is the key to rapid
> dynamics.
>
>
> particular flows should be handled (scheduled) in a service class, and as my
> previous example shows, that hint may be very useful in channel access
> But fq_codel teaches the lesson that packet scheduling combined with keeping
> TCP sane is a key improvement over handling either problem apart... In
> particular, the first packets of new flows/reappearing flows are vastly more
> "important" than other packets in terms of the latency cost to users of that
> service. Each flow has in essence its own queue in this service class, and
> we're using information from that to help schedule the packets in ways that
> minimize latency to the user.
>
>
> I know all this. Please can we keep to the conversation about how to avoid
> the 200ms signalling delay that fq_codel inflicts on each flow (and the
> similar signalling delays that other AQMs inflict).
>
>
>
> badly in the past by terminology issues getting in the way).  On HTTP/1.1 we
> wasted probably > 2 years talking past each other because we didn't have
> As I thought, we are talking past each other. We need to be able to have a
> conversation that is not always "Hmm, that's sounds like it might be
> interesting. Can I tell you about fq_codel now?"
>
>
>
> Bob
>
>
>
> I promised to suggest text for draft-ietf-aqm-recommendation about allowing
> the AQM's behaviour to be independent for ECN and non-ECN packets. In the
> process, I realised we can't talk about independent AQMs for ECN without
> also including Diffserv.
> many systems (e.g. 802.11, DOCSIS, etc.). This can be the difference between
> your VOIP working well, and not working well, on a busy 802.11 network as
> well as using the channel as efficiently as possible.
>
> Similarly, if you have packets you know are background, it's helpful to know
> that to ensure that they never contend for access to the medium but will
> always defer to other traffic, and just scavenge available space in other
> transmit opportunities where possible.
>
> particular, best effort traffic may want to be sent in the same aggregate as
> higher (or lower) priority traffic, if there is remaining space in the
> aggregate.
>
> So I'm not sure Bob's new section 3 here is how to best to state this (or to
> deal with the terminology problem).  Certainly, it may be the same instance
> of an AQM algorithm, rather than different instances, for example.  And "
> It SHOULD be possible to classify packets into these subsets at least by ECN
> codepoint [RFC3168] and Diffserv codepoint [RFC2474] (or the equivalent of
> these fields at lower layers).
> The recommendation to provide a separate instance of the AQM for ECN packets
> goes beyond the assumptions of RFC 3168, which assumed that only one
> instance of an AQM will handle both ECN-capable and non-ECN-capable packets.
>
>
> Bob
>
>
> ________________________________________________________________ Bob
> Briscoe,                                                  BT
> _______________________________________________ aqm mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/aqm
>
>

-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Wed, 11 Dec 2013 23:11:16 +0200 (EET)

On Wed, 11 Dec 2013, Dave Taht wrote:

> I wish i had a name for this second "seeking" phase that makes as much 
> sense as "congestion avoidance".
> 
> So asserting that you'll have a 200ms interval on a codled link always 
> is just blatantly incorrect. On first boot, yes. On a busy network, 
> never again.[1]

No no. The queue empties after CoDel overshoots the marking probability 
and then CoDel stops and starts from scratch. And yes, CoDel will always 
overshoot for sure because it _controls_ until the queue is in its
control, i.e., below the threshold. How big the overshoot is, of course 
depends.

What this means in terms of TCP:

The network/or you fq-queue (in case of fq_codel if you so want) won't be 
all that busy according to CoDel once CoDel kindly "coddled" the TCPs, 
that's the whole point of CoDel I'm told :-). This is because the queue 
happens to be "under control" only if TCP backs off to below 5ms + RTT 
level of utilization. ...Now remember the effect of beta here. If the 
network remains more than "5ms busy", CoDel thinks that the queue is not 
in control and keeps shooting again and until eventually the network is no 
longer "busy". The worst case beta * (5ms + RTT) is quite small 
utilization and it takes time for a TCP to recover network busyness.

> 4 in others), before resetting count to 1 and the resulting estimation

True, however, it only means that you'll overshoot more or the time was 
too short to retain the trained count in memory (and in that case CoDel 
forgets it like you admit). Or do you think that the magic number applied 
to count on the recall (was it "-2") works for all traffic?

-- 
 i.

---

From: Dave Taht <[email-masked]>
Date: Wed, 11 Dec 2013 13:56:19 -0800

On Wed, Dec 11, 2013 at 1:11 PM, Ilpo Järvinen
<[email-masked]> wrote:
> and then CoDel stops and starts from scratch.

No it doesn't. Usually. When the queue length drops below 5ms, codel
stops dropping packets,
and the queue does not drain to zero.

There have been several variants of the control law so far, with
varying degrees of success
across different ranges of bandwidths.

http://www.pollere.net/CoDel.html documents some of them.

If you would like to back up your assertion with data taken against
the linux codel variant, the more advanced ns2_codel, or fq_codel, please
do so. Patches are available for linux here for the more advanced stuff,
and please be aware of:

http://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel

One of the biggest problems we've seen with overshoot has been dealing
with TSO and GSO
and GRO offloads, which were not modeled in ns2 - and as of linux 3.12
much of the TSO and GSO
problem were mostly solved (and don't happen in the real world of
routers in the first place, but
only on testbeds not emulating the real world.

Another problem we've seen is scheduling latency in virtualized
environments can exceed
the target. Don't do that.

Hmm… what else.

The original paper contained an error that was corrected shortly after it hit
dead tree land. The as-deployed-in-linux variant has a tendency to
overshoot in some cases,
and has issues at really high levels of flows, but in the fq_codel
universe it is usually fine.

One problem we know exists in the real world is the 5ms target is
unachievable at
very low (<4mbit) bandwidths in which case we have been increasing the
target delay
for that rate, and there is also a patch under test that eliminates
the maxpacket check
that helps in that case as well.

the largest known fq_codel deployment (free.fr) uses a simple formula
to set the target
at those rates. I don't think it's correct but the packet scheduling
portion hides
the issues here.

No, codel did not fully achieve the parameterless goal, and also was
not targeted at the
data center environments where different rules of physics apply in
it's initial release.

Development continues, but has mostly been focused on the *fq_codel variants
which are superior in every benchmark we've tried against all other comers.

> And yes, CoDel will always

I don't understand what you mean by overshoot, it will reduce the
target delay to
less than 5ms.

it can certainly reduce it well below 5ms, but inducing only that much jitter
while still keeping utilization high is a goodness.

I don't have a problem if
people shoot for a higher target rates when using codel alone - pie as
submitted to the lkml has a target of 20ms and a very large estimation
window that I have not yet tested at lower bandwidths.

And nobody uses codel by itself, although given pie's problems I
am thinking of taking a harder thwack at making codel itself better,
abstracting out the hysterisis variable for example.

> utilization and it takes time for a TCP to recover network busyness.

You are trying to explain something to me in RED terms that doesn't
happen in the real world.

packets get smoothed into the "RTT" which serves as inflight storage.

Go take a measurement please. Supplying a packet
capture would help explain whatever you are seeing. tell us
what kernel you are using and follow the guidelines.

> to count on the recall (was it "-2") works for all traffic?

You are not looking at the current as-shipped code.

-- 
Dave Täht

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Thu, 12 Dec 2013 00:51:31 +0200 (EET)

> happen in the real world.

??? This has nothing to with RED. I really don't know where you got that 
one. It's all what TCP is doing, sawtooth etc. (I know Linux uses Cubic 
so no need to point that out).

> packets get smoothed into the "RTT" which serves as inflight storage.

That's exactly the point here. When RTT >> threshold, you take MD beta 
hit, the inflight storage will be affect as well.

> what kernel you are using and follow the guidelines.

That particular text snippet didn't come from any particular test, just 
the algorithms (both TCP and CoDel) running in my head.

> You are not looking at the current as-shipped code.

Heh, it was somewhat unfair to call me on that one considering how many 
different variants of that particular recall have existed ;-). But you're 
right, it's not exactly what would happen with that variant.

-- 
 i.

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 12 Dec 2013 13:53:01 +0000

Dave,

I was careful to say fq_codel, not just codel.
I was assuming the scenario that fq_codel is most 
pitched for; a home gateway or modem.

> > >> 4 in others), before resetting count to 1 and the resulting estimation

For fq_codel, new queues for new flows are coming 
and going all the time, with instances of CoDel 
coming and going as each queue comes and goes...

> > >> later. yes, more work is needed in this area.

An explanation would be useful because I admit I 
couldn't unpick the code enough to understand how 
the learning by one Codel instance in one queue 
can be picked up by another Codel instance in another queue...

Whatever, passing learning from one queue to 
another won't help whenever the whole link goes 
idle. And an access link is idle for the vast 
majority of the time. The average percent 
utilisation of a customer's link /during the peak 
hour/ is in the low single digits (ie 1-3%). The 
link is constantly idle for periods a lot longer 
than 8*interval (about a second) even when a user 
is actively using it, e.g. for browsing, and even 
streaming is only active in bursts (assuming HAS).

> > >> As debussy said: "Music is the *space* between the notes".

So, in the case that fq_codel is most targeted 
for, the algorithm will start from scratch at the 
start of the vast majority of new flows, which 
means it will delay the first drop signal by 
200ms. For TCP flows, this means slow-start will 
tend to cause tail drop and overshoot before 
fq_codel has hardly started to train. Further, 
thin flows constantly re-start so they will 
constantly experiencing this problem when their 
idle periods are outside CoDel's brief caching time.

In the rarer case that a flow is long enough (or 
enough other flows are present) for fq_codel to have trained...

> > >> After that approximation of ideal 
> drop/mark rate is obtained, the algorithm

I believe "consistently exceeded" means "exceeded 
for 'interval'" (100ms default). So for a long 
flow, the signalling delay after crossing 
threshold from below is not 200ms + RTT, but it is:
         100ms + RTT + previous drop delay + a little.
Even if the previous drop delay is tiny, the 
100ms interval still makes it very sluggish. E.g. 
if RTT = 20ms, signalling delay = 120ms = 6 round trips. Am I right?

That doesn't matter for a bulk transfer within 
its own fq_codel queue. But it does matter for a 
thin stream using TCP where the delay of each 
message within the stream matters. And it does matter for a r-t stream.

Further, these long flows are rare. So, given the 
link frequently goes inactive, the learning from 
one will rarely still be cached when another 
comes along. Which is why the rate of increase in 
drop frequency during training is important to 
get right (that's my other thread with Van & 
Kathie, which is still unanswered).

___________
I'm not saying Codel / fq_codel is particularly 
worse than other AQMs in respect of this 
signalling delay. My point was that all AQMs have 
to introduce some signalling delay if they use 
loss as a signal, but not if they use ECN. I'm 
just responding on CoDel / fq_codel in particular 
because that's where Jim took this conversation, which you picked up.

Bob

---

From: Jim Gettys <[email-masked]>
Date: Thu, 12 Dec 2013 14:35:14 -0500

On Wed, Dec 11, 2013 at 2:21 PM, Bob Briscoe <[email-masked]> wrote:

> algorithm apply to different subsets of packets that share the same queue. It
> SHOULD be possible to classify packets into these subsets at least by ECN
> "Certainly, it may be the same instance of an AQ
> 
> M algorithm, rather than different instances, for example."
>
>
>  This is my primary motivation for this wordsmithing - I'm trying allow
> us to move towards zero signalling delays in CoDel, PIE and RED (currently
> signalling delay.

No, again, see Dave's mail, and you are missing the flow scheduling aspect
of this and thinking in terms of a single queue and the usual mark/drop
cases; this is the exact cognitive problem I'm talking about.

The flow scheduling aspect of fq_codel is *more* important than what
mark/drop algorithm decides to signal the TCP's to regulate their servo
systems.  I really wish this algorithm had been called "fs_codel", rather
than "fq_codel", as it is so easy to confuse "fair" with "flow", and
"queue" with "scheduling".

Regulating TCP is absolutely essential to keep TCP "sane" and responsive,
but it isn't the most important part of what is going on.

Here's the case of a new TCP flow on a previously idle link: After the TCP
open handshake, you will have no more than 4 or (if IW10 is active)
packets for that flow in the queue (actually, 3 packets, as presumably at
least one packet is in process of transmission; @ 1Mbps, that first full
size packet takes 13ms).

If another flow starts, it will get preferentially scheduled to the
existing flow(s) until it has also built a queue, at which time it competes
"fairly" with the other flows in the packet scheduling.
This means, in the simple low bandwidth case, that as soon as you start the
second flow, it gets the next possible opportunity to be transmitted in
preference to any flow that has built a queue.

Here's the kicker:

We've just avoid 3 packets worth of "head of line blocking" latency for the
second flow to "do its thing".  @1Mbs, that is 40ms (3*13ms) saved just in
the TCP open, and more in that the first packet from the second flow then
gets scheduled immediately, getting that TCP moving.

Similarly for your voip packet; it saves those 40ms. And your bulk flow
gets its first packet through ASAP as well; for web traffic, that usually
contains the size information or other metadata required for the web
browser to unblock.  And that is for a *single* competing TCP connection.

Now, let's look at today's web: there are many embedded objects in a page.
 Once the base page is downloaded, the web browser (presuming its DNS
lookups are already cached), opens a pile of connections all at once.
 Whether I like it or not, (which I don't, as I tried to prevent it with
pipelining in HTTP/1.1 in the 1990's), you can easily have 5-30 TCP
connections start almost simultaneously. See the connections/page plot on
my web page explaining all this stuff in more detail at:
http://gettys.wordpress.com/2013/07/10/low-latency-requires-smart-queuing-traditional-aqm-is-not-enough/

Without any TCP flow control ever happening, I can easily get 40 (or even
several hundred packets!) arriving at near line rate (the initial window *
number of embedded objects).  Most of these TCP connections *won't ever
even get out of slow start*; the objects are typically pretty small.  I've
observed transient latency from HOL blocking of > 100ms on 50Mbps cable
service, on some web pages.

Nothing we will/can do at the TCP level can help this situation!!! Nothing,
other than in the long term getting the incentives right so that fewer TCP
connections might be preferable to gaming TCP, as the web already has.

That CoDel may take a while to figure out that it should start marking in
the idle case is really pretty irrelevant; the packets have already arrived
and unless we do better packet scheduling, you are fully stuck.  Dave's
trying to also explain that a number of people's understanding of how CoDel
works has been wrong.

But as I say, *the details of the mark/drop algorithm doesn't matter*.
 Having 3, or 100 packets queued in a FIFO queue will already mean you are
screwed for anything low latency at the bandwidths we all care about.

Once a flow drains to zero, it again gets treated as a new flow.
And what we choose to define a "flow" to be is arbitrary, though the code
today does the usual 5tuple.

In fq_codel this creates considerable self-delay for short flows or r-t
> apps, which kill their own latency before they get any loss signal to tell
> them to slow down.
>

Not likely in the common case.  See Dave's comments and also note that the
initial burst of web page load runs the queue up high immediately.

And as I keep saying, and will say again, the flow queuing decisions
avoiding HOL blocking explained are much more important to dealing with the
latency problems.

So go invent better algorithms that CoDel for drop/mark, that can be
applied to the flow queuing parts of the algorithm, and I'm happy.  Running
code please.

> Even for elastic flows, with congestion signals delayed by so much, they
> risk hitting themselves with a huge train of overshoot loss. This would be
> the same for fq_pie, except the number is 100ms + RTT.
>

When fq_pie exists to test, I'll be happy to see.

I keep saying, and I'll say again: while some mark/drop algorithm needs to
exist, flow scheduling is more important to getting good latency....

> not. So it doesn't want to slow down too quickly in response to delay
> growth in case it gets screwed by other traffic. Ie. using delay growth as
> a signal entails considerable signalling delay due to all the uncertainty.
>

Yes, I get that.  I got that a year or more ago. The idea has potential
merit. ECN as it is currently defined is not so useful.

And I wish it would get a different name so we could more easily talk about
the two different things now being called ECN.

I can see having ECN marks on a burst of packets may be helpful in having
the receiver judge things in a highly variable wireless scenario; it may
have additional information about the medium and know that that transient
has gone away, and that it may not be a wise idea to slow the connection at
all.

> generalised it from ECN to Diffserv too (because the implementation would
> be no different).
>

As noted before, Diffserv is still interesting, even though the packet
scheduling in fq_codel (or similar algorithms) makes it much less
necessary. There are two aspects to this:

1) higher priority contention to the medium.  If I have a real VOIP packet,
there are ways I can ask for higher priority access to the medium and
reduce the total number of transmit opportunities my traffic requires (and
that is often the scarcest resource on WiFi or Docsis).
2) any hint I can get helps (at the edge) so I can distinguish those
packets from the way the web has been gaming the network.

Even so, for Diffserv to be safely trusted and honored even in your home,
the end user (who is the network operator in this case), will have be able
to know that a device or application is using it and control whether or not
it's honored. Unless it is under the network operator's control (in this
case, you the home user) Diffserv can/will get gamed to uselessness by
application and device manufacturers. Ergo why Dave and I hack on home
routers again: this level of control is not currently present in these
devices, and must be for Diffserv to be useful.

> I need you to allow the other case into your mind for this conversation.
> The wording is specifically about the case where "different subsets of
> packets ... share the same queue".
>

And the word "queue" in most people's minds implies ordering, and FIFO
behavior.  This is the terminological issue I'm harping on.  It's also why
I think "bufferbloat" is a better term for our situation than "queue
bloat", which you liked and have harped at me about.  Buffers don't have
such an implication of ordering.

So if you talked about a buffer here, rather than a queue, I'd be a lot
happier. At least in my mind, queues are ordered.

> complicated way of doing it.
>
Given simple looks like it could work, why get complicated already?

Because the flow scheduling is such a win.  You can't solve the whole
problem just with mark/drop algorithms and FIFO queues and get reliably to
decent latencies.

Now, whether Fred's document can/should go into anything like that that
detail is far from clear (arguably not).

I just don't want to further the mythology we can get to decent latencies
at the edge of the network to continue with FIFO queues and an AQM
algorithm, and it's clear that many haven't yet internalized that its flow
scheduling combined with a self tuning adaptive mark/drop algorithm that we
must have, but that the flow scheduling makes the biggest difference.

> is a drop. With ECN, you don't have this dilemma. This is the key to rapid

But you also then have to solve the HOL blocking problem, and do so
urgently.  Ergo flow scheduling.... To get to where we need to go, you
have to worry about the order in which each packet is scheduled.

> decisions (e.g. voip on 802.11).
ECN doesn't help a bit with the head of line blocking problem; it actually
will make it worse with FIFO scheduling. ECN means that you can't even get
the packets out of the way.

Which says you'd better be doing more clever scheduling than a FIFO.

If you want ECN to be usable in the way you want it to be at low
bandwidths, you better become a fan of flow scheduling...

> similar signalling delays that other AQMs inflict).

As I said, it's not the big problem we have today at the edge of the
network, and Dave's mail explains your model of CoDel isn't so correct.

It's only very long lived flows where the signalling even matters, and
that's not what we get at the edge of today's network; instead, we get a
mix of a few larger (e.g. video player flows) with the DOS attack that is
web traffic, with some isochronous VOIP and teleconferencing traffic.

> As I thought, we are talking past each other.
>

Yes, in part because I think few have internalized what the web has done to
edge of the network.  It isn't what I had hoped it would be when I was
HTTP/1.1 editor.  Why this outcome occurred is too long a discussion for
this thread.

We need to be able to have a conversation that is not always "Hmm, that's
> sounds like it might be interesting. Can I tell you about fq_codel now?"
>

Running code of other algorithms very welcome. Fq_codel is running code.
 Pie is running code.

Maybe fq_pie will be running code we can test someday.

Even then, I want a low target latency so individual TCP's are kept
responsive, and I need an algorithm that can keep up quickly with the
dynamics of wireless, so most simplistic tests will not be useful (we don't
have such good evaluation tests today).

When it is, and if it is better than fq_codel, I'll be happy.  But the
mark/drop part of the algorithm *isn't* the most important part of the
algorithm. The packet scheduling decisions are....
                               - Jim

>  On Thu, Dec 5, 2013 at 1:13 PM, Bob Briscoe <[email-masked]> wrote: Fred,
> Gorry, all,
> algorithm apply to different subsets of packets that share the same queue. It
> SHOULD be possible to classify packets into these subsets at least by ECN
> {Then a new section to expand on this before the current Section 4.3.} 4.3{New}.
> Independent AQM Instances for ECN and Diffserv
> Briscoe,                                                  BT  _______________________________________________
> aqm mailing list [email-masked] https://www.ietf.org/mailman/listinfo/aqm
>
>
> I can imagine many other possible algorithms that CoDel for the mark/drop
algorithm; we happen to like CoDel atm for two reasons: 1) it is self
adapting to the line rate, and 2) head mark/drop signals the TCP's as soon
as a decision is made rather than possibly being applied much later (such
as random or tail drop).  We welcome and encourage further essays in the
art.

---

From: Dave Taht <[email-masked]>
Date: Thu, 12 Dec 2013 14:11:33 -0800

I do not have time to track this fork of the discussion right now. For
anyone interested in what I'm doing instead, feel free to read:

http://conferences.sigcomm.org/co-next/2013/program/p133.pdf -
microsecond burst latency issues "bullet trains" - which is something
that was just dramatically improved in the paced fq scheduler and new
TSQ code in linux 3.12

http://conferences.sigcomm.org/co-next/2013/program/p121.pdf - on
limitations of network acceleration

http://conferences.sigcomm.org/co-next/2013/program/p357.pdf -

http://cs.ucsb.edu/~laradeek/Secon13.pdf - nifty new wifi scheduling algo

http://conferences.sigcomm.org/co-next/2013/program/p345.pdf

http://conferences.sigcomm.org/co-next/2013/program/p369.pdf

and start a new thread on those. On the bloat list. where the theory
discussions are more apropos.

but quickly...

Bob, I object to your characterization of users links being busy 1-3%
of the time. That's an average. When they are busy, they are very busy
for short periods, typically 2-16 seconds in the case of web traffic,
then idle for minutes. DASH traffic is busy for 2+ seconds every 10 on
a 20mbit link, and so on, for 1.5 hours or so. Etc. In both cases
congestion exists, and in both cases AQM reaction time measured in
200ms or so is still vastly superior to what happens today, and packet
scheduling masks it to a large extent.

Jim, yes, I was trying to establish the groundwork for ensuring
everyone really understood codel-by-itself before talking about
fq_codel. I'm still not sure that's been established. Anyone here care
to calculate the number of drops on two flows going through codel and
fq_codel, starting with iw10, over a 10mbit 2ms RTT link? And when,
approximately, the queue becomes ideal? And how often the fq_codel
"fast queue" gets used in this case? Gold stars for everyone that gets
it right.

Jim, I'd like you to use larger download speeds than a mbit for your
examples. somewhere between 8 and 20mbit seems appropriate. (IMHO iw10
should not be used on the modern internet on sub 10Mbit links.) The
dynamics change significantly as you get more bandwidth than iw10
abuses.

Anyway…

After I got as far as describing fq_codel accurately in this thread,
then I'd hoped to be able to tackle the immediate ECN issue, the value
of randomness in pie, and the effectiveness and need for ECN on the
edge as it is currently defined.

and I figured that then I might have written enough to get closer to an rfc.

and when I started kibitzing on this thread I thought I was talking to
the DCTCP case which I've spent a few months studying up on, and
looking over alternative ideas there, like

http://conferences.sigcomm.org/co-next/2013/program/p49.pdf
http://conferences.sigcomm.org/co-next/2013/program/p151.pdf

Scalable, Optimal Flow Routing in Datacenters via Local Link Balancing

http://www.irt-systemx.fr/wp-content/uploads/2013/12/AINTEC.ppt

Sigh. I'll

On Thu, Dec 12, 2013 at 11:35 AM, Jim Gettys <[email-masked]> wrote:
>> depends on the propagation change characteristics of the medium), or packets
>> can pile up in a big way.
>>
> than "fq_codel", as it is so easy to confuse "fair" with "flow", and "queue"
> with "scheduling".
>
> open handshake, you will have no more than 4 or (if IW10 is active) packets
> for that flow in the queue (actually, 3 packets, as presumably at least one
> packet is in process of transmission; @ 1Mbps, that first full size packet
> takes 13ms).
>
> If another flow starts, it will get preferentially scheduled to the existing
> flow(s) until it has also built a queue, at which time it competes "fairly"
> with the other flows in the packet scheduling.
> Similarly for your voip packet; it saves those 40ms. And your bulk flow gets
> its first packet through ASAP as well; for web traffic, that usually
> contains the size information or other metadata required for the web browser
> to unblock.  And that is for a *single* competing TCP connection.
>
> Once the base page is downloaded, the web browser (presuming its DNS lookups
> are already cached), opens a pile of connections all at once.  Whether I
> like it or not, (which I don't, as I tried to prevent it with pipelining in
> HTTP/1.1 in the 1990's), you can easily have 5-30 TCP connections start
> almost simultaneously. See the connections/page plot on my web page
> explaining all this stuff in more detail at:
> number of embedded objects).  Most of these TCP connections *won't ever even
> get out of slow start*; the objects are typically pretty small.  I've
> So go invent better algorithms that CoDel for drop/mark, that can be applied
> to the flow queuing parts of the algorithm, and I'm happy.  Running code
> please.
>
>
>
>>
>> generalised it from ECN to Diffserv too (because the implementation would be
>> no different).
>
>
> scheduling in fq_codel (or similar algorithms) makes it much less necessary.
> There are two aspects to this:
>
> there are ways I can ask for higher priority access to the medium and reduce
> the total number of transmit opportunities my traffic requires (and that is
> often the scarcest resource on WiFi or Docsis).
> 2) any hint I can get helps (at the edge) so I can distinguish those packets
> from the way the web has been gaming the network.
>
> behavior.  This is the terminological issue I'm harping on.  It's also why I
> think "bufferbloat" is a better term for our situation than "queue bloat",
> which you liked and have harped at me about.  Buffers don't have such an
> implication of ordering.
> I just don't want to further the mythology we can get to decent latencies at
> the edge of the network to continue with FIFO queues and an AQM algorithm,
> and it's clear that many haven't yet internalized that its flow scheduling
> combined with a self tuning adaptive mark/drop algorithm that we must have,
> but that the flow scheduling makes the biggest difference.
>
>>
>>
> urgently.  Ergo flow scheduling.... To get to where we need to go, you have
> to worry about the order in which each packet is scheduled.
>
>>
>>
> If you want ECN to be usable in the way you want it to be at low bandwidths,
> you better become a fan of flow scheduling...
>>
>>
>> more "important" than other packets in terms of the latency cost to users of
>> that service. Each flow has in essence its own queue in this service class,
>> and we're using information from that to help schedule the packets in ways
>> that minimize latency to the user.
>>
>>
> that's not what we get at the edge of today's network; instead, we get a mix
> of a few larger (e.g. video player flows) with the DOS attack that is web
> traffic, with some isochronous VOIP and teleconferencing traffic.
>>
>>
>>
>>
>> packets goes beyond the assumptions of RFC 3168, which assumed that only one
> as a decision is made rather than possibly being applied much later (such as
> random or tail drop).  We welcome and encourage further essays in the art.
>
>

-- 
Dave Täht

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Dec 2013 00:05:18 +0000

Dave,

At 22:11 12/12/2013, Dave Taht wrote:
>of the time. That's an average.

I said it was an average. You're repeating and 
agreeing with what I said, but saying you object to me saying it?

>When they are busy, they are very busy
>a 20mbit link, and so on, for 1.5 hours or so. Etc.

Yes, again, you're agreeing with me.

The mean for a Web session is towards the low end 
of the 2-16 seconds range even now. And as we get 
the other latency-saving advances out there (e.g. 
removing TCP & TLS handshakes, proper pipelining, 
and a faster replacement for slow-start without 
overshoot), then there is potential for Web 
sessions to drop to 1-2 seconds or less, because 
they're usually a long way from being bandwidth limited.

You said fq_codel decays out its memory in 800ms. 
So it will typically have lost all its memory 
when each Web transfer starts and when each DASH transfer re-starts.

So I think you're agreeing that this 200ms 
signalling delay will be predominant?

>In both cases

You're saying it's OK to propose a solution that 
delays signalling congestion for about 10 typical 
CDN RTTs,... because it's better than nothing?

That's rich. I have to say this... You're one of 
the group of protagonists who has persuaded the 
world to embark on a programme of 
/implementation/ updates that will take years, 
and rubbished deploying the AQM that was already 
implemented (RED), even tho it was already much better than nothing too.

Yes, auto-config for line rate is a nice bell to 
add to the bicycle. If we are going to embark on 
new implementations, auto-config for RTT is no less important.

Lack of auto-config for either requires the fixed 
config to be at the slow end of the range. And 
both have a similar range of variability (the RTT 
range is actually wider). So lack of auto-config 
in either case leads to a similar order of 
unnecessary delay. Particularly given traffic is 
predominantly in short sparse bursts.

Bob

> >> depends on the propagation change 
> characteristics of the medium), or packets
> > scheduling in fq_codel (or similar 
> algorithms) makes it much less necessary.
> >> particular flows should be handled 
> (scheduled) in a service class, and as my
>Fixing bufferbloat with cerowrt: 
>http://www.teklibre.com/cerowrt/subscribe.html

---

From: Dave Taht <[email-masked]>
Date: Thu, 12 Dec 2013 16:33:07 -0800

bob, I have had 3% of my braincells free for this conversation for
the last week. I get twitchy whenever I see averages on traffic that
has microbursts - I pointed to one of the papers presented earlier
this week in my last email.

There is a great deal of wonderful work going on to fix the microburst
problems in the ongoing paced tcp work. I'm very encouraged, and also
agree that it is possible to optimize away much of slow start possibly
along the same way you are thinking,

I had hoped that by establishing what exists today, and describing it
properly, and getting everyone on the same page on what exists…

I could finally get to where I could start to make sense on the ecn
and immediate ecn ideas to work towards reducing congestion still
further in the data center, and along the edge.

and these days, I'm mostly working on wifi. If you can make red work
on wifi, I would love to know how. My take on this list was that we
were trying to standardize on some aqm technologies that exist and are
being deployed today. Bus is leaving the station.

On Thu, Dec 12, 2013 at 4:05 PM, Bob Briscoe <[email-masked]> wrote:
> I said it was an average. You're repeating and agreeing with what I said,
> but saying you object to me saying it?
>
>
> The mean for a Web session is towards the low end of the 2-16 seconds range
> even now. And as we get the other latency-saving advances out there (e.g.
> removing TCP & TLS handshakes, proper pipelining, and a faster replacement
> for slow-start without overshoot), then there is potential for Web sessions
> to drop to 1-2 seconds or less, because they're usually a long way from
> being bandwidth limited.
>
> You said fq_codel decays out its memory in 800ms. So it will typically have
> lost all its memory when each Web transfer starts and when each DASH
> transfer re-starts.
>
> So I think you're agreeing that this 200ms signalling delay will be
> predominant?
>
>
>> In both cases
> You're saying it's OK to propose a solution that delays signalling
> congestion for about 10 typical CDN RTTs,... because it's better than
> nothing?
>
> That's rich. I have to say this... You're one of the group of protagonists
> who has persuaded the world to embark on a programme of /implementation/
> updates that will take years, and rubbished deploying the AQM that was
> already implemented (RED), even tho it was already much better than nothing
> too.
>
> Yes, auto-config for line rate is a nice bell to add to the bicycle. If we
> are going to embark on new implementations, auto-config for RTT is no less
> important.
>
> Lack of auto-config for either requires the fixed config to be at the slow
> end of the range. And both have a similar range of variability (the RTT
> range is actually wider). So lack of auto-config in either case leads to a
> similar order of unnecessary delay. Particularly given traffic is
>> >> queue.
>> >> with
>> >> ECN-capable
>> >> Friday
>> >> us
>> >> favorably
>> >> part
>> > packets
>> > existing
>> > gets
>> > browser
>> > lookups
>> > even
>> > applied
>> >> growth
>> >> signal
>> >> would be
>> > reduce
>> > packets
>> >> conversation.
>> >> really
>> >> mapping
>> > have
>> > bandwidths,
>> >> apart...
>> >> avoid
>> > mix
>> >> burned
>> >> we
>> >> much
>> >> marked
>> >> between
>> >> aggregate as
>> >> of
>> >> (or
>> >> example.
>> >> queue.
>> >> only one

-- 
Dave Täht

---

From: Dave Taht <[email-masked]>
Date: Fri, 13 Dec 2013 00:15:18 -0800

For starters, the codel signaling delay from the onset of continuous
over 5ms delay on packets defaults to target 100ms, not 200ms. I don't
know who started saying 200ms but even I started believing it with the
few brain cells I've had to spare of late. 5x a CDN rtt in a world of
30-60k images sounds about right.

Secondly, codel drops/marks from head, not from tail, so the signal
gets back to the sender in 1/2  the real physical RTT after that,
rather than the tail of a queue that may be out of control at that
point. Much faster than pie.

There has been so much misinformation spread of late on these threads.
I'm hoping we're beginning to make a dent in it? I look forward to
making all this clear on the upcoming RFCs. I think I should stop now,
revisit the rest of this thread and see what else can be cleared up
before even beginning to tackle fq_codel after I get caught up on
sleep.

as for your other comments...

I have always said deploy RED and for that matter DRR, SFQ or SQF
where you can. I distinctly remember polling the crowd at the first
uknof I went to and being sad to discover only about 4% of the room
had (4 people).

I DO hold that red is too hard to configure for ordinary mortals, and
that it doesn't work at all on variable bandwidth links like cable, or
wireless, which happen to be the dominant form of end-user link
nowadays.

As for the hysteresis "problem", in practice it doesn't seem to be
much of a problem. things get well under control before a web page
completes.Same goes for my tests against DASH traffic. I have plenty
of plots and traces of this. Many are on the results webpage for
bufferbloat.net.

as for a good default for interval, a good number IS dependent on your
RTT, and without coupling the ingress and output queues, it's
difficult to determine or even auto tune that. Perhaps with connection
tracking or some other form of coupling, one day. The ACC code from
the gargoyle router project is worth looking at.

I am satisfied that fq_codel can be deployed on fixed rate lines
without any tuning on bandwidths ranging from 4mbit to 1gbit, today,
as it stands. I have done hundreds of thousands of tests to prove
that. Optimizations are helpful for the 3 band system that is what
mostly deployed today, such as smaller quantum's on slow asymmetric
links, and a smaller packet limit on low memory routers.

A larger target is working well on sub 4ms links. I think that could
auto tune better.

A lower target and interval seem right for data center use, but I have
yet to get anyone to run my suite of published tests.

A rate limiter is required to compensate for ISP's lousy
dslam/cmts/gpon head end and CPE at least until this code makes it
onto those devices. Long lead times predominate on this sort of
hardware - We have three years to get DOCSIS 3.1 right, as one
example.

These are second order problems that will be fixed over time. Wifi and
wireless remain problematic, but dents in those problems seem imminent
by next year, and many of the problems aren't aqm or packet scheduling
ones.

SO. damn straight, I'm one of the people pushing for deployment,
notably on boxes that are easy to upgrade and fix as we learn more
about what we should be doing. I'm definitely reluctant to hard code
stuff into big iron or hard to replace firmware as yet. But as matt
mathis said at ietf - what we have is such an improvement over what is
in place today, that it is time to deploy. After almost 3 years of
effort I'm happy to have a few million boxes in place to learn more
from. Aren't you?

We just have a couple billion boxes left to fix. Plenty of time to
tweak things as we go along. If you want RED, or ARED, in linux, it's
been fixed now for 2 years to perform as to the spec. Go for it. If
you could create something to automate RED configuration as I have for
the ceroshaper tool in cerowrt, let me know.

Any time someone has debloating code worth working on... I'm willing
to help. I've been helping on pie, and as you know I've been looking
over your DCTCP experiment carefully, finding and fixing bugs, and
moving the code forward to where it can be compared against a modern
kernel and a modern TCP and modern AQM and packet scheduling systems.

-- 
Dave Täht

---

From: Dave Taht <[email-masked]>
Date: Fri, 13 Dec 2013 00:22:51 -0800

On Fri, Dec 13, 2013 at 12:15 AM, Dave Taht <[email-masked]> wrote:
> over 5ms delay on packets defaults to target 100ms, not 200ms.

interval 100ms.

it has been a long week.

> I don't

s/4ms/4mbit/

-- 
Dave Täht

---

From: Daniel Havey <[email-masked]>
Date: Fri, 13 Dec 2013 01:29:45 -0800 (PST)

> hardware - We have three years to get DOCSIS 3.1 right, as one
Yeah, this stinks.  If I rate limit then I will lose the ability to borrow tokens from my neighbors when they are not using them.

...Daniel

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 13 Dec 2013 11:23:56 +0000

Dave,

I'm not arguing that we shouldn't be trying to 
improve things. I was reacting to your response 
that we only need to selectively improve the 
things that your solution improves.

You also seem to have a selective notion of "what 
exists today" that excludes reconfig of stuff 
that's already deployed. Just because you're an 
implementer, doesn't mean the only valid 
solutions have to involve implementation. One of 
the buses that I'm on had already left your station.

However, I'm sure many implementations (of RED) 
exist that cannot be config'd to do immediate 
ECN, or they will need firmware update but not 
hardware etc. And RED doesn't have the 
auto-config for line-rate. So I'm happy that 
other buses are leaving the station.

At this stage, all I'm asking of you is to at 
least make sure that the new implementation you 
are putting out there can at least be config'd to 
do immediate ECN. That requires:
* separation of variables into policy config 
parameters and variables monitoring the queue.
* Then ensuring that multiple instances of the 
algo can be applied to the same queue, such that 
they have different policy config parameters, but 
share the same queue monitoring variables.

________
Sorry, but I'm going to come back to your 2 other 
posts I see in my inbox - I have 3 major pieces 
of work with deadlines for this weekend. It 
sounds like you need some time to work too.

Bob