Subject: Re: [aqm] I-D Action: draft-ietf-aqm-recommendation-04.txt

From: Simon Barber <[email-masked]>
Date: Fri, 08 May 2015 20:42:56 -0700

I have a couple of concerns with the recommendations of this document as 
they stand. Firstly - implementing AQM widely will reduce or even 
possibly completely remove the ability to use delay based congestion 
control in order to provide a low priority or background service. I 
think there should be a recommendation that if you are implementing AQM 
then you should also implement a low priority service using DSCP, e.g. 
CS1. This will enable these low priority applications to continue to 
work in an environment where AQM is increasingly deployed. Unlike DSCPs 
that give higher priority access to the network, a background or low 
priority DSCP is not going to be gamed to get better service!

Secondly, there is a recommendation that AQM be implemented both within 
classes of service, and across all classes of service. This does not 
make sense. If you are implementing AQM across multiple classes of 
service, then you are making marks or drops while ignoring what class 
the data belongs to. This destroys the very unfairness that you wanted 
to achieve by implementing the classes in the first place.

Simon

On 5/14/2014 11:00 AM, [email-masked] wrote:
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>   This draft is a work item of the Active Queue Management and Packet Scheduling Working Group of the IETF.
>
>          Title           : IETF Recommendations Regarding Active Queue Management
>          Authors         : Fred Baker
>                            Godred Fairhurst
> 	Filename        : draft-ietf-aqm-recommendation-04.txt
> 	Pages           : 25
> 	Date            : 2014-05-14
>
> Abstract:
>     This memo presents recommendations to the Internet community
>     concerning measures to improve and preserve Internet performance.  It
>     presents a strong recommendation for testing, standardization, and
>     widespread deployment of active queue management (AQM) in network
>     devices, to improve the performance of today's Internet.  It also
>     urges a concerted effort of research, measurement, and ultimate
>     deployment of AQM mechanisms to protect the Internet from flows that
>     are not sufficiently responsive to congestion notification.
>
>     The note largely repeats the recommendations of RFC 2309, updated
>     after fifteen years of experience and new research.
>
>
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/
>
> There's also a htmlized version available at:
> http://tools.ietf.org/html/draft-ietf-aqm-recommendation-04
>
> A diff from the previous version is available at:
> http://www.ietf.org/rfcdiff?url2=draft-ietf-aqm-recommendation-04
>
>
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

---

From: John Leslie <[email-masked]>
Date: Sat, 9 May 2015 21:58:11 -0400

Simon Barber <[email-masked]> wrote:
> control in order to provide a low priority or background service.

   I agree that if AQM succeeds in reducing delay, that will reduce
the delay variation that "low priority" services depend upon.

   However, that strikes me a a problem that delay-based congestion-
control services will have to deal with regardless of AQM.

   Wouldn't we be better off to figure out how AQMs could signal what
these delay-based services actually care about?

> I think there should be a recommendation that if you are implementing
> AQM then you should also implement a low priority service using DSCP,
> e.g. CS1.

   I don't follow how that could help in practice, except for the case
where the AQM is implemented _very_ near the sender. (DSCP gets lost
pretty quickly at Autonomous System boundaries.)

> This will enable these low priority applications to continue to 
> work in an environment where AQM is increasingly deployed. Unlike
> DSCPs that give higher priority access to the network, a background
> or low priority DSCP is not going to be gamed to get better service!

   (I wish I believed we could get agreement to do this!)

> classes of service, and across all classes of service.

   I'm not finding this in the document: "Quality of Service" is found
in Section 2.1; but that's not "class of service". "Traffic class" is
found in Sections 2.1 and 4.4, neither of which mentions "across all
classes."

   ???

> This does not make sense.

   Agreed.

> If you are implementing AQM across multiple classes of service, then you
> are making marks or drops while ignoring what class the data belongs to.

   Alas, that doesn't make sense either. :^(

> This destroys the very unfairness that you wanted to achieve by
> implementing the classes in the first place.

   That's a funny way to phrase it...

--
John Leslie <[email-masked]>

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 12 May 2015 20:11:39 -0400

> to achieve by implementing the classes in the first place.

Hi Simon, thanks for your comments.

These comments appear to be in response to version -04 of the document,
from around 1 year ago.  The document is currently on version -11, has
past working group last call and IESG evaluation, and is in the RFC
Editor's queue.  I mention this, because it isn't clear to me how
applicable your comments are with regard to the current copy.

The current copy can be found at:
https://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/

The current revision does mention the impact to delay-based end-host
algorithms as an area for future research.

While I agree that in a lot of cases it seems like logically a good
idea to have a DiffServ configuration like you mention, I don't think
we have seen data on this yet in the working group.  Looking into this
could be part of that mentioned future work, though not something I'd
want to see hacked into this document today, so late in its publication
process.

-- 
Wes Eddy
MTI Systems

---

From: Simon Barber <[email-masked]>
Date: Tue, 12 May 2015 21:06:00 -0700

Hi John,

Where would be the best place to see if it would be possible to get 
agreement on a global low priority DSCP?

In the latest draft:

https://tools.ietf.org/html/draft-ietf-aqm-recommendation-11

Top of page 16, line 3 it says "AQM should be applied across the classes 
or flows as well as within each class or flow"

It does also say "AQM mechanisms need to allow combination with other 
mechanisms, such as scheduling, to allow implementation of policies for 
providing fairness between different flows."

But I'm still not happy with the statement that AQM should be applied 
'across the classes'.

Simon

---

From: Simon Barber <[email-masked]>
Date: Tue, 12 May 2015 21:17:45 -0700

Hi Wesley,

Thanks for considering my comments, and apologies for being so late in 
the process - I've only recently been able to put time into this area, 
and I understand it may be too late in the process to hack things in. I 
replied to John with where I'm concerned with the current -11 text.

Re: background / low priority streams. There are other ways to achieve a 
'lower priority', such as changing the AIMD parameters. Does not help if 
FQ is involved though. My concern is that implementing AQM removes a 
capability from the network, so doing so without providing a mechanism 
to support low priority is a negative for certain applications (backups, 
updates - and the impact these have on other applications). Would be 
good for this to be at least common knowledge. Is there any other 
document this could go in?

Simon

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 May 2015 05:01:16 +0000

> On May 12, 2015, at 9:06 PM, Simon Barber <[email-masked]> wrote:
> 
> Where would be the best place to see if it would be possible to get agreement on a global low priority DSCP?

I’d suggest

https://tools.ietf.org/html/rfc4594
4594 Configuration Guidelines for DiffServ Service Classes. J.
     Babiarz, K. Chan, F. Baker. August 2006. (Format: TXT=144044 bytes)
     (Updated by RFC5865) (Status: INFORMATIONAL)

It refers to

   [QBSS]     "QBone Scavenger Service (QBSS) Definition", Internet2
              Technical Report Proposed Service Definition, March 2001.

(http://mgoutell.free.fr/gridftp/QBSS/qbss-definition.txt) and states that

> Within QBone, traffic marked with DSCP 001000 (binary) shall be
> considered in the QBSS class and should be given the service described
> in this document.  Notice that while DSCP values generally have only
> local significance we are assigning global significance to this
> particular codepoint within QBone.  We refer to packets marked
> with DSCP 001000 as being marked with the "QBSS code point”.

That’s where we came up with recommending CS1 (001000) for the traffic class.

I’m pretty sure the latter ultimately resulted in an RFC, but for some reason I’m not finding it. The closest thing I see is

https://tools.ietf.org/html/rfc6297
6297 A Survey of Lower-than-Best-Effort Transport Protocols. M. Welzl,
     D. Ros. June 2011. (Format: TXT=46532 bytes) (Status: INFORMATIONAL)

---

From: [email-masked]
Date: Wed, 13 May 2015 07:42:50 +0100

Form my side, I think Scavenger support is still a topic worth trying to
take forward, and it does compliment AQM.

The recent discussion of  DiffServ interconnection classes and practice in
tsvwg at Dallas also touched on this topic. It is a topic that could
easily be discussed again at a tsvwg meeting, if there was interest.

Gorry

> I’m pretty sure the latter ultimately resulted in an RFC, but for some
> reason I’m not finding it. The closest thing I see is
>

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 13 May 2015 07:33:43 +0000

Found it. BTW, 4594 also refers to it, I just didn’t find it there either.

https://tools.ietf.org/html/rfc3662
3662 A Lower Effort Per-Domain Behavior (PDB) for Differentiated
     Services. R. Bless, K. Nichols, K. Wehrle. December 2003. (Format:
     TXT=39029 bytes) (Status: INFORMATIONAL)

   Either a Class Selector (CS) PHB [RFC2474], an Experimental/Local Use
   (EXP/LU) PHB [RFC2474], or an Assured Forwarding (AF) PHB [RFC2597]
   may be used as the PHB for the LE traffic aggregate.  This document
   does not specify the exact DSCP to use inside a domain, but instead
   specifies the necessary properties of the PHB selected by the DSCP.
   If a CS PHB is used, Class Selector 1 (DSCP=001000) is suggested.

> On May 12, 2015, at 11:42 PM, [email-masked] wrote:
> 
>> Iâ?Td suggest
>>> with DSCP 001000 as being marked with the "QBSS code pointâ?.
>> 
>> 
>> Thatâ?Ts where we came up with recommending CS1 (001000) for the traffic
>> class.
>> 
>> Iâ?Tm pretty sure the latter ultimately resulted in an RFC, but for some
>> reason Iâ?Tm not finding it. The closest thing I see is
>>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 13 May 2015 10:47:22 +0200 (CEST)

> on a global low priority DSCP?

Currently the general assumption among ISPs is that DSCP should be zeroed 
between ISPs unless there is a commercial agreement saying that it 
shouldn't. This is generally accepted (there are NANOG mailing list 
threads on several occasions in the past 5-10 years where this was the 
outcome).

The problem is quite complex if you actually want things to act on this 
DSCP value, as there are devices with default behaviour is 4 queue 802.1p, 
with 1 and 2 (which will match AF1x and AF2x) will have lower priority 
than 0 and 3 (BE and AF3x), and people doing DSCP based forwarding, 
usually does things the other way around.

It might be possible to get the last DSCP bits to map into this, because 
for DSCP-ignorant quipment, this would still be standard BE to something 
only looking at CSx (precedence), but that would be lower than 000000. So 
DSCP 000110 (high drop BE) might work, because it's incremental. Possibly 
DSCP 000010 (low drop BE) might be able to get some agreement because it 
doesn't really cause any problems in the existing networks (most likely) 
and it could be enabled incrementally.

I would suggest bringing this kind of proposal to operator organizations 
and the IETF. It needs to get sold to the ISPs mostly, because in this 
aspect the IETF decision will mostly be empty hand-waving unless the 
operators do something.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Simon Barber <[email-masked]>
Date: Mon, 18 May 2015 08:27:58 -0700

Thank you Mikeal, these are useful observations about the choice of exact 
DSCP value and various potential impacts. I agree that ultimately without 
operator agreement non of this matters. I do think that an important step 
towards garnering that operator agreement is to have the concerns clearly 
elucidated in this group's recommendations.

I found a study of the interaction between low priority background 
techniques, including LEDBAT and AQM.

http://www.enst.fr/~drossi/paper/rossi12conext.pdf

it's conclusion states:

"Shortly, our investigation confirms the negative interference: while AQM 
fixes the bufferbloat, it destroys the relative priority among Cc protocols."

Apparently a significant chunk of bittorrent traffic and Windows updates 
use these techniques to deprioritise their traffic. Widespread adoption of 
AQM will remove their ability to avoid impacting the network at peak times. 
Use of DSCP could be one way to mitigate this problem with AQM, and this 
merits further study.

Simon

Sent with AquaMail for Android
http://www.aqua-mail.com

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 18 May 2015 17:52:42 +0200

Hi,

On 13.05.2015 at 07:01 Fred Baker (fred) wrote:
>> Where would be the best place to see if it would be possible to
>> get agreement on a global low priority DSCP?
> 
> I’d suggest
> 
> https://tools.ietf.org/html/rfc4594 4594 Configuration Guidelines
> for DiffServ Service Classes. J. Babiarz, K. Chan, F. Baker. August
> 2006. (Format: TXT=144044 bytes) (Updated by RFC5865) (Status:
> INFORMATIONAL)
> 
> It refers to
> 

While QBSS may have gotten more attention the original idea is here:
https://tools.ietf.org/html/draft-bless-diffserv-lbe-phb-00
from there it finally went to a PDB definition in RFC 3662 as you
already found out, because the DiffServ chairs didn't want it to be
defined as PHB.

>> considered in the QBSS class and should be given the service
>> described in this document.  Notice that while DSCP values
>> generally have only local significance we are assigning global
>> significance to this particular codepoint within QBone.  We refer
>> to packets marked with DSCP 001000 as being marked with the "QBSS
>> code point”.
> 
> 

CS1 is maybe a problem because originally (rfc 2474) CS1 means better
priority than CS0. At that point in time of RFC3662 the discussion was
to use CS1, because also in 802.1p 1 means "background". However,
this inconsistency makes it now hard to rely on any semantics of DSCP
CS1. IIRC the Diffserv chairs were opposed to spend another DSCP on LE
and therefore proposed to use an existing one. In retrospect, this
seems to have been a wrong decision given the problems of rtcweb and
so on these days.

> I’m pretty sure the latter ultimately resulted in an RFC, but for
> some reason I’m not finding it. The closest thing I see is

Yes, https://tools.ietf.org/html/rfc3662.

Regards,
 Roland

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 18 May 2015 17:52:42 +0200

Hi,

Regards,
 Roland

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 18 May 2015 18:54:22 +0300

> On 18 May, 2015, at 18:27, Simon Barber <[email-masked]> wrote:
> 
> Apparently a significant chunk of bittorrent traffic and Windows updates use these techniques to deprioritise their traffic. Widespread adoption of AQM will remove their ability to avoid impacting the network at peak times. Use of DSCP could be one way to mitigate this problem with AQM, and this merits further study.

I’m working on a comprehensive algorithm (including AQM, FQ and Diffserv support and a shaper in one neat package) which does address this problem, or at least provides a platform for doing so.  Some information here:

	http://www.bufferbloat.net/projects/codel/wiki/Cake

This is working code, albeit still under development.  I’m actively dogfooding it, and I’m not the only one doing so.

The Diffserv layer provides a four-class system by default, corresponding in principle with the 802.1p classes - background, best-effort, video and voice.  It does not inherit the naive mapping from DSCPs to those classes, though - only CS1 (001000) is mapped to the background class.

An important part of the Diffserv support in Cake is that the enhanced priority given to the video and voice classes applies only up to given shares of the overall bandwidth.  If traffic in those classes exceeds that allocated share, deprioritisation occurs.  This ensures that improperly marked traffic cannot starve the link, and attempts to incentivise correct marking.

 - Jonathan Morton

---

From: [email-masked]
Date: Mon, 18 May 2015 17:03:41 +0100

I agree that if AQM is deployed it would be good to revisit this topic
(and note that some methods include a notion of flow-queuing, which also
interacts). Coincidentally this at a time when TSVWG is trying to help
inter-domain diffserv connection...

As I noted at the last IETF TSV WG meeting, this topic has the attention
of  "this" TSV WG Chair - and I think it would be useful to AT LEAST
review where we have reached, although I suggest "what do we do next"
discussions belong on the TSV mailing list.

Feel free to take this topic up there!

Gorry
(TSVWG Co-Chair)

---

From: Dave Taht <[email-masked]>
Date: Mon, 18 May 2015 10:00:58 -0700

> http://www.enst.fr/~drossi/paper/rossi12conext.pdf

That paper was continually extended and revised. (I have had very
little to do with it since the first release.)

http://perso.telecom-paristech.fr/~drossi/paper/rossi14comnet-b.pdf

While it is pretty good... my fav part of that paper is table 3 where
the authors ignore the 7 second delay on the link but otherwise
showing the optimal ratio between real tcp and utp in their testbed.

LEDBAT was probably my first concern and area of research before
entering this project full time. I *knew* we were going to break
ledbat, but the two questions were:

how badly? (ans: pretty badly)
and did it matter? (not that much, compared to saving seconds overall
in induced delay)

> fixes the bufferbloat, it destroys the relative priority among Cc

Yep.

I do wish the paper was updated to account for 4 concepts

0) never got around to trying ns2/ns3 fq_codel or the sqm_scripts against it
1) utp has a lower IW. With the move to IW10 in linux tcp iw10 tcp
knocks utp more out of the way (note that a ton of torrent clients
still use tcp and thus they are getting an "advantage" now by using
iw10 that they shouldn't be). Anyway, most web traffic knocks utp out
of the way handily
2) ledbat when first proposed had a 25ms target for induced delay. I
would not mind that tried again.
3) coupled congestion control (one app, many flows)

> Apparently a significant chunk of bittorrent traffic and Windows updates use
> these techniques to deprioritise their traffic.

So, torrent and ledbat are different things. torrent has LOTS of flows
(worst case 6 active/torrent, (50 or more connected, and switching one
into an active state every 15 seconds)).

ledbat is just a cc algorithm that torrent and some other heavy apps use.

> Widespread adoption of AQM
> will remove their ability to avoid impacting the network at peak times.

No. A single ledbat flow will behave like a single tcp flow.
Widespread adoption of AQM will make it easier for many flows to share
the network with low latency.
I don't see any impact from continued use of ledbat for applications
like updates, backups, etc.

My own recomendation is merely to try torrent today with your aqm or
fq system of choice and see what happens. I did, and stopped worrying
about ledbat.

> Use
> of DSCP could be one way to mitigate this problem with AQM, and this merits
> further study.

while we have long recommended CS1 be set on torrent, it turns out
that a lot of gear actually prioritizes that over BE, still. It helps
on the outbound where you can still control your dscp settings.
Many torrent users have reported just setting their stuff to max
outbound and rate limiting inbound, and observing no real effects on
their link.

-- 
Dave Täht
Open Networking needs **Open Source Hardware**

https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67

---

From: Dave Taht <[email-masked]>
Date: Mon, 18 May 2015 10:18:38 -0700

>         http://www.bufferbloat.net/projects/codel/wiki/Cake

This is partially an outgrowth of some of the ideas and problems I
attempted to discuss at ietf90.

https://www.ietf.org/proceedings/90/slides/slides-90-aqm-6.pdf

Since then various other working groups (like dart) attempted to
answer some of the same questions.

I am pretty convinced (now) that inbound policing on cpe can be
improved to better "fool" dumb upstream rate limiters (like those in
cmtses), but haven't got around to doing the work (it's called
"bobbie"). The biggest problem we have with applying a shaper + fq/aqm
algorithm to inbound traffic on an already be-ing dumbly rate limited
link is that a burst can backup in the upstream cmts and stay backed
up - a rate differential of 90 to 100 takes a long time for an aqm to
bring under control. Analysis of "smoothness" might also help.

When the ratios are 10 or 1000s to 1 and there is only one bottleneck
link, we do better.

> This is working code, albeit still under development.  I’m actively dogfooding it, and I’m not the only one doing so.

Pushing it into openwrt soon, we hope. As it stands cake is a win
across the board on cpu cost and fairness, it does saner things with
ecn, and so on...

We have discussed a few more advanced ideas that are not currently in
cake on the cake mailing list, including better coupling between
flows, more rapid response to overload, etc.

> The Diffserv layer provides a four-class system by default, corresponding in principle with the 802.1p classes - background, best-effort, video and voice.  It does not inherit the naive mapping fro…

I see a ton of traffic remarked to CS1 from comcast. Others may be more lucky.

Since dart I have basically come to the conclusion we need at least
one new diffserv priority class for scavaging traffic.

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 18 May 2015 20:52:51 +0300

> On 18 May, 2015, at 20:18, Dave Taht <[email-masked]> wrote:
> 

Scavenging traffic is, of course, the rationale behind assigning CS1 to the background class (which has lower priority than best-effort) in Cake.  If another DSCP is recommended in future for this purpose, it would be straightforward to assign that to the background class as well.  Something in the 000xxx range might be more compatible with legacy equipment.

I’m therefore disappointed that existing recommendations for practical Diffserv deployment ignore the Low Priority class.

But it’s important to note that assigning traffic to separate queues - whether by FQ or Diffserv - only matters at bottlenecks.  Even if there is legacy equipment on the path that happens to interpret CS1 as “elevated priority”, it will have no effect if that isn’t a bottleneck link.  Likewise, AQM only makes a difference at a bottleneck.

It may be worthwhile to emphasise that deployment of AQM should be prioritised on links which are regularly saturated (on timescales of seconds, not minutes).  Over-provisioned links can usually look after themselves, as well as probably presenting the most difficult technical challenge to implementing good AQM.  Saturated links are usually slow enough (10Gbps and down) for AQM implementation to be a tractable problem, even in software.  The most visible problems are usually at or near the last mile, which so far is 1Gbps and down, with the most important deployment locations involving 100Mbps and under.  In fact, the lower the bandwidth of the link, the more important *and* the easier the implementation and deployment of good AQM.

For most traffic patterns, Diffserv isn’t even necessary.  The combination of AQM and FQ does an excellent job - *unless* there is a saturating application that uses many flows.  In that case, FQ ends up giving that application a large share of the bandwidth, and can starve latency-sensitive applications that use only one flow.  It is that scenario - which is exercised by the likes of BitTorrent - which convinced me to add Diffserv support to Cake.  Then, whether the latency-sensitive application sets a high-priority DSCP or the many-flows saturator sets CS1 (or, preferably, both), the problem is resolved satisfactorily.

 - Jonathan Morton

---

From: David Lang <[email-masked]>
Date: Mon, 18 May 2015 12:25:53 -0700 (PDT)

On Mon, 18 May 2015, Simon Barber wrote:

> these techniques to deprioritise their traffic. Widespread adoption of AQM 
> will remove their ability to avoid impacting the network at peak times. Use 

Does it matter? If such traffic doesn't interfere significantly with what the 
user is trying to do in the forground, it shouldn't matter that it's not being 
deprioritized as much.

The big problem before was that such traffic would cause the buffers to build up 
and slow everything down. With AQM in place, that concern basically disappears 
and the only resulting problem would be that you are maxing out the available 
bandwidth unless you deprioritize such traffic.

But if I'm sitting waiting for Windows to download an update, that's now 
forground traffic that is preventing me from doing anything else. I don't want 
it to be deprioritized behind other people in the house doing other things. I 
want it to get it's fair share of bandwidth.

The vast majority of the time, users are going to have enough bandwidth to be 
able to share it with such update mechansims. Those of us stuck on low bandwidth 
links (I'm currently unable to upgrade beyond 3Mb on my DSL line), are going to 
have to accept that when we have multiple things going on, they are going to 
impact each other because the sum of the bandwidth each wants is higher than the 
available bandwidth.

David Lang

---

From: Dave Taht <[email-masked]>
Date: Mon, 18 May 2015 12:56:08 -0700

On Tue, May 12, 2015 at 9:17 PM, Simon Barber <[email-masked]> wrote:
> Hi Wesley,
>
> Thanks for considering my comments, and apologies for being so late in the
> process - I've only recently been able to put time into this area, and I
> understand it may be too late in the process to hack things in. I replied to
> John with where I'm concerned with the current -11 text.

I am glad you are able to put time in, you have been a long way away.

> 'lower priority', such as changing the AIMD parameters. Does not help if FQ
> is involved though.

There are many ways to do lower priority streams if fq is present.

Simplest:

1) Send
3 packets back to back, timestamped. First packet arrives in an empty
queue, gets sent out immediately, 2nd and third packet are affected by the
total number of flows extant.  (fq_codel) (or in SFQ all are affected by
total number of flows)

keep that to 1/2 OWD (or less) plus fuzz/smoothing and you have a solution
for how much additional load you are willing to add to the network.

2) for coupled congestion control on say 6 flows from one app do the
same sort of bunching and measure,
then drop off when one or more of the flows experiences excessive delay.

In both cases the timestamps would be received differently and in
order via pure aqm or drop tail most of the time.

It is relatively easy to get "low priority" in other words in a fq'd
system. It is harder to get to an optimal bandwidth while still
staying low priority and somewhat hard to figure out if you are being
fq'd in the first place.

> My concern is that implementing AQM removes a capability
> from the network, so doing so without providing a mechanism to support low
> priority is a negative for certain applications (backups, updates - and the
> impact these have on other applications). Would be good for this to be at
> least common knowledge. Is there any other document this could go in?

see dart.

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 21 May 2015 15:33:10 +0000

> On May 18, 2015, at 8:27 AM, Simon Barber <[email-masked]> wrote:
> 
> "Shortly, our investigation confirms the negative interference: while AQM fixes the bufferbloat, it destroys the relative priority among Cc protocols."

I think I would phrase that a little differently.

The concept of rearranging traffic in a queue - prioritizing it, deprioritizing it, making it proceed at some rate, and so on - depends on the system in question making choices. It can only make choices when it has multiple things to choose among. Even if a queue consistently has only two waiting elements, it has the opportunity to make choices. However, it also has less need to - if the objective was to reduce jitter (which is why we prioritize voice-on-IP), a shallow queue already has that effect.

In addition, we are talking about stochastic systems, the kind that Kleinrock studied and wrote about. AQM, LEDBAT, CalTech FAST, and so on each moderate the behavior of a data stream so that the inter-arrival intervals approximate mean observed departure intervals, and manage the arrival rate of traffic such that the math tells us that the queues will be less full. A side-effect of doing so, in the Internet, is that queues occasionally completely empty.

I would say that any technology that automatically reduces mean latency reduces the need to manage mean latency. LEDBAT, Delay-based TCP/SCTP congestion control technologies like CalTech FAST, and various AQM technologies all have that property.

---

From: Dave Taht <[email-masked]>
Date: Thu, 21 May 2015 09:31:24 -0700

>> On May 18, 2015, at 8:27 AM, Simon Barber <[email-masked]> wrote:

I don't think the authors of the paper under discussion are on the aqm
list, cc'd. lastest version:

> I think I would phrase that a little differently.

Heh. The language was less moderate in the first drafts, and I never
got them to put in "and destroying the relative priority between the
CC protocols doesn't matter because aqm reduces overall delays far
below the current ledbat targets". :) The ledbat folk would certainly
like to see delay based cc take over from loss based....

And in all cases, slow start (e.g. web) behaviors relative to the
traffic types has been inadequately documented.

And there are two issues at play here, intertwined. One is reducing
the delays, and the other is reducing the bandwidth share of the lower
priority protocol to a scavenging state. Of these the latter is (now)
more interesting than it used to be.

> The concept of rearranging traffic in a queue - prioritizing it, deprioritizing it, making it proceed at some rate, and so on - depends on the system in question making choices. It can only make ch…

I make a clear distinction between ledbat the single flow cc algo, and
bitorrent the swarming protocol. Originally the ledbat-ers tried for
25ms induced delay, and failed to get there with the underlying OS and
queuing facilities, settling on 100ms. I wouldn't mind seeing newer
things like pacing, spreading, and reduced iws tried for the former,
and that, and coupling for the latter.

> I would say that any technology that automatically reduces mean latency reduces the need to manage mean latency. LEDBAT, Delay-based TCP/SCTP congestion control technologies like CalTech FAST, and…

Agree.

Delay gradient TCP just got a writeup in lwn.net btw. haven't read it yet.

I would like it if there were more work into "delay based ccs in a aqm
or fq" environment, building on the tools in the paper referenced
above.

---

From: Simon Barber <[email-masked]>
Date: Thu, 21 May 2015 22:18:29 -0700

On 5/18/2015 10:00 AM, Dave Taht wrote:
> ledbat, but the two questions were: how badly? (ans: pretty badly) and 
> did it matter? (not that much, compared to saving seconds overall in 
> induced delay)
LEDBAT is about more than just reducing the delay caused by the steam - 
it's also about the bandwidth impact. AQM solves the delay situation, 
but breaks the bandwidth reduction that LEDBAT can achieve today when 
other traffic is present.

> on the outbound where you can still control your dscp settings. Many 
> torrent users have reported just setting their stuff to max outbound 
> and rate limiting inbound, and observing no real effects on their link.

Do you have examples of the gear that prioritizes CS1 over best effort? 
How often have you seen it? Did you see it in places where it would be 
important?

Simon

---

From: Dave Taht <[email-masked]>
Date: Thu, 21 May 2015 22:51:21 -0700

On Thu, May 21, 2015 at 10:18 PM, Simon Barber <[email-masked]> wrote:
>> LEDBAT was probably my first concern and area of research before entering
>> this project full time. I *knew* we were going to break ledbat, but the two
>> questions were: how badly? (ans: pretty badly) and did it matter? (not that
>> much, compared to saving seconds overall in induced delay)
>
> LEDBAT is about more than just reducing the delay caused by the steam - it's
> also about the bandwidth impact. AQM solves the delay situation, but breaks
> the bandwidth reduction that LEDBAT can achieve today when other traffic is
> present.

In pointing out that paper I have to stress that their "good" ledbat result,
(read the text around table 3)

"was look! it's scavenging!"

And mine was:

"with over 7 seconds of inherent delay on the link!"

Revisiting the data sets with reasonable amounts of buffering on the link,
a correctly functioning tcp stack, and a few other variables more under
control would be good... (much as I pushed for dctcp to be looked at
once real patches landed for it)

... as would investigating actual behavior of ledbat on real links with
aqm and fq technologies on them. While I poked into it quite a lot,
I did not do much more rigorous than observe that web traffic
worked a lot better when torrent was present in a fq/aqm'd environment
and that cubic outcompeted it slightly, generally.

There was supposed to be someone else updating the tcp_ledbat
kernel module we used, but that never got fixed, and it is in a dire
need of update since the change to usec from msec and other
major tcp modifications in the linux kernel.

>
>> while we have long recommended CS1 be set on torrent, it turns out that a
>> lot of gear actually prioritizes that over BE, still. It helps on the
>> outbound where you can still control your dscp settings. Many torrent users
>> have reported just setting their stuff to max outbound and rate limiting
>> inbound, and observing no real effects on their link.
>
>
> Do you have examples of the gear that prioritizes CS1 over best effort? How
> often have you seen it? Did you see it in places where it would be
> important?

Yes. a lot. and yes. More details I can do later.

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 22 May 2015 12:39:05 +0200

Hi all,

quick comment on ledbat. The main use case for bitTorrent was huge buffers in home routers (where in fact the configuration could be changed by the user but usually the user does not know what to do and just keeps the default config and complains about bitTorrent disturbing Skype calls). This is the reason why the target value is 100ms which is very large but there are (I guess still) home routers which have even larger buffers. For those cases ledbat will still help the problem because these home routers will not suddenly implement AQM. However, the large target value of 100ms has further the effect that if the bottleneck is somewhere deeper in the network where buffers usually are anyway smaller than 100ms (because the link speed is much higher), ledbat traffic will compete equally with other traffic (of other users), which is more or less intentionally as also the bitTorrent user wants to finish its download at some point.

Mirja

> Am 22.05.2015 um 07:51 schrieb Dave Taht <[email-masked]>:
>

---

From: Simon Barber <[email-masked]>
Date: Sun, 24 May 2015 10:33:13 -0700

I've been looking at all the recommended DSCPs recently. CS1 is 
explicitly mentioned as 'low priority'. The AF1 class might also work 
well for 'background' or 'bulk' traffic. Reading RFC4594 it's 
recommended for high throughput applications, and all the applications 
described don't seem to have any interactive, or very time sensitive 
component. This to me means it's more of a background queue. The code 
points are 10, 12 & 14 (for AF11, AF12 and AF13).

Simon

---

From: Simon Barber <[email-masked]>
Date: Sun, 24 May 2015 11:02:04 -0700

Hi Roland,

My recent attention to DSCP has come from looking at what correct 
mappings to 802.1D (now 802.1Q) would be. I have also run across a 
couple of comments that legacy IP Precedence maps CS1 -> higher priority 
than BE. Do you have any knowledge of how prevalent this interpretation 
would be today, and whether it happens in any place that would be a 
problem? (i.e. are there applications that would generate these values, 
and rely on the behaivour, or routers that mis-prioritize things at 
places that are likely a bottleneck)? I.E. How important is it to 
consider these legacy behaivours today?

Simon

On 5/18/2015 8:52 AM, Bless, Roland (TM) wrote:
> to use CS1, because also in 802.1p 1 means "background". However, this 
> inconsistency makes it now hard to rely on any semantics of DSCP CS1. 
> IIRC the Diffserv chairs were opposed to spend another DSCP on LE and 
> therefore proposed to use an existing one. In retrospect, this seems 
> to have been a wrong decision given the problems of rtcweb and so on 
> these days.

> Regards, Roland

---

From: Simon Barber <[email-masked]>
Date: Sun, 24 May 2015 11:08:11 -0700

Hi Mikael,

I can't find reference to DSCP 000010 or 000110, where are they defined?

I know the title 'assured forwarding' seems to imply better than best 
effort, but I think this is a mistake for AF1 - which seems to be 
recommended for bulk traffic that is not latency sensitive. You can't 
make everything high priority! I believe AF1 according to the list of 
recommended applications, would be better served at less than best 
effort priority - so the 4 queue 1a mapping based on the top 3 bits of 
the TOS byte would be OK. AF2 -> lower than best effort would be wrong 
however.

Simon

On 5/13/2015 1:47 AM, Mikael Abrahamsson wrote:
> Currently the general assumption among ISPs is that DSCP should be 
> zeroed between ISPs unless there is a commercial agreement saying that 
> it shouldn't. This is generally accepted (there are NANOG mailing list 
> The problem is quite complex if you actually want things to act on 
> this DSCP value, as there are devices with default behaviour is 4 
> queue 802.1p, with 1 and 2 (which will match AF1x and AF2x) will have 
> lower priority than 0 and 3 (BE and AF3x), and people doing DSCP based 
> forwarding, usually does things the other way around.
>
> It might be possible to get the last DSCP bits to map into this, 
> because for DSCP-ignorant quipment, this would still be standard BE to 
> something only looking at CSx (precedence), but that would be lower 
> than 000000. So DSCP 000110 (high drop BE) might work, because it's 
> incremental. Possibly DSCP 000010 (low drop BE) might be able to get 
> some agreement because it doesn't really cause any problems in the 
> existing networks (most likely) and it could be enabled incrementally.
>
> I would suggest bringing this kind of proposal to operator 
> organizations and the IETF. It needs to get sold to the ISPs mostly, 
> because in this aspect the IETF decision will mostly be empty 
> hand-waving unless the operators do something.
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 24 May 2015 20:30:56 +0200 (CEST)

> I can't find reference to DSCP 000010 or 000110, where are they defined?

What do you mean? I mapped the drop probability bits to BE and suggested 
this might be used.

> the TOS byte would be OK. AF2 -> lower than best effort would be wrong 

This is already impossible to do in real life, see my notes that AF1 and 
AF2 being lower priority in a default configured 4-queue TOS->.1p L2 
environment.

My suggestions is from what might be incrementally deployable in todays 
real life networks. I don't care about history or existing documents, I 
care about what might actually get used Internet-wide.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 24 May 2015 20:33:30 +0200 (CEST)

> consider these legacy behaivours today?

If ISPs today allowed DSCP marking to get propagated Internet wide and 
didn't change their settings from what they have today, some would treat 
AF1 and AF2 higher than BE, some would treat AF1 lower than BE and AF2 
higher than BE, some would treat AF1 and AF2 lower than BE.

That's why I'm saying AF1 and AF2 for less-than-BE isn't incrementally 
deployable.

---

From: Dave Taht <[email-masked]>
Date: Sun, 24 May 2015 11:36:18 -0700

On Sun, May 24, 2015 at 11:33 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> My recent attention to DSCP has come from looking at what correct mappings
>> to 802.1D (now 802.1Q) would be. I have also run across a couple of comments
>> that legacy IP Precedence maps CS1 -> higher priority than BE. Do you have
>> any knowledge of how prevalent this interpretation would be today, and
>> whether it happens in any place that would be a problem? (i.e. are there
>> applications that would generate these values, and rely on the behaivour, or
>> routers that mis-prioritize things at places that are likely a bottleneck)?
>> I.E. How important is it to consider these legacy behaivours today?
>
>
> didn't change their settings from what they have today, some would treat AF1
> and AF2 higher than BE, some would treat AF1 lower than BE and AF2 higher
> than BE, some would treat AF1 and AF2 lower than BE.
>

Which is why I gave up and suggested that utterly new codepoints for
background and least effort traffic be created.

---

From: Simon Barber <[email-masked]>
Date: Sun, 24 May 2015 11:41:22 -0700

OK - so this is within ISP networks. Could this be avoided by mapping 
the DSCPs on entry and exit of their network? Do you know about CS1 
within ISP networks? Or any impact at the edge?

Simon

On 5/24/2015 11:33 AM, Mikael Abrahamsson wrote:
>> couple of comments that legacy IP Precedence maps CS1 -> higher 
>> priority than BE. Do you have any knowledge of how prevalent this 
>> interpretation would be today, and whether it happens in any place 
>> that would be a problem? (i.e. are there applications that would 
>> generate these values, and rely on the behaivour, or routers that 
>> mis-prioritize things at places that are likely a bottleneck)? I.E. 
>> How important is it to consider these legacy behaivours today?
>
> didn't change their settings from what they have today, some would 
> treat AF1 and AF2 higher than BE, some would treat AF1 lower than BE 
> and AF2 higher than BE, some would treat AF1 and AF2 lower than BE.
>

---

From: Simon Barber <[email-masked]>
Date: Sun, 24 May 2015 11:48:50 -0700

I do like the idea of introducting a new 'low priority' code point, were 
the top 3 bits are 0, so that legacy equipment that makes the wrong 
interpretation (higher priority) treats the traffic as BE. There is a 
mess of different interpretations out there, and the downside would be 
legacy equipment that does interpret CS1 correctly would now treat the 
traffic as BE. I don't know enough about the prevalence of legacy 
equipment to know which might be better.

Simon

On 5/24/2015 11:30 AM, Mikael Abrahamsson wrote:
> What do you mean? I mapped the drop probability bits to BE and 
> suggested this might be used.
>
>> effort priority - so the 4 queue 1a mapping based on the top 3 bits 
>> of the TOS byte would be OK. AF2 -> lower than best effort would be 
>> wrong however.
>
> This is already impossible to do in real life, see my notes that AF1 
> and AF2 being lower priority in a default configured 4-queue TOS->.1p 
> L2 environment.
>
> My suggestions is from what might be incrementally deployable in 
> todays real life networks. I don't care about history or existing 
> documents, I care about what might actually get used Internet-wide.
>

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Mon, 25 May 2015 02:42:44 +0000

> On May 24, 2015, at 11:02 AM, Simon Barber <[email-masked]> wrote:
> 
> Hi Roland,
> 
> My recent attention to DSCP has come from looking at what correct mappings to 802.1D (now 802.1Q) would be. I have also run across a couple of comments that legacy IP Precedence maps CS1 -> higher priority than BE. Do you have any knowledge of how prevalent this interpretation would be today, and whether it happens in any place that would be a problem? (i.e. are there applications that would generate these values, and rely on the behaivour, or routers that mis-prioritize things at places that are likely a bottleneck)? I.E. How important is it to consider these legacy behaivours today?
> 
> Simon

What, specifically, does this have to do with the subject line? Could I trouble you to change the subject line, or at least reference a current internet draft?

>> CS1 is maybe a problem because originally (rfc 2474) CS1 means better priority than CS0. At that point in time of RFC3662 the discussion was to use CS1, because also in 802.1p 1 means "background". However, this inconsistency makes it now hard to rely on any semantics of DSCP CS1. IIRC the Diffserv chairs were opposed to spend another DSCP on LE and therefore proposed to use an existing one. In retrospect, this seems to have been a wrong decision given the problems of rtcweb and so on these days.
> 
>> Regards, Roland
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 25 May 2015 06:31:50 +0200 (CEST)

> within ISP networks? Or any impact at the edge?

I don't understand the difference between AF1 and CS1. Please elaborate.

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 25 May 2015 08:39:16 +0300

> On 25 May, 2015, at 07:31, Mikael Abrahamsson <[email-masked]> wrote:
> 

> I can't find reference to DSCP 000010 or 000110, where are they defined?

It’s important to remember that the DSCP field and, to a lesser extent, the ECN bits (which both reside in the IP-header TOS byte) overlap with the legacy Precedence and Type-Of-Service fields, and are only partly backwards-compatible with them.  Probably a lot of network (and network hardware) engineers still think in terms of the legacy definitions, even though DSCP and ECN together have been around for almost 15 years now.

Precedence and TOS used to be orthogonal concepts.  Precedence meant “priority”, and TOS was meant to influence routing decisions; the example given was satellite (high delay, high bandwidth) versus leased-line (low delay, low bandwidth), which just goes to show how assumptions about technology go obsolete over time.

CS0, CS1, CS2 … CS7 are DSCPs which correspond to the legacy Precedence field set to the given value, and the TOS field (not to be confused with the TOS *byte*) set to zero.  The legacy interpretation of these is that CS7 is highest priority, and CS0 is lowest priority; each was given military-style names which are mostly forgotten (and better so).  It is an unfortunate but understandable historical artefact that the default setting was the lowest priority.

AF1x and CS1 both set the legacy Precedence field to 001, which is all that many hardware priority systems look at.  However, a DSCP is a homogeneous field with all six bits significant, and they differ in the fourth and fifth bits (001xx0).  According to my cheat-sheet compiled from RFC4594, AF1x corresponds to “high throughput data” with three different “drop priority” levels (in common with the other three Assured Forwarding groups: AF2x, AF3x, AF4x).

DSCP 000010 has the shorthand TOS2, because it corresponds to the legacy Precedence 000, TOS 010.  It was the legacy method of specifying “high throughput”.

TOS1 (000001) meant “maximum reliability”, which doesn’t really correspond to any modern DSCP, and TOS4 (000100) meant “minimum delay”, which best corresponds to EF (101110).

AFAIK, there is no standardised meaning for DSCP 000110.  It also has ambiguous meaning in the legacy scheme, and is unlikely to have been used by well-behaved applications.  I would consider it an available spare.

All DSCPs matching xxxx11 are reserved for local and experimental use.  They do escape their environments on occasion.

There was also a fourth TOS bit which was meant to specify “minimum cost”.  However, I understand it was specified rather late in the Precedence/TOS lifecycle, and in any case it overlaps with the ECN bits.  It sure would have been useful to still have that available - but having ECN is worth the trouble.

To me, it’s somewhat ambiguous whether “high throughput” should mean “low priority”.  It strikes me that many application developers would seize on it as meaning “faster”.  If they found that it actually made things slower (in a magical future where functional Diffserv support was ubiquitous), they might give up on Diffserv entirely as a bad job.

Nevertheless, when I worked out an eight-class Diffserv scheme as a modest simplification of the 12 classes described in RFC4594, I classed TOS2 / AF1x “high throughput” as lower than best-effort, but just above CS1 “background”.  This was on the basis that throughput had to be traded off against latency, and Cake’s priority scheme thus gave “higher” classes more priority but less throughput.  In the four-class scheme that’s now the default in Cake, however, I lumped the “high throughput” classes in with “best effort”, with only CS1 being distinct as “background".

Here’s my cheat-sheet, covering the 25 well-known DSCPs:

/*	List of traffic classes in RFC 4594:
		(roughly descending order of contended priority)
		(roughly ascending order of uncontended throughput)

	Network Control (CS6,CS7)      - routing traffic
	Telephony (EF,VA)         - aka. VoIP streams
	Signalling (CS5)               - VoIP setup
	Multimedia Conferencing (AF4x) - aka. video calls
	Realtime Interactive (CS4)     - eg. games
	Multimedia Streaming (AF3x)    - eg. YouTube, NetFlix, Twitch
	Broadcast Video (CS3)
	Low Latency Data (AF2x,TOS4)      - eg. database
	Ops, Admin, Management (CS2,TOS1) - eg. ssh
	Standard Service (CS0 & unrecognised codepoints)
	High Throughput Data (AF1x,TOS2)  - eg. web traffic
	Low Priority Data (CS1)           - eg. BitTorrent

	Total 12 traffic classes.
 */

It’s a mess, isn’t it?

 - Jonathan Morton

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Mon, 25 May 2015 06:12:11 +0000

> On May 24, 2015, at 9:31 PM, Mikael Abrahamsson <[email-masked]> wrote:
> 

RFC 2597: any AF class (there are four rate-based classes, named AFx) has three different quanta (called AFxy) at which drop probability can change. One has a low drop probability if, during a given time interval, one has sent less than some quantum of data. If one has sent more than that but less than a second amount, one has an increased drop probability. If one has sent more than the second but less than a third amount, one has an even more increased drop probability. Crossing that third line, All traffic is presumably dropped.

RFC 3662: assuming one has a priority queuing system, CS1 traffic occupies the lowest priority, and a packet in that queue might be held for at most some time interval or dropped when the queue becomes too deep. The description in the RFC isn't CoDel, but CoDel would work well there.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 25 May 2015 09:32:13 +0200 (CEST)

> RFC 3662: assuming one has a priority queuing system, CS1 traffic occupies the lowest priority, and a packet in that queue might be held for at most some time interval or dropped when the queue bec…

Ok, thanks Fred and Jonathan, I had a hunch this was implied. For me, the 
TOS field bits are the same, that's why I didn't see the distincton.

I am an operational kind of guy. I think I have a decent idea what people 
do out there in the wild. Let me elaborate a bit more than I have done 
before.

Some routers will (default) take the TOS field (3 bits) and put into 
802.1p bit when doing vlan encapsulation. Some switches will by default do 
802.1p = 0x1 and 0x2 to be lower priority than 0x0 and 0x3 when doing 
queuing.

Some other ISPs will have implemented DSCP, and treats AF[123]x as higher 
priority than BE.

A lot of ISPs will zero at least the TOS field (3 bits) at their edge, 
especially because they have business VPN QoS customers and want to assure 
that these have higher priority QoS-wise compared to Internet traffic 
(where DDOS might occur).

So with above, I don't see how we can incrementally implement either 
scheme. That's why I suggested that we create new DSCP class along the 
existing drop probability, which has the benefit of keeping TOS field=0 
(so if the ISP zeros TOS field they might not zero the last 3 bits, and 
the above DSCP ISP won't have to change all their business customer 
configuration), which will also work for the 802.1p TOS using ISP I 
mentioned first, because it'll work the same as default.

So now an ISP that wants to deploy this new "Internet-wide" BE-lowprio, 
BE-highprio, BE-default scheme can do this incrementally regardless of 
what other QoS scheme they're currently using.