Subject: [aqm] Last Call: <draft-ietf-aqm-fq-codel-05.txt> (FlowQueue-Codel) to Experimental RFC

From: The IESG <[email-masked]>
Date: Thu, 03 Mar 2016 09:20:22 -0800

The IESG has received a request from the Active Queue Management and
Packet Scheduling WG (aqm) to consider the following document:
- 'FlowQueue-Codel'
  <draft-ietf-aqm-fq-codel-05.txt> as Experimental RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
[email-masked] mailing lists by 2016-03-17. Exceptionally, comments may be
sent to [email-masked] instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract

   This memo presents the FQ-CoDel hybrid packet scheduler/AQM
   algorithm, a powerful tool for fighting bufferbloat and reducing
   latency.

   FQ-CoDel mixes packets from multiple flows and reduces the impact of
   head of line blocking from bursty traffic.  It provides isolation for
   low-rate traffic such as DNS, web, and videoconferencing traffic.  It
   improves utilisation across the networking fabric, especially for
   bidirectional traffic, by keeping queue lengths short; and it can be
   implemented in a memory- and CPU-efficient fashion across a wide
   range of hardware.

The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-aqm-fq-codel/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-aqm-fq-codel/ballot/

No IPR declarations have been submitted directly on this I-D.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 18 Mar 2016 10:35:48 +0000

IESG, authors,

1. Safe?

My main concern is with applicability. In particular, the sentence in 
section 7 on Deployment Status: "We believe it to be a safe default and 
encourage people running Linux to turn it on: ...". and a similar 
sentiment repeated in the conclusions. "and we believe it to be safe to 
turn on by default, as has already happened in a number of Linux 
distributions."

Can one of the authors explain why a solution with the limitations in 
section 6 can still be described as "safe"? Doesn't "safe" mean "no 
unintended side-effects"? For instance, the limitations section says 
FQ_CoDel will schedule all the flows within an IPsec VPN as one entity - 
equivalent to a single microflow outside the VPN. Also, it says FQ_CoDel 
overrides the attempt of a scavenger flow to use less capacity than 
other flows (consequently causing foreground flows to get less capacity 
than intended). Why do the authors feel the need to say that these 
behaviours are safe?

Indeed, these sentences seem rather Orwellian. They assert the current 
group-think as fact, even though it is the opposite of the facts stated 
earlier.

Would it not be correct instead to say that FQ_CoDel has been made the 
default in a number of Linux distributions despite not being safe in 
some circumstances?

2. Default?

If a draft saying "We believe it to be a safe default..." is published 
as an RFC, it means "The IETF/IESG/etc believes..."
Only one solution can be default, so if the IETF says that FQ_CoDel is a 
safe default, and no other AQM RFC makes any claim to being a safe 
default (which they do not at the moment), it could be read as the IETF 
recommending FQ_CoDel for default status and, by implication, other AQMs 
(like PIE, say) are not recommended for default status.

As far as I know, unlike the listed FQ_CoDel limitations, no limitations 
of PIE have been identified. I don't think anyone is claiming that the 
performance of FQ_CoDel is
awesomely better than PIE. May be a bit better, may be a bit worse, 
depending on circumstances, and depending on which you value most out of 
low queuing delay, high utilization, or low loss.

So, if the authors want the IETF to recommend a default AQM on the basis 
of safety (and I agree safety is the most important factor when choosing 
a default), the most likely candidate would be PIE, wouldn't it?  
FQ_CoDel has unintended side-effects, which implies it is not a good 
candidate for default; it should only be configured deliberately by 
those who can live with the side-effects.

I believe these unintended side-effects were the main reason PIE rather 
than FQ_CoDel was defined as the minimum requirement for a DOCSIS 3.1 
cable modem.

For those reading this who haven't been following the AQM WG, I can 
assure you I'm not associated with the authors of PIE (or FQ_CoDel). 
Indeed, I provided an extensive critique of PIE during the WG phase.

3. A Detail
I also have a concern about the way the limitations are written 
(typically, each limitation is stated, followed by a arm-waving 
qualification attempting to create an impression that there is not 
really a limitation). To keep the thread clean, I'll send that in a 
follow-up email.

Indeed, rather than downplay each limitation, it would be more 
appropriate (and it is common IETF practice) to flag applicability 
limitations up front, in the abstract.

Regards

Bob

On 03/03/16 17:20, The IESG wrote:
> sent [email-masked]  instead. In either case, please retain the

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Fri, 18 Mar 2016 13:47:38 +0100

Hi Bob

Thank you for your timely and constructive comments. Please see the
inline responses below.

> section 7 on Deployment Status: "We believe it to be a safe default
> and encourage people running Linux to turn it on: ...". and a similar
> sentiment repeated in the conclusions. "and we believe it to be safe
> to turn on by default, as has already happened in a number of Linux
> distributions."
>
> section 6 can still be described as "safe"?

"We believe it to be a safe default" means that we have not seen any of
the theoretical limitations we have documented in section 6 be a concern
*in practice* in any of the extensive number of deployments FQ-CoDel has
seen already. And that the benefits of turning on FQ-CoDel are
sufficient that nudging people in that direction is a good idea.

> Indeed, these sentences seem rather Orwellian.

I can assure you that we are not attempting to exert "draconian control
by propaganda, surveillance, misinformation, denial of truth, and
manipulation of the past" (quoting
https://en.wikipedia.org/wiki/Orwellian here). But thank you for
implying it :)

> default in a number of Linux distributions despite not being safe in

At the time it was made the default in OpenWrt (several years ago now,
if memory serves me right), there was not a whole lot of real-world
deployment experience, due to the chicken-and-egg problem of not wanting
to change the default before we have gathered more experience. However,
today the situation is quite different, thanks in part to the boldness
of the OpenWrt devs. So no, I do not believe that to be the case any
longer.

> 2. Default?
>
> If a draft saying "We believe it to be a safe default..." is published as an
> RFC, it means "The IETF/IESG/etc believes..."
> Only one solution can be default, so if the IETF says that FQ_CoDel is a safe
> default, and no other AQM RFC makes any claim to being a safe default (which
> they do not at the moment), it could be read as the IETF recommending FQ_CoDel
> for default status and, by implication, other AQMs (like PIE, say) are not
> recommended for default status.

This is certainly not my reading. This is an experimental RFC saying "we
believe it to be safe as a default" not a standards track RFC saying
"this should be the default". This is an important difference; we are
not mandating anything, but rather expressing our honest opinion on
the applicability of FQ-CoDel as a default, should anyone wish to make
it one in their domain.

> As far as I know, unlike the listed FQ_CoDel limitations, no
> limitations of PIE have been identified. I don't think anyone is
> claiming that the performance of FQ_CoDel is awesomely better than
> PIE. May be a bit better, may be a bit worse, depending on
> circumstances, and depending on which you value most out of low
> queuing delay, high utilization, or low loss.

Well, for CoDel and PIE that is certainly true. But FQ-CoDel in many
cases reduces latency under load by an order of magnitude compared to
both of them, while improving throughput.

> So, if the authors want the IETF to recommend a default AQM on the
> basis of safety (and I agree safety is the most important factor when
> choosing a default), the most likely candidate would be PIE, wouldn't
> it? FQ_CoDel has unintended side-effects, which implies it is not a
> good candidate for default; it should only be configured deliberately
> by those who can live with the side-effects.

I'm not sure it would be possible for the AQM group to agree on a
recommendation for a default. But I suppose it might be a good
bikeshedding exercise. And as noted above, this is not what we intend to
do in this case.

> really a limitation). To keep the thread clean, I'll send that in a

It is certainly not our intention to "create an impression that there is
not really a limitation". Rather, we are trying to suggest ways in which
each limitation can be mitigated by people who are concerned about it,
but still want to realise the benefits of deploying FQ-CoDel. Sure, some
of those proposals are not exactly at the "running code" stage, but
dismissing them as arm-waving is hardly fair.

I'll add, as I noted initially, that many of the limitations we have
noted are of a theoretical nature (in the sense that we are not aware of
any deployments where they have caused issue in practice). This does not
make it any less important to document them, of course, and we have been
grateful for the feedback from the working group that the section grew
out of (you yourself were among the people providing this feedback, I
believe). However, this also means that it is difficult to do more than
point out each issue. We can't quantify them, for instance.

If you have concrete suggestions for language that would make things
clearer, do tell (though I suppose that's exactly what you'll do in your
follow-up mail). :)

-Toke

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 21 Mar 2016 18:04:30 +0000

Toke,

Sorry for not yet sending the follow-up. Straight after that email, I 
got roped into becoming a makeshift ambulance driver and then ... long 
story...

Thanks for taking my comments constructively, as intended. Responses 
embedded.

> sufficient that nudging people in that direction is a good idea.
This is perhaps because "we" (ie the people looking) tend to have 
significantly more bandwidth than the majority of Internet users (those 
in the developing world). When you have less bandwidth, long-running 
flows last longer, so they tend to overlap more. Given bloat problems 
are only seen intermittently in the first place [Hohlfield14], the 
average person isn't going to see these limitations very often. But if 
you are a homeworker using a VPN (for instance), you will be dogged by 
these problems all the time.

So the main problem here is with the assumption that the test has to be 
"whether we observe these limitations in practice".

Few people observed problems with NATs at the time they were introduced 
(otherwise they wouldn't have sold successfully). So those arguing 
against them tended to be ignored by mainstream comms engineers. But 
then the "theoretical" limitations started to bite. And we ended up 
having to make do with a subset of the potential of the Internet. Those 
sounding the warning bells could see the potential of the Internet, and 
they could see how NATs would close that off. Those ignoring the warning 
bells believed they were right to only be concerned with the here and now.

My concern is about precluding future desirable developments in 
application behaviour. It will be rare to observe such cases by random 
inspection, they may not appear while using existing applications on 
existing high speed links. But, they will occur very frequently in 
scenarios prone to them. That's often the nature of side-effects.

My concern is particularly about fq technology in the network precluding 
improvements in the quality of regular best efforts service that we can 
expect through changes in applications and transports alone.

When I was arguing against FQ_CoDel (back in 2013 at the latency 
workshop - you were there too), numerous people were saying that 
FQ_CoDel is much more subtle than regular FQ. At which point I quietened 
down, because I trusted enough of those people. However, in the recent 
tests with HAS (criticised at length elsewhere), one thing that can be 
said with certainty was that FQ_CoDel just becomes a regular fq 
scheduler when you have two or more long-running flows that can always 
keep their queues from emptying. Whatever instantaneous rate the 
application tries to run at, FQ overrides it and runs at 1/N of the 
capacity. That is not good for a video coming off a camera at a variable 
information rate. FQ skims off all the peaks, so the VBR codec adapts 
down to the worst-case peak rate, not the worst-case average rate.

> https://en.wikipedia.org/wiki/Orwellian here). But thank you for
Well, stating the limitations in the draft, then denying their truth in 
the conclusions by using the word safe to describe them is classic 
Orwellian Newspeak 
<https://en.wikipedia.org/wiki/Newspeak#To_control_thought>.
> of the OpenWrt devs. So no, I do not believe that to be the case any
The experience that led me to understand this problem was when a bunch 
of colleagues tried to set up a start-up (a few years ago now) to sell a 
range of "equitable quality" video codecs (ie constant quality variable 
bit-rate instead of constant bit-rate variable quality). Then, the first 
ISP they tried to sell to had WFQ in its Broadband remote access 
servers. Even tho this was between users, not flows, when video was the 
dominant traffic, this overrode the benefits of their cool codecs (which 
would have delivered twice as many videos with the same quality over the 
same capacity.

Now, by your test, you will never see the limitations these videos 
suffered. Because they never got developed. Because the developers gave 
up. You can think of FQ_CoDel as nice well-meaning people (the Linux 
community) creating a new middlebox problem.

> both of them, while improving throughput.
OK, I have seen such figures, and it makes sense that FQ will give 
single RTT flows v low latency.

My concern is that of course the IESG will want to sign off an RFC with 
this cool performance, given they read that the limitations are not 
important. Whereas I believe the limitations have been downplayed.
> bikeshedding exercise. And as noted above, this is not what we intend to
If we don't want the IETF (or the AQM WG) to make this call, we should 
make it clear that we are not making this call.

My concern is that, years down the line, when the context has been lost, 
these sentences could be interpreted as making this call.
For comparison, consider how we have been trying to understand what 
RFC2309 (the RED manifesto) intended to say.

> clearer, do tell (though I suppose that's exactly what you'll do in your
See the next email (like I promised before).

Cheers

Bob

[Hohlfeld14] Hohlfeld, O., Pujol, E., Ciucu, F., Feldmann, A. & Barford, 
P., "A QoE Perspective on Sizing Network Buffers," In: Proc. Internet 
Measurement Conf (IMC'14) pp.333-346 ACM (November 2014)

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Dave Taht <[email-masked]>
Date: Mon, 21 Mar 2016 21:41:46 -0700

I don't even know where to start bob. This part of the language has
been in the draft for 2 years, and you are the only person to object
that I can recall.

It's an experimental RFC. By "safe" we mean that deploying it, within
the guidelines, won't break anything to any huge extent to enormous
benefits. "unsafe", for example, would be promoting use of dctcp while
it still responds incorrectly to packet loss.

Verses your decades long quest for better variable rate video, we've
had over a decade of the bufferbloat problem to deal with on all
traffic, particularly along the edge, and even after solutions started
to appear in mid 2012, we haven't made a real dent in what's deployed,
except for the small select group of devs, academics, ISPs, and
manufacturers willing to try something new.  I'd like to imagine
things are shifting to the left side of the green line here, but under
load, most users are still experiencing latency orders of magnitude in
excess of what can be achieved
http://www.dslreports.com/speedtest/results/bufferbloat?up=1

I've been testing the latest generation of wifi APs of late, and the
"best" of them, under load, in a single direction, has over 2 seconds
at the lower rates. Applying any of these algorithms to wifi is
proving hard, and it's where the bottlenecks are shifting to at least
in my world, where the default download speed is hovering at around
75mbit, and wifi starts breaking down long before that is hit.

...

I tore apart that HAS experiment you cited here:
https://lists.bufferbloat.net/pipermail/bloat/2016-February/007198.html
- where I was, at least, happy
to see fq_codel handle the onslought of dctcp traffic, gracefully. (It
makes me nervous to have such tcps loose on the internet where a
configuration mistake might send that at the wrong people. fq_codel,
"safe" - not, perhaps, optimal - in the face of dctcp.)

my key objections to nearly all the experiments on your side are
non-reproducability, no competing traffic (not even bothering to
measure web PLT in
that paper, for example), no competing upload traffic, and no
inclusion of the typical things that are latency sensitive at all
(voip, dns, tcp neg, ssl neg, etc).

with competing download and upload traffic, fq_codel *dramatically*
improves the responsiveness and utilization of the link, for all
traffic. Above 5mbits pretty much the only thing that matters for web
traffic is RTT, the google cite for this is around somewhere.

I tend to weigh low latency for every other form of traffic...
today... over marginal improvements in a contrived video download
scenario someday.

As for pie vs fq_codel, despite making pie widely available (for
example, it's shipped in the ubuntu 15.10 release) and just as easily
configurable in the sqm-scripts as fq_codel and cake are, it's been
very difficult to get users to stick with it for a decent a/b test.
This is perhaps the best comparison of the two I know of:

http://burntchrome.blogspot.com/2014/05/i-dont-like-this-pie-more-bufferbloat.html

The best hope I have for a set of subjective QoE experiences, is
perhaps this summer we'll have real pie and fq_codel capable modems
and be able to do a blind "taste test". The pie that we have to play
with is not enough like the draft to trust the results from. We're
still screwed in that the CMTSes are not fixed and in most of my test
sites, download bufferbloat is experienced more often than upload, in
one by a factor of 6000x1, so we need to compare three things -
default cmts behavior down + pie/fq_codel up. And with the cmts
throttled down (as we do it today) via htb + pie/fq_codel.

My core sadness in the flowqueue codel draft is that we spend way more
time talking about the caveats than the observed benefits.

In the case of torrent traffic, utp (especially) works well for me, on
my workloads. I run through a vpn all day without bothering to do
anything special about it. GF uploads big fat images all day without
ever bothering me. The gamers on campus are generally happy. (The only
prioritization I do is dns traffic).

The HUGE benefit of having the fq-aqm tech in place is how much better
it is than anything that exists in the field. I cannot remember the
last non-wifi buffering event I've had on any streaming media, nor the
last time netflix paused. I can't remember the last time I got annoyed
in a ssh or mosh shell. I hold videoconferences on a high rate
conference bridge (60mbit down/8 up) all the time on a 75mbit link
with "cake" while testing other stuff... but that is perhaps just me
and a rag-tag group of people that have bothered to actually install
and configure the software, and their workloads.

Without more implementations of all these bufferbloat fighting
technologies, made deployable, with more people trying them, we're not
going to get anywhere. There's still billions of boxes left to deploy
and a long tail of deployed gear that will probably last 10 years or
more.

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 23 Mar 2016 19:37:34 +0000

David,

Quick reply for now...

DualQ and DCTCP is a separate issue to this question, because it has 
only just started development.
That's why I only mentioned PIE & FQ_CoDel, which have been maturing for 
some time.

Safe means "without unintended side-effects". The word doesn't alter its 
meaning when the thing in question is really cool in other respects.

If a teleworker is using a VPN for their everyday work from home, and 
every time their kids upload or download an elephant they squeeze all 
the VPN traffic to nearly nothing, that is enough to require a support 
call, and explanation of how to switch FQ_CoDel to a safer (but less 
performant) AQM (such as PIE, say).

This is not an academic argument. There are a few billion people using 
the Internet. Even if 0.01% are hit by one of the listed limitations, 
that's a huge volume of support calls. This is why it is important to 
use the word "safe" precisely. Yes, FQ_CoDel has awesome performance. 
But it is awesome and unsafe in certain circumstances.

If that muddies the marketing message, well then seriously consider 
marketing PIE instead. There is no room for "not invented here" in this 
decision.
Otherwise, FQ_CoDel will get bad press later. Then, after riding the 
hype curve of coolness it will fall over the cliff of disillusionment.

Bob

---

From: Dave Taht <[email-masked]>
Date: Wed, 23 Mar 2016 15:26:31 -0700

Bob:

Look. To me the time for discussing this was on the AQM list, months
ago. I see no reason to drag all these cc's in. There is an awful lot
of context and old business here, and you've said two things that
really ticked me off.

On Wed, Mar 23, 2016 at 12:37 PM, Bob Briscoe <[email-masked]> wrote:
> David,
>
> Quick reply for now...
>
> DualQ and DCTCP is a separate issue to this question, because it has only
> just started development.

This is true and I'm sorry I dragged it into it.

"cake", on my side is pretty far along, and we are making serious
headway on fixing wifi of late. Perhaps I'll be able to give a talk
about the issues in wifi we've almost fixed by ietf berlin. I have
most of a blog written around todays' wonderful dataset.

> That's why I only mentioned PIE & FQ_CoDel, which have been maturing for

I do not know of any substantial deployments of pie as yet. I'm
looking forward to seeing DOCSIS-pie shipped and tested sometime in
the next year or so.

> If a teleworker is using a VPN for their everyday work from home, and every
> time their kids upload or download an elephant they squeeze all the VPN
> traffic to nearly nothing, that is enough to require a support call, and
> explanation of how to switch FQ_CoDel to a safer (but less performant) AQM
> (such as PIE, say).

Your assertion that in this case performance would be squeezed to
nearly nothing is incorrect. In the case of a single upload vs a vpn,
life converges to half for each, rapidly. In pie, less rapidly. In the
overbuffered field of today's non-buffer managed ISP environments,
they converge in 10s of seconds, or more.

In the case of that kid torrenting madly, both pie and fq_codel share
the link in roughly the same proportions as the number of extant
flows, with fq_codel only, having a tighter definition of a flow and
converging faster.

In either case, today, without some form of queue management, that kid
doing an upload is generally going to hose the link entirely and make
dad mad in the first place.

...

You and I share a focus in opposite ways, you'd like to selectively
gain priority (without diffserv) via some means, I'd like to
selectively lose priority (without diffserv). To me your focus is a
game theory fail - if someone wins, someone has to lose, and at in the
latter case... well... see section 6.3 of the draft for the cite
there.

I felt that research was important (which is why it is cited in the
draft, and I would not mind if it made pie's and yours), but at least
in my case I'm pretty sure how to go about getting less priority than
a fq+aqm'd system will want to give you, just haven't got around to
resuming the work... (after proving to my satisifaction that uTP was
mostly doing the right things anyway)... and still... for those that
like knobs, we suggest a 3 tier diffserv system in the draft and cake
has it built in.

And I HATE diffserv.

...

As for considering one technology more "safe" than the other....

We've already shown in detail how badly the single queue aqms respond
to bursts of mixed traffic in particular, and the damage GRO is
causing.

The current state of seeing 2+ seconds of bufferbloat on so many link
types is decidedly *unsafe*. But the internet still seems to keep
working despite all that...

> This is not an academic argument. There are a few billion people using the
> Internet. Even if 0.01% are hit by one of the listed limitations, that's a

I look forward to pleasing the other 99.99 percent and reducing their
support calls.

Hell, I'd settle for just 10 or 20 million more this year, that's
still well behind the growth curve of the internet itself.

Doesn't seeing a billion new devices deploy in the last few years
without sane queue management, bother you?

> huge volume of support calls. This is why it is important to use the word
> "safe" precisely. Yes, FQ_CoDel has awesome performance. But it is awesome
> and unsafe in certain circumstances.

I wish you had raised your objections to this part of the language in
draft long before it hit last call. There are plenty of drafts extant
(ecn benefits being the one that still sticks in my craw) - where I
raised my objections during the process, then gave up.

> marketing PIE instead.

It's just one of several competing experimental RFCs at this point.
Near as I can tell, we're going to ship them all at the same time.
Can't we get some hardware built and let the market decide?

> There is no room for "not invented here" in this
> decision.

I resent this. I have as even-handedly as possible treated every new
advancement in this field with as much scientific integrity as I could
muster... and went and tested the hell out of it, whenever I could.

as for bufferbloat.net's efforts: We've published all the source code,
all the (flent.org) benchmark code, made all the code available widely
for anyone to try for under 50 bucks worth of hardware that can be
reflashed with openwrt (well, 54 dollars on amazon, for the edgerouter
X), and begged interested parties to try *every* bufferbloat-fighting
technology we have.  I jumped all over pie when it came out, helped
polish the code, added ecn support, and got it out there so it could
be tested by as many as possible, as soon as possible.

I thought fq_pie was pretty neat. I'd fiddle with your latest stuff if
you'd just fix dctcp's behaviors vs loss. I love the work on BQL and
on fixing TCPs, like pacing in sch_fq.

My principal interest is in ending bufferbloat in my lifetime via any
means possible. And I did not, and do not, intend to make a career out
of it.

If I, personally, can just get to where a few more pieces of gear can
be bought off the shelf with stuff that has the products of the AQM wg
in it - hopefully including wifi, 3g, and homeplug! - I can go back to
things I consider far, far, far more interesting.

and... Jeebus, it's just one experimental RFC.

> Otherwise, FQ_CoDel will get bad press later. Then, after riding the hype
> curve of coolness it will fall over the cliff of disillusionment.

We'll see. Never in my life have I seen a set of ideas so
enthusiastically adopted by those that have adopted it, with so few
complaints.

The only things that bother me at this point are behaviors below
~2mbit sans tuning, and the ecn support, for which we have research
ongoing in cake that we can easily fold back into fq_codel if we need
to.

---

From: grenville armitage <[email-masked]>
Date: Thu, 24 Mar 2016 14:01:17 +1100

On 03/18/2016 21:35, Bob Briscoe wrote:
> IESG, authors,
>
> 1. Safe?
>
> My main concern is with applicability. In particular, the sentence in section 7 on Deployment Status: "We believe it to be a safe default and encourage people running Linux to turn it on: ...". and a similar sentiment repeated in the conclusions. "and we believe it to be safe to turn on by default, as has already happened in a number of Linux distributions."

At the risk of incurring further wrath, and noting that the IESG did request "final comments on this action" (hence all the CCs), I think there's something to Bob's observation about the word "safe".

What about:

Section 1: "...and we believe it to be safe to turn on by default, ..." -> "...and we believe it to be significantly beneficial to turn on by default, ..."
Section 7: "We believe it to be a safe default and ..." -> "We believe it to be a significantly beneficial default and ..."

(Yes, this is going to be an Experimental RFC. And yes, turning on FQ_CoDel generally results in awesome improvements wrt pfifo. But the two instances of "safe" in draft-ietf-aqm-fq-codel-05.txt do imply to me a wider degree of applicability than is probably warranted at this juncture. I just hadn't noticed until Bob mentioned it.)

cheers,
gja

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Thu, 24 Mar 2016 11:28:09 +0100

grenville armitage <[email-masked]> writes:

> What about:
>
> Section 1: "...and we believe it to be safe to turn on by default, ..." ->
> "...and we believe it to be significantly beneficial to turn on by default, ..."
> Section 7: "We believe it to be a safe default and ..." -> "We believe it to be
> a significantly beneficial default and ..."

Aha! Finally someone is being constructive! Thank you!

> (Yes, this is going to be an Experimental RFC. And yes, turning on FQ_CoDel
> generally results in awesome improvements wrt pfifo. But the two instances of
> "safe" in draft-ietf-aqm-fq-codel-05.txt do imply to me a wider degree of
> applicability than is probably warranted at this juncture. I just hadn't noticed
> until Bob mentioned it.)

Still not sure I agree that having the word 'safe' in there is such a
big deal, but, well, if multiple people think it's an issue that in
itself might be reason enough to change it. And I can live with your
alternative formulation. :)

-Toke

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Thu, 24 Mar 2016 11:32:07 +0100

Dave Cridland <[email-masked]> writes:

> Actually I'd read that as more of a recommendation than merely safe. I
> think by safe, the authors mean that no significant harm has been
> found to occur.

What we meant to say was something along the lines of "You want to turn
this on; it'll do you good, so get on with it! You won't regret it! Now
go fix the next 100 million devices!". The current formulation in the
draft is an attempt to be slightly less colloquial about it... ;)

-Toke

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Thu, 24 Mar 2016 14:01:39 +0100

Dave Cridland <[email-masked]> writes:

> Well, I have to ask why, in this case, it's Experimental and not
> Standards-Track?

Heh. Well, I guess the short answer is "because there wasn't WG
consensus to do that". Basically, the working group decided that all the
algorithms we are describing will be experimental rather than standards
track, at least for now. Because they are queueing algorithms and not
protocols (and so do not have the same interoperability requirements),
this was deemed an acceptable way forward, and a way to get it "out
there" without having to have to agree to push for The One True AQM(tm).

(This is my understanding; I'm sure someone will chime in and correct me
if I'm wrong).

Personally, I would have no problem with this being standards track :)

-Toke

---

From: Dave Cridland <[email-masked]>
Date: Thu, 24 Mar 2016 07:57:31 +0000

section 7 on Deployment Status: "We believe it to be a safe default and
encourage people running Linux to turn it on: ...". and a similar sentiment
repeated in the conclusions. "and we believe it to be safe to turn on by
default, as has already happened in a number of Linux distributions."
>
>
> At the risk of incurring further wrath, and noting that the IESG did
request "final comments on this action" (hence all the CCs), I think
there's something to Bob's observation about the word "safe".
>
> What about:
>
> Section 1: "...and we believe it to be safe to turn on by default, ..."
-> "...and we believe it to be significantly beneficial to turn on by
default, ..."
> Section 7: "We believe it to be a safe default and ..." -> "We believe it
to be a significantly beneficial default and ..."

Actually I'd read that as more of a recommendation than merely safe. I
think by safe, the authors mean that no significant harm has been found to
occur. Simply restating that the protocol is experimental should be enough,
I'd have thought, though if you really want:

Although Experimental, this is believed to do no harm as a default in
practise, and ...

> (Yes, this is going to be an Experimental RFC. And yes, turning on
FQ_CoDel generally results in awesome improvements wrt pfifo. But the two
instances of "safe" in draft-ietf-aqm-fq-codel-05.txt do imply to me a
wider degree of applicability than is probably warranted at this juncture.
I just hadn't noticed until Bob mentioned it.)

---

From: Dave Cridland <[email-masked]>
Date: Thu, 24 Mar 2016 10:56:15 +0000

Please try to explain as if I haven't read your draft and wouldn't
understand it if I did, but you seem to be saying this is an applicability
statement indicating you want wide deployment, but, from RFC 2026:

   (d)  Limited Use:  The TS is considered to be appropriate for use
      only in limited or unique circumstances.  For example, the usage
      of a protocol with the "Experimental" designation should generally
      be limited to those actively involved with the experiment.

If what you're saying is that you do believe this is "ready" for wide
deployment, you should be publishing on the Standards Track, surely?

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 24 Mar 2016 13:44:34 +0000

> there" without having to have to agree to push for The One True AQM(tm).
If this isn't standards track because there's no WG consensus for a single
algorithm (and we'll argue over whether a queueing algorithm is a protocol
or not some other time), then I think this WG document should reflect that
consensus and hold back on the recommendations, then, unless you really
have WG consensus for that position.

If this were an individual submission, it'd be different, but a WG document
must reflect the Working Group as a whole and not just the authors.

Of course, this isn't even my biscuit to dunk, let alone my hill to die on.

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 24 Mar 2016 09:51:07 -0400

> Personally, I would have no problem with this being standards track :)

I am one of the WG chairs and document shepherd.  The AQM charter does 
allow for publication on the Standards Track, but at this point in time 
there did not seem to be a consensus that this was necessary, plus given 
some of the open research questions, it seemed like a prudent choice.  
We can always go stronger and make a standard later on.

I think Bob's concerns here, and the disagreement about what happens in 
reality, make it very obvious that Experimental is the right choice!  
The indications so far are that this has a lot of promise to help, but 
there are questions, and it could benefit from even more experience 
deploying in the wild, and watching what happens.

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Thu, 24 Mar 2016 15:05:20 +0100

Dave Cridland <[email-masked]> writes:

> algorithm (and we'll argue over whether a queueing algorithm is a protocol or
> not some other time), then I think this WG document should reflect that
> consensus and hold back on the recommendations, then, unless you really have WG
> consensus for that position.
>
> If this were an individual submission, it'd be different, but a WG document must
> reflect the Working Group as a whole and not just the authors.

Yes, well, ensuring that it does is what the WG last call and review
process is for, isn't it? Which the draft has been through without
anyone taking issue with it. Not even sure what (if any) the proper
process for handling this is at this time (the tracker lists the status
as "Submitted to IESG for Publication")...?

I explained the reasoning behind the current language in a previous
email. The only proposal for alternative language has been from
Grenville, and as I said I can live with that. However, I'm not terribly
inclined to spend more time editing this until I'm sure that it will
actually put the issue to rest.

-Toke

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 24 Mar 2016 22:08:22 +0200

> On 21 Mar, 2016, at 20:04, Bob Briscoe <[email-masked]> wrote:
> 
> The experience that led me to understand this problem was when a bunch of colleagues tried to set up a start-up (a few years ago now) to sell a range of "equitable quality" video codecs (ie constant quality variable bit-rate instead of constant bit-rate variable quality). Then, the first ISP they tried to sell to had WFQ in its Broadband remote access servers. Even tho this was between users, not flows, when video was the dominant traffic, this overrode the benefits of their cool codecs (which would have delivered twice as many videos with the same quality over the same capacity.

This result makes no sense.

You state that the new codecs “would have delivered twice as many videos with the same quality over the same capacity”, and video “was the dominant traffic”, *and* the network was the bottleneck while running the new codecs.

The logical conclusion must be either that the network was severely under-capacity and was *also* the bottleneck, only twice as hard, under the old codecs; or that there was insufficient buffering at the video clients to cope with temporary shortfalls in link bandwidth; or that demand for videos doubled due to the new codecs providing a step-change in the user experience (which feeds back into the network capacity conclusion).

In short, it was not WFQ that caused the problem.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 25 Mar 2016 19:04:56 +0000

Jonathan,

It does make sense.
Inline...

On 24/03/16 20:08, Jonathan Morton wrote:
> The logical conclusion must be either that the network was severely under-capacity
Nope. The SVLAN buffer (Service VLAN shared by all users on the same 
DSLAM) at the Broadband Network Gateway (BNG) became the bottleneck 
during peak hour, while at other times each user's CVLAN (Customer VLAN) 
at the BNG was the bottleneck. The proposition was to halve the SVLAN 
capacity serving the same CVLANs by exploiting the multiplexing gain of 
equitable quality video... explained below.
> and was *also* the bottleneck, only twice as hard, under the old codecs; or that there was insufficient buffering at the video clients to cope with temporary shortfalls in link bandwidth;
I think you are imagining that the bit-rate of a constant quality video 
varies around a constant mean over the timescale that a client buffer 
can absorb. It doesn't. The guys who developed constant quality video 
analysed a wide range of commercial videos including feature films, 
cartoons, documentaries etc, and found that, at whatever timescale you 
average over, you get a significantly different mean. This is because, 
to get the same quality, complex passages like a scene in a forest in 
the wind or splashing water require much higher bit-rate than simpler 
passages, e.g. a talking head with a fixed background. A passage of 
roughly the same visual complexity can last for many minutes within one 
video before moving on to a passage of completely different complexity.

Also, I hope you are aware of earlier research from around 2003 that 
found that humans judge the quality of a video by the worst quality 
passages, so there's no point increasing the quality if you can't 
maintain it and have to degrade it again. That's where the idea of 
constant quality encoding came from.

The point these researchers made is that the variable bit-rate model of 
video we have all been taught was derived from the media industry's need 
to package videos in constant size media (whether DVDs or TV channels). 
The information rate that the human brain prefers is very different.

A typical (not contrived) example bit-rate trace of constant quality 
video is on slide 20 of a talk I gave for the ICCRG in May 2009, when I 
first found out about this research: 
http://www.bobbriscoe.net/presents/0905iccrg-pfldnet/0905iccrg_briscoe.pdf
As it says, the blue plot is averaged over 3 frames (0.12s) and red over 
192 frames (7.68s). If FQ gave everyone roughly constant bit-rate, you 
can see that even 7s of client buffer would not be able to absorb the 
difference between what they wanted and what they were given.

Constant quality videos multiplex together nicely in a FIFO. The rest of 
slide 20 quantifies the multiplexing gain:
* If you keep it strictly constant quality, you get 25% multiplexing 
gain compared to CBR.
* If all the videos respond to congestion a little (ie when many peaks 
coincide causing loss or ECN), so they all sacrifice the same proportion 
of quality (called equitable quality video), you get over 200% 
multiplexing gain relative to CBR. That's the x2 gain I quoted originally.

Anyway, even if client buffering did absorb the variations, you wouldn't 
want to rely on it. Constant quality video ought to be applicable to 
conversational and interactive video, not just streamed. Then you would 
want to keep client buffers below a few milliseconds.

> or that demand for videos doubled due to the new codecs providing a step-change in the user experience (which feeds back into the network capacity conclusion).
Nope, this was a controlled experiment (see below).
> In short, it was not WFQ that caused the problem.
Once they worked out that the problem might be the WFQ in the Broadband 
Network Gateway, they simulated the network with and without WFQ and 
proved that WFQ was the problem.

References

The papers below describe Equitable Quality VIdeo, but I'm afraid there 
is no published write-up of the problems they encountered with FQ - an 
unfortunate side-effect of the research community's bias against 
publishing negative results.

Mulroy09 is a more practical way of implementing equitable quality 
video, while Crabtree09 is the near-perfect strategy precomputed for 
each video:
[Mulroy09] Mulroy, P., Appleby, S., Nilsson, M. & Crabtree, B., "The Use 
of MulTCP for the Delivery of Equitable Quality Video," In: Proc. Int'l 
Packet Video Wkshp (PV'09) IEEE (May 2009)
[Crabtree09] Crabtree, B., Nilsson, M., Mulroy, P. & Appleby, S., 
"Equitable quality video streaming over DSL," In: Proc. Int'l Packet 
Video Wkshp (PV'09) IEEE (May 2009)

Either can be accessed from:
http://research.microsoft.com/en-us/um/redmond/events/pv2009/default.aspx

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 25 Mar 2016 22:20:17 +0200

> Nope. The SVLAN buffer (Service VLAN shared by all users on the same DSLAM) at the Broadband Network Gateway (BNG) became the bottleneck during peak hour, while at other times each user's CVLAN (Customer VLAN) at the BNG was the bottleneck.

In other words, yes the network *was* congested…

> The proposition was to halve the SVLAN capacity serving the same CVLANs by exploiting the multiplexing gain of equitable quality video... explained below.

…and this is one of the key facts that helps me understand the use-case.

I’d be a lot more sympathetic if, as your original description strongly implied, it was all about getting better quality or more capacity from a given network, rather than attempting to *halve the capacity* of a part of the network that was *already congested* at peak hour.  The latter strategy is wrong-headed, especially from the “customer experience” point of view you espouse.

> A typical (not contrived) example bit-rate trace of constant quality video is on slide 20 of a talk I gave for the ICCRG in May 2009, when I first found out about this research: http://www.bobbriscoe.net/presents/0905iccrg-pfldnet/0905iccrg_briscoe.pdf

I understand CQR versus VBR versus CBR.  Now I also understand that “equitable quality” means that the codec adapts the quality to the available bandwidth - which is not the same as CQR, and is more akin to VBR.  This is the other key fact that was omitted from your previous description.

It’s also implicitly clear that the video sources are under the ISP’s control, hence relatively centralised, otherwise they wouldn’t be able to dictate which codec was used.  In theory they could therefore share information explicitly about network conditions each individual stream experiences.

Instead, it seems information was shared only implicitly, by relying on the specific behaviour of a single queue at each bottleneck, which is to give the same congestion signal (whether that be loss, delay, or ECN) to all flows through it.  WFQ broke that assumption, as would any other flow-isolation scheme which avoided impeding lightweight flows when controlling bulk flows, regardless of the precise definition of a “flow".

But I do have to ask: what is the *qualitative* difference between the action of WFQ to equalise capacity per flow, and the off-peak scenario where each flow is bottlenecked at the CVLANs?  I don’t see it.

 - Jonathan Morton

---

From: Steven Blake <[email-masked]>
Date: Fri, 25 Mar 2016 16:35:40 -0400

[Trimming cc list]

On Fri, 2016-03-25 at 19:04 +0000, Bob Briscoe wrote:
> > > The experience that led me to understand this problem was when a
> > > bunch of colleagues tried to set up a start-up (a few years ago
> > > now) to sell a range of "equitable quality" video codecs (ie
> > > constant quality variable bit-rate instead of constant bit-rate
> > > variable quality). Then, the first ISP they tried to sell to had
> > > WFQ in its Broadband remote access servers. Even tho this was
> > > between users, not flows, when video was the dominant traffic,
> > > this overrode the benefits of their cool codecs (which would have
> > > delivered twice as many videos with the same quality over the
> > > same capacity.
> > You state that the new codecs “would have delivered twice as many
> > videos with the same quality over the same capacity”, and video
> > “was the dominant traffic”, *and* the network was the bottleneck
> > while running the new codecs.
> > 
> > and was *also* the bottleneck, only twice as hard, under the old
> > codecs; or that there was insufficient buffering at the video
> > clients to cope with temporary shortfalls in link bandwidth;

This was all well understood in the 90's, when a ton of VBR video
research was ongoing.

> difference between what they wanted and what they were given.

FQ only gives everyone roughly constant bit-rate when everyone is
backlogged (sending at or above their fair-share rate).  Which
contradicts the idea that the traffic is highly stat-muxable.  

> > or that demand for videos doubled due to the new codecs providing a
> > step-change in the user experience (which feeds back into the
> > network capacity conclusion).

Do you know what (if any) AQM was used with the BRAS WFQ implementation
tested?  WFQ+drop-tail or WFQ+RED - where the queue depths/RED
parameters are set based on the queue fair-share rate - would probably
perform much worse than a delay-based FQ-AQM such as FQ-CoDel, because
in a delay-based FQ-AQM a deep flow queue does not necessarily imply
high delay if that flow queue is currently being served at > fair share
rate (since the traffic is so nicely stat-muxable).

Regards,

// Steve