Subject: [aqm] RED implementation on Linux 3.14.22

From: Simone Ferlin-Oliveira <[email-masked]>
Date: Mon, 15 Dec 2014 08:51:32 +0100

All,

I am doing some work with shared bottleneck detection that requires
some evaluation with different AQM, in particular, RED. Since I
haven't been following the evolution of the implementation,  I would
like to ask about your experience with the code on Linux 3.14 (and
newer).

*Any* help is appreciated.

Thanks,
Simone

---

From: Jim Gettys <[email-masked]>
Date: Mon, 15 Dec 2014 08:41:45 -0500

> like to ask about your experience with the code on Linux 3.14 (and

​I know that Dave Taht ran into bugs in RED a while back, which I believe
have been fixed for quite a while.

You should always be looking at whether code has been patched in the
current kernel.org system for a module like that you are interested in, so
do a diff between 3.14 and the current Linux system. 3.14 is recent enough
that it may be viable for experiments, for the time being.  Planning to
keep up with Linux development is wise long term in any case, as the rate
of improvement/change in the networking stack is very high at the moment as
draining the bufferbloat swamp and other performance work continues.

Also note that underlying device drivers may have (sometimes lots) of
buffering out of control of the Linux queue discipline.  For Ethernet
devices, you should ensure that that the drivers have BQL support
implemented to minimize this buffering.  Other classes of drivers are more
problematic, and may have lots of buffering to surprise you.

Also be aware that ethernet flow control may move the bottleneck from where
you expect to somewhere else, and that switches in networks also have to be
well understood.  Most consumer switches have this *on* by default, and
mixed 1G/100Mb networks can be particularly "entertaining" in this regard.
Cable modems, unfortunately, typically do not implement flow control, but
some DSL modems do (putting the bottleneck into your router, rather than in
the modem).

> *Any* help is appreciated.

​Hope this helps.
​

>
> Thanks,
> Simone
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: Dave Taht <[email-masked]>
Date: Mon, 15 Dec 2014 07:54:24 -0800

On Mon, Dec 15, 2014 at 5:41 AM, Jim Gettys <[email-masked]> wrote:
> I know that Dave Taht ran into bugs in RED a while back, which I believe

The power of git to answer questions like this is unparalleled. Taking
a look at my current kernel tree and doing a:

git log net/sched/sch_red.c

shows eric fixed 2 bugs in Linux RED in

commit 1ee5fa1e9970a16036e37c7b9d5ce81c778252fc
Author: Eric Dumazet <[email-masked]>
Date:   Thu Dec 1 11:06:34 2011 +0000

   sch_red: fix red_change

...

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1ee5fa1e9970a16036e37c7b9d5ce81c778252fc

ARED was added slightly later, and sfqred (a first attempt at blending
fq + AQM together) shortly after that. (which doesn't do ared (never
made it to mainline as fq_codel landed soon after that)

Advice: Keep track of net-next, do git pulls regularly, and watch "git
log net" for changes.

> You should always be looking at whether code has been patched in the current
> kernel.org system for a module like that you are interested in, so do a diff
> between 3.14 and the current Linux system. 3.14 is recent enough that it may
> be viable for experiments, for the time being.  Planning to keep up with
> Linux development is wise long term in any case, as the rate of
> improvement/change in the networking stack is very high at the moment as

Important changes since 3.14: pie added, DCTCP added, gso/tso offloads
seriously reworked and made gentler, sch_fq's pacing improved. The
last kernel rounds (3.18,3.19) were seriously productive: hystart
improved at longer RTTs, still more TSO/gso improvements, and
xmit_more support was added for some devices. Also support for per
route congestion control settings (primarily targetted at DCTCP) was
just added.

I believe some of the long RTT falloff we saw in toke's paper was due
to hystart issues, as I have been unable to duplicate some of his
results with this upcoming release.

I have basically thrown out all my 3.14 results at this point and am
starting over with the soon-to-stablize 3.19 release. (Well, in fact,
I ended up starting over 3 times in the last 2 months as each of the
new features above landed in the kernel) (but as for red, no changes
except in the underlying TCPs and device drivers)

Relevant commits were:

Hystart change:

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=42eef7a0bb0989cd50d74e673422ff98a0ce4d7b

xmit_more:

http://netoptimizer.blogspot.com/2014/10/unlocked-10gbps-tx-wirespeed-smallest.html
very good lwn article on it http://lwn.net/Articles/615238/

one of several GSO fixes:

commit d649a7a81f3b5bacb1d60abd7529894d8234a666
Author: Eric Dumazet <[email-masked]>
Date:   Thu Nov 13 09:45:22 2014 -0800

    tcp: limit GSO packets to half cwnd

... etc. Do a git log net. :)

preso that convinced systemd to switch to fq_codel:
http://lwn.net/Articles/616241/

> problematic, and may have lots of buffering to surprise you.

+10 (or rather, -10). It's up to 25 devices now. I note that TSO/GSO used to
interact very badly with soft rate limiting (htb), it seems better now.

> some DSL modems do (putting the bottleneck into your router, rather than in

I should probably put red back into my test matrixes. I stopped
benchmarking it and pfifo_fast a long time ago. A netperf-wrapper data
set that predates the hystart fix, testing 3 RTTs:

http://snapon.lab.bufferbloat.net/~d/comprehensive.puck/
or:
http://snapon.lab.bufferbloat.net/~d/comprehensive_puck.tgz

> Hope this helps.

-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

---

From: Dave Taht <[email-masked]>
Date: Mon, 15 Dec 2014 08:17:29 -0800

>>> like to ask about your experience with the code on Linux 3.14 (and

I need to clarify something about "newer". The third parameter in Linux is
for bug fixes only. 3.14 is the major release, a 3.14.22 was 22 bug
fix releases. A -X or 4th parameter, if it exists, is distro specific
changes, which can often, particularly in major distros like redhat or
ubuntu, be quite extensive.

"New features", such as the ones I mentioned in the previous email, generally
do not make it to the bug fix releases, and I don't know if (for
example) the hystart
change or GSO half cwnd change will make it to the -stable tree for
older releases (without checking), as usually only security or crash
critical bugs make it into stable.

I mention this in light of a fairly recent DCTCP paper which used a
pre-bufferbloat-fixes kernel of 3.2.something, discussed (Well, ranted
about slightly, apologies) here.

https://lists.bufferbloat.net/pipermail/bloat/2013-November/001736.html

(I would dearly like to see that paper's experiments revised and
updated in light of that discussion, now that all these other fixes
have landed, and DCTCP is now in mainline linux.)

I try to publish a simple debian kernel build script, and my own patch
set of the codel-related research in progress regularly, somewhere:

http://snapon.lab.bufferbloat.net/~d/codel_patches/

and will probably restart publishing a separate debloat-testing tree
for the upcoming make-wifi-fast effort, as that set of changes is
going to be quite extensive, and buggy, for a while.

-- 
Dave Täht

http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks