Subject: [aqm] TCP BBR paper is now generally available

From: Dave Taht <[email-masked]>
Date: Fri, 2 Dec 2016 07:52:36 -0800

http://queue.acm.org/detail.cfm?id=3022184

-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

---

From: David Collier-Brown <[email-masked]>
Date: Fri, 2 Dec 2016 11:21:59 -0500

> http://queue.acm.org/detail.cfm?id=3022184
Very interesting, but the embedded editorial comments in a different 
voice can be jarring.

--dave

-- 
David Collier-Brown,         | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
[email-masked]           |                      -- Mark Twain

---

From: Aaron Wood <[email-masked]>
Date: Fri, 2 Dec 2016 11:15:41 -0800

This is really fascinating reading.

The following made me stop for a second, though:

"The bucket is typically full at connection startup so BBR learns the
underlying network's BtlBw, but once the bucket empties, all packets sent
faster than the (much lower than BtlBw) bucket fill rate are dropped. BBR
eventually learns this new delivery rate, but the ProbeBW gain cycle
results in continuous moderate losses. To minimize the upstream bandwidth
waste and application latency increase from these losses, we added policer
detection and an explicit policer model to BBR."

So, how is this likely to be playing with our qos_scripts and with cake?

Given we have people from both Google and qos_scripts/cake development
here, do we need to compare some notes on how these interact?  Are there
settings in the HTB setup used by qos_scripts that will make it play more
nicely with BBR (smaller quantums, smaller burst sizes, etc)?

-Aaron

On Fri, Dec 2, 2016 at 7:52 AM, Dave Taht <[email-masked]> wrote:

> _______________________________________________
> Bloat mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/bloat
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 2 Dec 2016 22:32:14 +0200

> On 2 Dec, 2016, at 21:15, Aaron Wood <[email-masked]> wrote:
> 

Cake’s deficit-mode shaper behaves fairly closely like an ideal constant-throughput link, which is what BBR is supposedly designed for.  I haven’t read that far in the paper yet, but it shouldn’t trigger any “bucket detection” algorithms, because it doesn’t have a “bucket”.  It is capable of bursting, but only to the minimum extent required to reconcile required throughput with timer resolution and scheduling latency; I’ve tested it with millisecond timers.

The older schemes involving HTB and HFSC *do* have token-bucket behaviour, with an explicitly configured burst size (this excess traffic will collect in downstream buffers).  However, these are shapers, not policers, so they will start delaying packets (leaving them in child qdiscs) when the bucket is empty, not simply dropping them.

The interaction with AQM-related marking and dropping will be interesting to read, though.  It’s not a-priori obvious how much a shaper-AQM combination looks like a policer.

 - Jonathan Morton

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 2 Dec 2016 17:22:23 -0500

On Fri, Dec 2, 2016 at 3:32 PM, Jonathan Morton <[email-masked]>
wrote:

> Cake’s deficit-mode shaper behaves fairly closely like an ideal
> constant-throughput link, which is what BBR is supposedly designed for.

Great. Yes, that's right: BBR's favorite case is a constant-throughput link
or shaper, since that's the easiest to model.

>   I haven’t read that far in the paper yet, but it shouldn’t trigger any
> “bucket detection” algorithms, because it doesn’t have a “bucket”.  It is
> capable of bursting, but only to the minimum extent required to reconcile
> required throughput with timer resolution and scheduling latency; I’ve
> tested it with millisecond timers.
>

That's also good to hear. If it doesn't have a "bucket" or allow
unsustainable bursts, then it should work well with BBR, and shouldn't
trigger the long-term/policer model.

Of course, if we find important use cases that don't work with BBR, we will
see what we can do to make BBR work well with them.

cheers,
neal

---

From: Eric Dumazet <[email-masked]>
Date: Fri, 02 Dec 2016 15:31:26 -0800

On Fri, 2016-12-02 at 23:40 +0100, Steinar H. Gunderson wrote:
> On Fri, Dec 02, 2016 at 05:22:23PM -0500, Neal Cardwell wrote:
> I have one thing that I _wonder_ if could be BBR's fault: I run backup over
> SSH. (That would be tar + gzip + ssh.) The first full backup after I rolled
> out BBR on the server (the one sending the data) suddenly was very slow
> (~50 Mbit/sec); there was plenty of free I/O, and neither tar nor gzip
> (well, pigz) used a full core. My only remaining explanation would be that
> somehow, BBR didn't deal well with the irregular stream of data coming from
> tar. (A wget between the same machines at the same time gave 6-700 Mbit/sec.)
> 
> I will not really blame BBR here, since I didn't take a tcpdump or have time
> to otherwise debug properly (short of eliminating the other things I already
> mentioned); most likely, it's something else. But if you've ever heard of
> others with similar issues, consider this a second report. :-)
> 
> /* Steinar */

It would be interesting to get the chrono stats for the TCP flow, with
an updated ss/iproute2 command and the kernel patches :

efd90174167530c67a54273fd5d8369c87f9bd32 tcp: export sender limits chronographs to TCP_INFO
b0f71bd3e190df827d25d7f19bf09037567f14b7 tcp: instrument how long TCP is limited by insufficient send buffer
5615f88614a47d2b802e1d14d31b623696109276 tcp: instrument how long TCP is limited by receive window
0f87230d1a6c253681550c6064715d06a32be73d tcp: instrument how long TCP is busy sending
05b055e89121394058c75dc354e9a46e1e765579 tcp: instrument tcp sender limits chronographs

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 3 Dec 2016 08:03:50 -0500

Thanks for the report, Steinar. This is the first report we've had
like this, but it would be interesting to find out what's going on.

Even if you don't have time to apply the patches Eric mentions, it
would be hugely useful if the next time you have a slow transfer like
that you could post a link to a tcpdump packet capture (headers only
is best, say -s 120). Ideally the trace would capture a whole
connection, so we can see the wscale on the SYN exchange.

thanks,
neal

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 12:20:15 -0800

On Sat, 2016-12-03 at 20:13 +0100, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 08:03:50AM -0500, Neal Cardwell wrote:
> I tried reproducing it now. I can't get as far down as 50 Mbit/sec,
> but it stopped around 100 Mbit/sec, still without any clear bottlenecks.
> cubic was just as bad, though.
> 
> I've taken two tcpdumps as requested; I can't reboot this server easily
> right now, unfortunately. They are:
> 
>   http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg
>   http://storage.sesse.net/bbr2.pcap -- wget between same hosts
> 
> /* Steinar */

Hi Steinar

Huge ACK decimation it seems.

11:13:28.762772 IP6 S > C: Flags [.], seq 777672:779100, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 1428
11:13:28.763145 IP6 S > C: Flags [.], seq 779100:783384, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 4284
11:13:28.763190 IP6 S > C: Flags [.], seq 783384:789096, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 5712
11:13:28.763239 IP6 S > C: Flags [.], seq 789096:790524, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 1428
11:13:28.763334 IP6 S > C: Flags [.], seq 790524:791952, ack 3432, win 179, options [nop,nop,TS val 3498278784 ecr 864447699], length 1428
11:13:28.764109 IP6 S > C: Flags [.], seq 791952:794808, ack 3432, win 179, options [nop,nop,TS val 3498278785 ecr 864447699], length 2856
11:13:28.764138 IP6 S > C: Flags [.], seq 794808:800520, ack 3432, win 179, options [nop,nop,TS val 3498278785 ecr 864447699], length 5712
11:13:28.764189 IP6 S > C: Flags [.], seq 800520:804804, ack 3432, win 179, options [nop,nop,TS val 3498278785 ecr 864447699], length 4284
11:13:28.764980 IP6 S > C: Flags [.], seq 804804:806232, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 1428
11:13:28.765034 IP6 S > C: Flags [.], seq 806232:811944, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712
11:13:28.765086 IP6 S > C: Flags [.], seq 811944:817656, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712
11:13:28.765905 IP6 S > C: Flags [.], seq 817656:823368, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712
11:13:28.765956 IP6 S > C: Flags [.], seq 823368:829080, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 5712
11:13:28.766005 IP6 S > C: Flags [.], seq 829080:831936, ack 3432, win 179, options [nop,nop,TS val 3498278787 ecr 864447700], length 2856
11:13:28.766869 IP6 S > C: Flags [.], seq 831936:834792, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 2856
11:13:28.766898 IP6 S > C: Flags [.], seq 834792:840504, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 5712
11:13:28.766947 IP6 S > C: Flags [.], seq 840504:841932, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 1428
11:13:28.766997 IP6 S > C: Flags [.], seq 841932:843360, ack 3460, win 179, options [nop,nop,TS val 3498278789 ecr 864447700], length 1428
11:13:28.767532 IP6 C > S: Flags [.], ack 783384, win 4106, options [nop,nop,TS val 864447710 ecr 3498278784], length 0
11:13:28.767541 IP6 C > S: Flags [.], ack 789096, win 4151, options [nop,nop,TS val 864447710 ecr 3498278784], length 0
11:13:28.767543 IP6 C > S: Flags [.], ack 791952, win 4173, options [nop,nop,TS val 864447710 ecr 3498278784], length 0
11:13:28.767544 IP6 C > S: Flags [.], ack 794808, win 4195, options [nop,nop,TS val 864447710 ecr 3498278785], length 0
11:13:28.767546 IP6 C > S: Flags [.], ack 800520, win 4240, options [nop,nop,TS val 864447710 ecr 3498278785], length 0
11:13:28.767547 IP6 C > S: Flags [.], ack 804804, win 4273, options [nop,nop,TS val 864447710 ecr 3498278785], length 0
11:13:28.767549 IP6 C > S: Flags [.], ack 811944, win 4329, options [nop,nop,TS val 864447710 ecr 3498278787], length 0
11:13:28.767550 IP6 C > S: Flags [.], ack 817656, win 4374, options [nop,nop,TS val 864447710 ecr 3498278787], length 0
11:13:28.767552 IP6 C > S: Flags [.], ack 823368, win 4418, options [nop,nop,TS val 864447711 ecr 3498278787], length 0
11:13:28.767553 IP6 C > S: Flags [.], ack 829080, win 4463, options [nop,nop,TS val 864447711 ecr 3498278787], length 0
11:13:28.767554 IP6 C > S: Flags [.], ack 831936, win 4485, options [nop,nop,TS val 864447711 ecr 3498278787], length 0
11:13:28.767556 IP6 C > S: Flags [.], ack 834792, win 4508, options [nop,nop,TS val 864447711 ecr 3498278789], length 0
11:13:28.767557 IP6 C > S: Flags [.], ack 840504, win 4552, options [nop,nop,TS val 864447711 ecr 3498278789], length 0
11:13:28.767559 IP6 C > S: Flags [.], ack 843360, win 4575, options [nop,nop,TS val 864447711 ecr 3498278789], length 0

Just to be clear,what is the kernel version at the sender ?

Thanks !

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 3 Dec 2016 22:26:24 +0200

> On 3 Dec, 2016, at 22:20, Eric Dumazet <[email-masked]> wrote:
> 

That extract does not show ACK decimation.  It shows either jumbo frames or offload aggregation in a send burst, and ordinary delayed-acks each covering at most two packets received.  Nothing particularly weird or unusual appears to be happening in the network.

 - Jonathan Morton

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 13:07:40 -0800

On Sat, 2016-12-03 at 22:26 +0200, Jonathan Morton wrote:
> That extract does not show ACK decimation.  It shows either jumbo
> frames or offload aggregation in a send burst, and ordinary
> delayed-acks each covering at most two packets received.  Nothing
> particularly weird or unusual appears to be happening in the network.

I do not attend IETF meetings so maybe my words are not exact.

What I meant was that we receive ACKS in bursts, with huge gaps between
them.

Look at tsval/tsecr, and that all ACKS are received in a 20 usec time
window, covering data that was sent in a 5 ms window.

Looks the reverse path is severely congested or a device does ack
filtering.

11:14:05.468643 IP6 C > S: Flags [.], ack 364280308, win 23430, options [nop,nop,TS val 864456877 ecr 3498315454], length 0
11:14:05.468645 IP6 C > S: Flags [.], ack 364288528, win 23407, options [nop,nop,TS val 864456877 ecr 3498315454], length 0
11:14:05.468646 IP6 C > S: Flags [.], ack 364296748, win 23415, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
11:14:05.468648 IP6 C > S: Flags [.], ack 364302460, win 23400, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
11:14:05.468650 IP6 C > S: Flags [.], ack 364304968, win 23430, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
<huge gap>
11:14:05.510870 IP6 C > S: Flags [.], ack 364313188, win 23407, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
11:14:05.510885 IP6 C > S: Flags [.], ack 364321408, win 23415, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
11:14:05.510888 IP6 C > S: Flags [.], ack 364328548, win 23396, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
11:14:05.510890 IP6 C > S: Flags [.], ack 364335340, win 23419, options [nop,nop,TS val 864456877 ecr 3498315455], length 0
11:14:05.510892 IP6 C > S: Flags [.], ack 364337848, win 23430, options [nop,nop,TS val 864456877 ecr 3498315456], length 0
11:14:05.510894 IP6 C > S: Flags [.], ack 364353208, win 23389, options [nop,nop,TS val 864456877 ecr 3498315456], length 0
11:14:05.510895 IP6 C > S: Flags [.], ack 364362508, win 23411, options [nop,nop,TS val 864456877 ecr 3498315456], length 0
11:14:05.510897 IP6 C > S: Flags [.], ack 364370728, win 23415, options [nop,nop,TS val 864456877 ecr 3498315456], length 0
11:14:05.510899 IP6 C > S: Flags [.], ack 364378948, win 23419, options [nop,nop,TS val 864456877 ecr 3498315456], length 0
11:14:05.510900 IP6 C > S: Flags [.], ack 364384660, win 23404, options [nop,nop,TS val 864456877 ecr 3498315457], length 0
11:14:05.510902 IP6 C > S: Flags [.], ack 364387168, win 23430, options [nop,nop,TS val 864456877 ecr 3498315457], length 0
11:14:05.510904 IP6 C > S: Flags [.], ack 364394308, win 23411, options [nop,nop,TS val 864456877 ecr 3498315457], length 0
11:14:05.510905 IP6 C > S: Flags [.], ack 364403608, win 23411, options [nop,nop,TS val 864456877 ecr 3498315457], length 0
11:14:05.510907 IP6 C > S: Flags [.], ack 364411828, win 23419, options [nop,nop,TS val 864456878 ecr 3498315457], length 0

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 3 Dec 2016 23:38:18 +0200

> On 3 Dec, 2016, at 23:07, Eric Dumazet <[email-masked]> wrote:
> 

Nevertheless, there are no acks obviously *missing*.  Decimation would be aggregation of acks such that all but the last (over some interval) is dropped.  Here, they are merely delivered in rapid succession.

It could be that the receiver has poor receive-interrupt latency, or that it is aggregating the reverse send path too heavily.  The variable spacing within the ack burst (in particular, in both examples the gap between first and second is larger than the rest) favours the former explanation.

 - Jonathan Morton

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 13:50:37 -0800

On Sat, 2016-12-03 at 22:34 +0100, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 01:07:40PM -0800, Eric Dumazet wrote:
> Note, the tcpdump is done at the receiver. I don't know if this changes the
> analysis.

If you have access to the receiver, I would be interested to know
NIC/driver used there ?

Thanks

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 14:55:37 -0800

On Sat, 2016-12-03 at 23:13 +0100, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 01:50:37PM -0800, Eric Dumazet wrote:
> root@blackhole:~# lspci  | grep Ethernet
> 01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
> 02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
> 
> root@blackhole:~# lspci -n | grep 01:00.0
> 01:00.0 0200: 8086:10d3
> 
> root@blackhole:~# ls -l /sys/class/net/eth0/device/driver
> lrwxrwxrwx 1 root root 0 des.   3 23:17 /sys/class/net/eth0/device/driver -> ../../../../bus/pci/drivers/e1000e
> 
> root@blackhole:~# uname -a
> Linux blackhole 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
> 
> Do note, both eth0 and eth1 are a bridge, although only one of the cards are
> actually connected to anything (it's just so the remote hands can connect to
> any port and things will come up fine).

Perfect.

Note that starting from linux-4.4, e1000e gets gro_flush_timeout that
would help this precise workload

https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=32b3e08fff60494cd1d281a39b51583edfd2b18f

Maybe you can redo the experiment in ~5 years when distro catches up ;)

Thanks.

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 15:02:41 -0800

> would help this precise workload

Also it appears the sender uses a lot of relatively small segments (8220
bytes at a time), with PSH, so GRO wont be able to help.

I wonder how these PSH are forced.

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 15:09:21 -0800

> I wonder how these PSH are forced.

One possibility would be that the sender is not using fq/pacing,
or some driver is using skb_orphan(), killing TCP Small queue.

-> No pressure on qdisc, auto corking does not trigger,
and application does small writes.

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 15:15:40 -0800

On Sun, 2016-12-04 at 00:03 +0100, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 02:55:37PM -0800, Eric Dumazet wrote:
> I can always find a backport, assuming the IPMI is still working. But it
> really wasn't like this earlier :-) Perhaps something changed on the path,
> unrelated to BBR.

If the tcpdump is taken on receiver, how would you explain these gaps ?
Like the NIC was frozen for 40 ms  !

11:14:06.023821 IP6 C > S: Flags [.], ack 368888520, win 23422, options [nop,nop,TS val 864457007 ecr 3498315976], length 0
11:14:06.023823 IP6 C > S: Flags [.], ack 368891028, win 23430, options [nop,nop,TS val 864457007 ecr 3498315976], length 0
11:14:06.023829 IP6 C > S: Flags [.], ack 368899248, win 23407, options [nop,nop,TS val 864457007 ecr 3498315976], length 0
11:14:06.023834 IP6 C > S: Flags [.], ack 368907468, win 23415, options [nop,nop,TS val 864457007 ecr 3498315976], length 0
11:14:06.023835 IP6 C > S: Flags [.], ack 368915688, win 23392, options [nop,nop,TS val 864457007 ecr 3498315976], length 0
<gap : Not tsval is still 864457007 >
11:14:06.062751 IP6 C > S: Flags [.], ack 368923908, win 23415, options [nop,nop,TS val 864457007 ecr 3498315977], length 0
11:14:06.062767 IP6 C > S: Flags [.], ack 368929620, win 23400, options [nop,nop,TS val 864457007 ecr 3498315977], length 0
11:14:06.062770 IP6 C > S: Flags [.], ack 368932128, win 23430, options [nop,nop,TS val 864457007 ecr 3498315977], length 0
11:14:06.062771 IP6 C > S: Flags [.], ack 368943204, win 23400, options [nop,nop,TS val 864457007 ecr 3498315977], length 0
11:14:06.062773 IP6 C > S: Flags [.], ack 368948568, win 23422, options [nop,nop,TS val 864457008 ecr 3498315977], length 0
11:14:06.062775 IP6 C > S: Flags [.], ack 368956788, win 23400, options [nop,nop,TS val 864457008 ecr 3498315977], length 0
11:14:06.062776 IP6 C > S: Flags [.], ack 368965008, win 23415, options [nop,nop,TS val 864457008 ecr 3498315977], length 0
11:14:06.062778 IP6 C > S: Flags [.], ack 368976084, win 23385, options [nop,nop,TS val 864457008 ecr 3498315977], length 0

---

From: Eric Dumazet <[email-masked]>
Date: Sat, 03 Dec 2016 15:24:28 -0800

> Like the NIC was frozen for 40 ms  !

Wait a minute. If you use fq on the receiver, then maybe your old debian
kernel did not backport :

https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=9878196578286c5ed494778ada01da094377a686

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 3 Dec 2016 22:18:08 -0500

>    http://storage.sesse.net/bbr.pcap -- ssh+tar+gnupg

I agree with Eric that for the ssh+tar+gnupg case the ACK stream seems
like the culprit here. After about 1 second, the ACKs are suddenly
very stretched and very delayed (often more than 100ms). See the
attached screen shots.

I like Eric's theory that the ACKs might be going through fq.
Particularly since the uplink data starts having issues around the
same time as the ACKs for the downlink data.

neal

---

From: Eric Dumazet <[email-masked]>
Date: Sun, 04 Dec 2016 09:13:19 -0800

On Sun, 2016-12-04 at 09:44 +0100, Steinar H. Gunderson wrote:
> On Sat, Dec 03, 2016 at 03:24:28PM -0800, Eric Dumazet wrote:
> I checked, and it does not seem that the patch is in the backport, indeed.
> I suppose I can try just turning off fq on the receiver?

You could turn off pacing , and keep fq.

tc qdisc change dev eth0 root fq nopacing

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 6 Dec 2016 16:31:05 -0500

On Tue, Dec 6, 2016 at 12:20 PM, Steinar H. Gunderson <
[email-masked]> wrote:

> > https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/
> commit/?id=9878196578286c5ed494778ada01da094377a686
>
> I upgraded to 4.7.0 (newest backport available). I can get up to ~45
> MB/sec,
> but it seems to hover more around ~22 MB/sec in this test:
>
>   http://storage.sesse.net/bbr-4.7.0.pcap

Thanks for the report, Steinar. Can you please clarify whether the BBR
behavior you are seeing is a regression vs CUBIC's behavior, or is just
mysterious?

It's hard to tell from a receiver-side trace, but this looks to me like a
send buffer limitation. The RTT looks like about 50ms, and the bandwidth is
a little over 500 Mbps, so the BDP is a little over 3 Mbytes. Looks like
most RTTs have a flight of about 2 MBytes of data, followed by a silence
suggesting perhaps the sender ran out of buffered data to send. (Screen
shot attached.)

What are your net.core.wmem_max and net.ipv4.tcp_wmem settings on the
server sending the data?

What happens if you try a bigger wmem cap, like 16 MBytes:

  sysctl -w net.core.wmem_max=16777216 net.ipv4.tcp_wmem='4096 16384
16777216'

If you happen to have access, it would be great to get a sender-side
tcpdump trace for both BBR and CUBIC.

Thanks for all your test reports!

cheers,
neal

---

From: Alan Jenkins <[email-masked]>
Date: Wed, 7 Dec 2016 16:28:15 +0000

>>> I have one thing that I _wonder_ if could be BBR's fault: I run
 >>> backup over SSH. (That would be tar + gzip + ssh.) The first full
 >>> backup after I rolled out BBR on the server (the one sending the
 >>> data) suddenly was very slow (~50 Mbit/sec); there was plenty of
 >>> free I/O, and neither tar nor gzip (well, pigz) used a full core.
 >>> My only remaining explanation would be that somehow, BBR didn't
 >>> deal well with the irregular stream of data coming from tar. (A
 >>> wget between the same machines at the same time gave 6-700
 >>> Mbit/sec.)

 >>  would be hugely useful if the next time you have a slow transfer
 >> like that you could post a link to a tcpdump packet capture
 >> (headers only is best, say -s 120). Ideally the trace would
 >> capture a whole connection, so we can see the wscale on the SYN
 >> exchange.
 >
 > but it stopped around 100 Mbit/sec, still without any clear
 > bottlenecks. cubic was just as bad, though.
 >
 > I've taken two tcpdumps as requested; I can't reboot this server
 > easily right now, unfortunately. They are:
 >

Since no-one's explicitly mentioned this: be aware that SSH is known for 
doing application-level windowing, limiting performance.

E.g. see https://www.psc.edu/index.php/hpn-ssh/638

---

From: Alan Jenkins <[email-masked]>
Date: Wed, 7 Dec 2016 17:03:11 +0000

On 07/12/2016, Steinar H. Gunderson <[email-masked]> wrote:
> On Wed, Dec 07, 2016 at 04:28:15PM +0000, Alan Jenkins wrote:
> Hm, I thought this was mainly about scp, not ssh?
>
> But yes, hpn-ssh is a sad story; maintenance has been very up and down over
> the years, and there's no end in sight for an upstream merge.
>
> /* Steinar */
> --
> Homepage: https://www.sesse.net/

Sorry, I meant to check the dates as well.  I see now this is old, I
don't know myself whether it is still relevant or not.

I'm sure there was a window at the SSH level.  It looks like SFTP had
a window on top of that

http://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/flow-control.html

I'm not certain if the same applies to the use of SCP, and if so which
layer(s) had the biggest problem.

Alan