Subject: [iccrg] BBR and acking stagnant windows

From: Martin Duke <[email-masked]>
Date: Tue, 19 Nov 2019 12:59:08 +0800

Neal,

I'd like to follow up on our discussion in Singapore. I have no special
knowledge of why Linux doesn't ack when the window doesn't move, but
perhaps you can help me reason about it.

>From the most abstract principle, it seems sensible that if an endpoint can
only process 100kbps, I wouldn't give it more bandwidth than that. That's
what the current code does.

However, considering the practical case, what would be the impact of
removing this line of code?
1) if the connection is almost done, it can complete faster by consuming
the relevant rwnd.
2) the RTT estimate will be closer to min_rtt, for algorithms that care.
3) if not done, we will end up rwnd-limited rather than cwnd-limited. I am
less comfortable reasoning about this, but this may lead to burstier
traffic and periods of idleness waiting for zero-window probes.

Anyway, you alluded to some particular RPC use cases that causes more
serious problems. Can you elaborate?

Martin

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 20 Nov 2019 17:36:49 -0500

On Mon, Nov 18, 2019 at 11:59 PM Martin Duke <[email-masked]> wrote:
>
> Neal,
>
> I'd like to follow up on our discussion in Singapore. I have no special knowledge of why Linux doesn't ack when the window doesn't move, but perhaps you can help me reason about it.
>
> From the most abstract principle, it seems sensible that if an endpoint can only process 100kbps, I wouldn't give it more bandwidth than that. That's what the current code does.
>
> However, considering the practical case, what would be the impact of removing this line of code?
> 1) if the connection is almost done, it can complete faster by consuming the relevant rwnd.
> 3) if not done, we will end up rwnd-limited rather than cwnd-limited. I am less comfortable reasoning about this, but this may lead to burstier traffic and periods of idleness waiting for zero-window probes.
>
> Anyway, you alluded to some particular RPC use cases that causes more serious problems. Can you elaborate?

Hi Martin,

Thanks for following up.

This issue first showed up with an RPC workload with many client connections
sending large RPC requests to a server, which replied with small RPC responses.
Relative to the previous CC, the bbr2 flows showed latency increases at the
tail of the distribution:

                       % change from baseline to bbr2
       ==============================
       Latency (Avg)     3.5 +/- 0.1
       Latency (50%)    -3.9 +/- 0.1
       Latency (95%)    64.4 +/- 0.3
       Latency (99%)   103.3 +/- 0.8

As best as I can tell from analyzing the packet traces (like the one in the
BBR v2 IETF 106  ICCRG slide deck):

(1) With an immediate ACK for cases where unacked data > RCV.MSS,
    the connection stays pipelined:

  o there are almost always two offload bursts in flight

  o as soon as the receiver receives one of the offload bursts,
    it ACKs immediately

  o as soon as the sender receives that ACK, it sends the next data burst

(2) With the ACK delayed until the next application read(), the connection
    almost always stops pipelined behavior and degenerates into "stop and wait"
    behavior:

  o there is typically just one offload burst (the whole cwnd) in flight

  o when the receiver receives one of the data bursts,
    it waits until the app reads the data before ACKing

  o the data arrives in bursts, so often it takes the app a while
    to get around to reading a particular connection

  o while waiting for the app to read the data, and then for the
    ACK to reach the data sender, and then for the next chunk of
    data to be sent, the network link may go idle

AFAICT, by waiting for the application to read the data out of the buffer
before generating the ack, the receive buffer loses its ability to be a good,
well, buffer. It is no longer a "buffer" to deal with temporary rate mismatches
between network and receiving application. Instead, the behavior has tightly
coupled the timing of the network sending process to the details of the timing
of the application reading process. And the reading process may be delayed or
bursty, due to many threads servicing a bursty client workload and having to
get around to reading from the many sockets.

We saw similar performance problems with the previous TCP CC when we tried
operating without faster ACKs and instead relying on the upstream linux "ack if
new rwin >= old rwin" policy.

I agree there is a possibility that the receiver delays feed into a reducing in
pacing rate, as I believe Martin mentioned as a possibility. If so, then this
effect is apparently happening both for BBR-style pacing and for the pacing in
the base Linux TCP stack, which is a function of: pacing_rate = k * cwnd / srtt.

Whatever the exact set of dynamics leading to the performance problem,
the problem is not limited to BBRv2.

thanks,
neal