Subject: [iccrg] Buffer Size in "The Game Theory Behind Running Cubic and BBR on the Internet"

From: Ayush Mishra <[email-masked]>
Date: Thu, 18 Nov 2021 23:19:48 +0800

(Responding to this thread
<https://mailarchive.ietf.org/arch/msg/iccrg/wA4Zpr9-0BEWnp1CHnOdTg_IZbw/>
since
there seems to be something wrong with my other email.)

Hi Szilveszter,
Thank you for your comments! Please see my responses inline below:

"When watching the $SUBJECT presentation I noticed that 100% BBR is always
optimal when the buffer was ½ BDP. Is that so?"

==> According to our controlled experiments, this does indeed seem to be
the case. Theoretically, we expect BBR to perform better than CUBIC in
shallow buffers (with CUBIC being loss-based) - therefore I think it is
reasonable to expect the balance to tip in either algorithm's favor in the
opposite ends of the *buffer spectrum*. That is, in game-theory terms, BBR
would become the dominant strategy in very shallow buffers, and CUBIC would
be the dominant strategy in very deep buffers. The question then is where
this 'very shallow buffer' and 'very deep buffer' tipping point exists.
Empirically, we have found this tipping point to be 1 BDP to make BBR the
dominant strategy in the network.

Our current hypothesis is that this number is down to 1.) BBR becoming
cwnd-bound when competing with CUBIC and 2.) BBR has an inflight cap of 2
BDP. In other words, because BBR ends up putting 1 BDP (bw_bbr * RTT) in
the buffer, it is able to have a clear advantage over CUBIC when the buffer
size is *less* than *1 BDP (btl_bw * **RTT)**. *One way to think of it is
that since BBR is somewhat loss-agnostic, it is able to maintain 1 BDP
(bw_bbr * RTT) in the bottleneck buffer even when the buffer size is
*less* than
1 BDP (btl_bw * RTT). This doesn't allow the CUBIC packets to make enough
room in the buffer for their own packets, and therefore the throughput of
the CUBIC flows suffers.

We have also been modeling the interaction between CUBIC and BBR to predict
the tipping point at the opposite end of the spectrum, which is when CUBIC
becomes the dominant strategy/winner. According to another popular model (Ware
et al. <https://www.cs.cmu.edu//~rware/assets/pdf/ware-imc2019.pdf>) BBR's
RTT overestimation should increase indefinitely in deeper buffers and CUBIC
*should* never become the dominant strategy. However, we have found that in
very deep buffers, BBR is not always cwnd-limited, and is therefore bound
to run out of steam at some buffer size.

"Also have you studied smaller buffers and/or more data points in the 0 BDP
range? I hope that these smaller buffers will become typical, also
addressing bufferbloat to some extent."

==> Thank you for the suggestion! We have not tried running experiments
without any buffers/super tiny buffers. I will report back here if I find
something interesting 🙂

"We also have some related results in http://ppv.elte.hu/buffer-sizing/ for
the smaller buffer size ranges with BBRv2.

Maybe this paper on buffer sizing is also somewhat relevant:
https://arxiv.org/abs/2109.11693"

==>  Thanks! I will check these out.

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 18 Nov 2021 11:22:16 -0500

Thanks, Ayush and Szilveszter.

I believe most folks in the ICCRG community are aware of this already, but
it's probably worth clarifying in this thread that the discussion here
about "BBR" is "BBRv1" in this case. One of the focuses with BBRv2 is
improving coexistence with CUBIC/Reno, so the results for BBRv2 should
generally be a more even sharing of bandwidth for both shallow and deep
buffers.

thanks,
neal

On Thu, Nov 18, 2021 at 10:20 AM Ayush Mishra <[email-masked]>
wrote:

> <https://mailarchive.ietf.org/arch/msg/iccrg/wA4Zpr9-0BEWnp1CHnOdTg_IZbw/> since
> opposite ends of the *buffer spectrum*. That is, in game-theory terms,
> BBR would become the dominant strategy in very shallow buffers, and CUBIC
> would be the dominant strategy in very deep buffers. The question then is
> where this 'very shallow buffer' and 'very deep buffer' tipping point
> exists. Empirically, we have found this tipping point to be 1 BDP to make
> BBR the dominant strategy in the network.
>
>
> *less* than 1 BDP (btl_bw * RTT). This doesn't allow the CUBIC packets to
> make enough room in the buffer for their own packets, and therefore the
> throughput of the CUBIC flows suffers.
>
>
> We have also been modeling the interaction between CUBIC and BBR to
> predict the tipping point at the opposite end of the spectrum, which is
> when CUBIC becomes the dominant strategy/winner. According to another
> popular model (Ware et al.
> <https://www.cs.cmu.edu//~rware/assets/pdf/ware-imc2019.pdf>) BBR's RTT
> overestimation should increase indefinitely in deeper buffers and CUBIC
> *should* never become the dominant strategy. However, we have found that
> in very deep buffers, BBR is not always cwnd-limited, and is therefore
> bound to run out of steam at some buffer size.
>
>
>
> "Also have you studied smaller buffers and/or more data points in the 0
> BDP range? I hope that these smaller buffers will become typical, also
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg
>

---

From: Ayush Mishra <[email-masked]>
Date: Fri, 19 Nov 2021 01:26:13 +0800

> generally be a more even sharing of bandwidth for both shallow and deep
==> Thanks for the clarification, Neal. Yes, we expect very different
results for BBRv2. I can confirm that our preliminary experiments with
BBRv2 are showing BBRv2 and CUBIC to be very competitive over a range of
buffer sizes. However, at the current stage of our project/mathematical
model, it's tough to make any predictions about BBRv2 as we have for BBRv1.

@Neal, while working with BBRv1, we found that RTT over-estimation is one
of the key factors in deciding how well BBRv1 competes with CUBIC. Do you
expect BBRv2 to suffer from this RTT over-estimation too? IIRC, BBRv2's
cwnd is still dependant on the BDP, and therefore the RTT_min estimate.
Does this mean that RTT over-estimation will still be the key predictor for
how aggressive BBRv2 will be while competing with other flows?

We have been working on our own congestion control algorithm and for us,
the only way to get rid of this problem (if you can call it that!) is to
keep the buffer occupancy independent of the BDP (i.e. cwnd=BDP+x, rather
than cwnd=x*BDP)