Subject: Re: [iccrg] BBR question (was: Re: Bufferbloat and Congestion Control)

From: Toerless Eckert <[email-masked]>
Date: Mon, 3 Apr 2017 01:22:18 +0200

Thanks, Neal

I am still not sure in this scenario what predicted rate i would see
in a scenario like this, and therefore what pacing i get:

Lets be more specific: 6 systems, each with 100Mbps ethernet connected to
100Mbps switch (*), 5 of them are sending a 10 Mpbs flow each (in 1/60 bursts),
system #6 is the receiver for all 5 flows/tcp-sessions.

With your answer it seems i get the following behavior:

The senders BBR stacks each determine a bottleneck rate of 100Mbps, so they 
pace to 100Mbps. Every 1/60 of a second a sender gets a 166kbps burst from
the sender app. The pacer checks how much it can send in a 2msec timeslot,
which is 100Mbps*0.002 = 200kbps, so the bursts go through without 
further pacing.

Now, if those burst happen to all collide on the switches egres queue to
system #6 (assuming no buffer drops), there would be delay increase,
which seems to be taken into account by the bottleneck rate measurement,
but lets assume this does manage to measure the actual bottleneck share
of 20Mbps/sender. In that case, the per 2msec pacer amount would be
20Mbps*0.002 = 40kbps, so the sender app bursts would get shaped to roughly 4
intervals. But of course i am not sure if BBR would ever get down to
measure the rarte down to 20Mbps given how there is no ongoing congestion,
but just burst collision based delay increases.

If i had this state, there would be a lot less burst aggregation, so
the periodic uprating of BBR might kick in, increasing the predicted 
rate, so one might end up pacing into 3 2msec slots. And it might
swing back then given how there would be more randomn queuing delay then.

Is it possible to more accurately predict the behavior of BBR in cases
like this ? Is there even a single long term stable predicted rate in a
case like this or would it fluctuate over time like i imagine it could,
and if so, what would be the predicted rate and why..

Would be quite important to understand and be able to predict the
behavior in this non_congestion case.

Thanks!
    Toerless

On Fri, Mar 31, 2017 at 12:46:19PM -0400, Neal Cardwell wrote:
> On Fri, Mar 31, 2017 at 11:03 AM, Toerless Eckert <[email-masked]> wrote:
> >> > One relevant alternative I have not seen mentioned in this thread is
> >> > BBR congestion control. (For BBR info, including the CACM paper and
> >> > links to the BBR talks at the last 2 ICCRG sessions, see:
> >> > https://groups.google.com/d/forum/bbr-dev )
> >
> > One question i have about BBR for which i could not figure out the answer
> > conclusively from reading the papers:
> >
> > Assume you a video flow (over BBR) where the sender sends eg: once every 1/60
> > second a burst of packets - eg: video frame - for simplicity say those bursts
> > have all the same size, and the average bitrate of such a flow is eg: 10 Mbps.
> >
> > We put eg: 5 such flows across a 100 Mbps paths, so no ongoing congestion.
> > With TCP,no-BBR i would see TCP window sized bursts (aka: RTT plays a role), but
> > what would i see with BBR ? I understand how BBR would be pacing packets to
> > minimize those bursts whenever there is actual congestion, but in this case
> > there is no ongoing congestion: 50 Mbps aggregate on 100 Mbps path. Would i
> > still see pacing, or would each of these flows burstyness be similar or equal to
> > pre-BBR TCP ?
> 
> In BBR, pacing is the primary mechanism for controlling the in-flight
> packets, and cwnd is a fallback.  So, in this scenario, you would
> still see BBR pacing out the packets for each of the chunks written 60
> times per second. BBR paces in order to try to match the sending rate
> to the available bandwidth at the bottleneck, in order to reduce
> queuing, to reduce queuing delays and packet loss.
> 
> The exact level of burstiness you would see in this case with BBR
> would depend on the quantization scheme used by the pacing
> implementation involved. For example, in this scenario the Linux TCP
> stack would aim to pace out a 2-packet TSO burst roughly every 2ms.
> 
> It should also be said that some pre-BBR TCP implementations could
> also employ pacing. Google used paced TCP CUBIC for a number of years
> before switching to BBR; see Yuchung and Eric's presentation from IETF
> 88:
>   https://www.ietf.org/proceedings/88/slides/slides-88-tcpm-9.pdf
> 
> neal

-- 
---
[email-masked]

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 3 Apr 2017 20:51:38 -0400

> With your answer it seems i get the following behavior:

I suspect the ultimate behavior would depend on factors like whether
the bursts from the senders  were synchronized, or randomized, etc. If
someone is curious about BBR behavior in a scenario like this then I
suspect testing it with a recent Linux kernel and netem is the way to
go (using the fq qdisc with pacing enabled).

neal

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 4 Apr 2017 03:27:21 +0200

Sure, if i had the means and time for this lab test, i would do it.  Would be great
though if you folks at google would consider it to be usefu too to be able to
conceptually model the behavior of BBR under such non-congestion situation, eg:
will the algorithm create a stable scenario when the offered loads are stable,
or would one see oscillations in calulated rates - and therefore oscillations in
burstyness.

---

From: Michael Welzl <[email-masked]>
Date: Tue, 4 Apr 2017 08:33:59 +0200

Hi,

I'll add something to that with my chair hat on - so this is a neutral observation and not critique of BBR per se (e.g., it applies equally to previously presented work on new AQM algorithms):

The quality (read: depth) of evaluations presented in ICCRG has clearly been declining. If I think back at the good old Cubic vs. HTCP etc. etc. days, that was a very different thing, there were comparisons against many other mechanisms, done under a large variety of circumstances. The onus of doing such work has always been on the authors - or otherwise their work doesn't go anywhere.
Now clearly, the requirements for Google's work to "go anywhere" are lower, so they don't appear to see a need for such evaluations. I find that worrying, and I would very much welcome if at least the academics in the room could fill that gap. I think this is a relatively low-hanging fruit for academics to pick.

Cheers,
Michael

> On 04 Apr 2017, at 03:27, Toerless Eckert <[email-masked]> wrote:
> 
> 
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg