Subject: [tsvwg] Fwd: [Network-quality-workshop] Evidence of Bufferbloat?

From: Bob Briscoe <[email-masked]>
Date: Mon, 25 Oct 2021 21:32:49 +0100

Neal,

Forwarding this email in support of a conversation about your WGLC 
comment on the [Hohlfeld14] paper discussed below.

Bob

-------- Forwarded Message --------
Subject: 	Re: [Network-quality-workshop] Evidence of Bufferbloat?
Date: 	Tue, 21 Sep 2021 15:25:00 +0100
From: 	Bob Briscoe <[email-masked]>
To: 	Rich Brown , Brandon Schlinker
CC: 	[email-masked],

Rich,

I was not questioning that the bufferbloat problem exists. I was stating 
that there is no scientific study to properly quantify its prevalence - 
both how often it affects various types of user and what proportion of 
users it affects. If there was, I would have been able to quote it at 
Brandon and that could have been the end of the matter.

When I talked about this on the slack, I misremembered. I said that I am 
only aware of one scientific study [Hohlfeld14] and that it was flawed. 
Actually I know of two studies: [Allman12] and [Hohlfeld14]. In the 
process of writing this email I found a third [Chirichella13], but I've 
only just scanned it, so I won't comment on it yet.

Also, my apologies go to Oliver Hohlfeld and his co-authors, because it 
was [Allman12] that was flawed (IMO). [Hohlfeld14] puts an impressive 
amount of effort into its methodology. However I still wouldn't want to 
use it as a definitive answer to the question of prevalence of 
bufferbloat, because there are still some significant problems with the 
methodology (see below).

Both papers say that the impact of bufferbloat isn't that significant. 
However, I think this is more a symptom of some people "crying wolf" too 
exaggeratedly about bufferbloat. So when researchers investigate 
scientifically and discover the effects are quite intermittent, they 
conclude that they have not found the problem to be anything like as bad 
as they had been led to believe. This is particularly true in the case 
of [Hohlfeld14], which finds what I would call a significant amount of 
delay for a significant proportion of users:
     "Only 2.8% (1%) experience excessive queueing delays of more than 
500ms (1000ms)"
but it concludes that:
     "Two very recent studies [...] concluded that the magnitude of 
bufferbloat is modest. Our results [...] further substantiate these 
findings."

Detailed comments on each paper.

[Allman12] used data collected in 2011-12 from 90 homes on an 
experimental fibre to the home network, with symmetric 1Gb/s. RTT 
measurements focused on one direction and did not include the portion of 
the path in the FttH network, so it was looking for bloat in the 
/downlink/ buffers of peers who were doing p2p file transfers /from/ 
these FttH users. Therefore it was not set up to look for bufferbloat in 
any uplink. While searching just now, I found this posting at the time 
from Keith Winstein who obviously had similar concerns to mine:
http://mailman.postel.org/pipermail/end2end-interest/2013-January/008580.html

[Hohlfeld14] includes a much more extensive study (see Fig 1c for 
results - altho it's hard to read off accurately). However, it uses the 
range of the smoothed RTT metric from the Linux kernel to estimate 
queueing delay. This metric (deliberately) includes delay between 
packets due to TCP's delayed ACK mechanism, so it's not appropriate for 
estimating queuing delay (it measures RTT samples from the oldest 
outstanding packet not the newest). [Hohlfeld14] also only reports the 
mean of this metric (which is already a moving average internally). So, 
on the one hand, perceived delay is over-estimated, and on the other it 
is under-estimated. But there's no reason to believe that the two errors 
happen to be equal and therefore cancel.

So I'm afraid we're still without a scientifically rigorous answer to 
the question of how prevalent bufferbloat is.

BTW, in [Hohlfeld14], only §3 is intended to characterize queuing delay 
over the real Internet. The large majority of the paper (§4 onwards) is 
devoted to deriving QoE metrics for three applications (VoIP, streamed 
video and Web) from a testbed loaded with synthetic traffic over varying 
buffer sizes. None of that is intended to characterize real traffic, so 
it's not relevant to the current conversation (but it's a very 
worthwhile read, altho IMO there are flaws in that part too).

      References

[Allman12] Allman, M. "Comments on Bufferbloat" Computer Communication 
Review, ACM, 43(1):30–37 (Jan 2012)

[Hohlfeld14] Hohlfeld, O.; Pujol, E.; Ciucu, F.; Feldmann, A. & Barford, 
P. A "QoE Perspective on Sizing Network Buffers" In Proc. Internet 
Measurement Conf (IMC'14), ACM, 333-346 (2014)

[Chirichella13]  Chiara Chirichella and D. Rossi  "To the Moon and back: 
are Internet bufferbloat delays really that large" In IEEE INFOCOM TMA 
Workshop (2013).

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 25 Oct 2021 17:04:46 -0400

On Mon, Oct 25, 2021 at 4:32 PM Bob Briscoe <[email-masked]> wrote:

> Neal,
>
> Forwarding this email in support of a conversation about your WGLC comment
> on the [Hohlfeld14] paper discussed below.
>

Thanks, Bob. I agree with your assessment.

Along the lines of what you are saying, I'm also not aware of any rigorous
study of the global prevalence of bufferbloat that can be easily used as a
replacement for the [Hohlfeld14] reference.

The closest data point I can think of from our own team's experience is
that when we compare CUBIC vs BBRv1 for YouTube streaming, BBRv1 sees 32%
lower RTTs  than CUBIC, on average across globally sampled YouTube streams.
This data point is from our slides from IETF 98:

https://www.ietf.org/proceedings/98/slides/slides-98-iccrg-an-update-on-bbr-congestion-control-00.pdf#page=9

Of course that's just one data set, and YouTube is not a straight bulk
download so it's not going to show the worst case bloat. Also, the data set
is probably biased against showing bufferbloat, since for paths with
serious bufferbloat probably users don't try watching YouTube videos, or
try and then give up quickly.

FWIW, IMHO there is a strong enough consensus in the IETF and TCP stack
developer community that queuing delay is worth optimizing, that the
paragraph in draft-ietf-tsvwg-l4s-arch-10 discussing the prevalence of queuing
delay problems (and citing [Hohlfeld14]) can be omitted.

best,
neal

> From: Bob Briscoe <[email-masked]> <[email-masked]>
> Actually I know of two studies: [Allman12] and [Hohlfeld14]. In the process
> of writing this email I found a third [Chirichella13], but I've only just
> scanned it, so I won't comment on it yet.
>
> amount of effort into its methodology. However I still wouldn't want to use
> it as a definitive answer to the question of prevalence of bufferbloat,
> because there are still some significant problems with the methodology (see
> below).
>
> conclude that they have not found the problem to be anything like as bad as
> they had been led to believe. This is particularly true in the case of
> [Hohlfeld14], which finds what I would call a significant amount of delay
> for a significant proportion of users:
> [Allman12] used data collected in 2011-12 from 90 homes on an experimental
> fibre to the home network, with symmetric 1Gb/s. RTT measurements focused
> on one direction and did not include the portion of the path in the FttH
> network, so it was looking for bloat in the /downlink/ buffers of peers who
> were doing p2p file transfers /from/ these FttH users. Therefore it was not
> set up to look for bufferbloat in any uplink. While searching just now, I
> found this posting at the time from Keith Winstein who obviously had
> similar concerns to mine:
>
> [Hohlfeld14] includes a much more extensive study (see Fig 1c for results
> - altho it's hard to read off accurately). However, it uses the range of
> the smoothed RTT metric from the Linux kernel to estimate queueing delay.
> This metric (deliberately) includes delay between packets due to TCP's
> delayed ACK mechanism, so it's not appropriate for estimating queuing delay
> (it measures RTT samples from the oldest outstanding packet not the
> newest). [Hohlfeld14] also only reports the mean of this metric (which is
> already a moving average internally). So, on the one hand, perceived delay
> is over-estimated, and on the other it is under-estimated. But there's no
> reason to believe that the two errors happen to be equal and therefore
> cancel.
>
> So I'm afraid we're still without a scientifically rigorous answer to the
> question of how prevalent bufferbloat is.
>
> it's not relevant to the current conversation (but it's a very worthwhile
> read, altho IMO there are flaws in that part too).
>
> References
> Bob Briscoe                               http://bobbriscoe.net/
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 27 Oct 2021 22:19:05 +0100

Neal,

Thanks, see [BB] inline

On 25/10/2021 22:04, Neal Cardwell wrote:
>
>
> On Mon, Oct 25, 2021 at 4:32 PM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Neal,
>
> Along the lines of what you are saying, I'm also not aware of any 
> rigorous study of the global prevalence of bufferbloat that can be 
> easily used as a replacement for the [Hohlfeld14] reference.
>
> The closest data point I can think of from our own team's experience 
> is that when we compare CUBIC vs BBRv1 for YouTube streaming, BBRv1 
> sees 32% lower RTTs  than CUBIC, on average across globally sampled 
> YouTube streams. This data point is from our slides from IETF 98:
> <https://www.ietf.org/proceedings/98/slides/slides-98-iccrg-an-update-on-bbr-congestion-control-00.pdf#page=9> 
>

[BB] Yes, that was very interesting data. Thank you.

Can you say more about what that means?
* Were CUBIC and BBR sessions run at different times to the same users? 
Or were some users always CUBIC and others always BBR? Or was the RTT in 
each case normalized to the min RTT during the session?
* Was it that some percentage of the streams saw a lot lower RTT with 
BBR, while others saw no difference? Or was it a fairly even reduction 
across the board? More insight into the distribution of the reductions 
would give insight into how frequently users experience bufferbloat 
(unless some other effect is confusing us as well).
* What is "the RTT of a stream"? The average of the RTT of all the 
packets? Or some percentile? Or excluding some start-up perod? Or what? 
Are long streams weighted in the average more than short streams?

> download so it's not going to show the worst case bloat. Also, the 
> data set is probably biased against showing bufferbloat, since for 
> paths with serious bufferbloat probably users don't try watching 
> YouTube videos, or try and then give up quickly.
>
> FWIW, IMHO there is a strong enough consensus in the IETF and TCP 
> stack developer community that queuing delay is worth optimizing, that 
> the paragraph in draft-ietf-tsvwg-l4s-arch-10 discussing the 
> prevalence of queuing delay problems (and citing [Hohlfeld14]) can be 
> omitted.

[BB] To be clear, I wasn't questioning whether something needed to be 
done about bufferbloat. I am saying we still need proper scientific 
evidence of how prevalent the impact of bufferbloat is, by percentage of 
users and percentage of time online of each user, and what the 
distribution of actual delay increases are, not just the horror 
stories... *in order to convince doubters*, particularly those who 
operate the companies that we expect to shoulder the cost of remedying it.

1) Originally, when I posted the message that I quoted below, it was in 
response to a comment at an IAB workshop from Brandon Schlinker from 
Facebook who said he had looked and found v little evidence of 
bufferbloat. And I was saying it's sad that, with all the banging on 
about bufferbloat, we don't have a good scientific study that we can 
just throw at a question like that to say "Look this is the evidence of 
how prevalent the problem is." Instead, AFAICT the three research 
studies I have found about bufferbloat all say it's not prevalent.

2) I want to encourage those who are evanglizing bufferbloat to always 
explain that the effects are intermittent. Yes, it's explained that it 
occurs when a long flow coincides with other traffic, but few if any 
people have an accurate model in their heads of how often that happens. 
Due to these omissions, many people think that this bufferbloat problem 
must be impacting them all the time. Then when they discover it 'only' 
has an impact 2-3% of the time (say), they are inclined to think that 
bufferbloat is not such a problem, and nothing needs to be done about it.

And I think #2 partly explains #1. Researchers are saying this problem 
is not as bad as they had been led to believe, probably because no-one 
told them it was intermittent. So when they discover the truth, they 
think someone has been crying wolf and dismiss it all.

Bob

>     From: 	Bob Briscoe <[email-masked]> <mailto:[email-masked]>
>     CC: 	[email-masked]
>     <mailto:[email-masked]>,
>
>
>
>     Rich,
>
>     I was not questioning that the bufferbloat problem exists. I was
>     stating that there is no scientific study to properly quantify its
>     prevalence - both how often it affects various types of user and
>     what proportion of users it affects. If there was, I would have
>     been able to quote it at Brandon and that could have been the end
>     of the matter.
>
>     When I talked about this on the slack, I misremembered. I said
>     that I am only aware of one scientific study [Hohlfeld14] and that
>     it was flawed. Actually I know of two studies: [Allman12] and
>     [Hohlfeld14]. In the process of writing this email I found a third
>     [Chirichella13], but I've only just scanned it, so I won't comment
>     on it yet.
>
>     Also, my apologies go to Oliver Hohlfeld and his co-authors,
>     because it was [Allman12] that was flawed (IMO). [Hohlfeld14] puts
>     an impressive amount of effort into its methodology. However I
>     still wouldn't want to use it as a definitive answer to the
>     question of prevalence of bufferbloat, because there are still
>     some significant problems with the methodology (see below).
>
>     Both papers say that the impact of bufferbloat isn't that
>     significant. However, I think this is more a symptom of some
>     people "crying wolf" too exaggeratedly about bufferbloat. So when
>     researchers investigate scientifically and discover the effects
>     are quite intermittent, they conclude that they have not found the
>     problem to be anything like as bad as they had been led to
>     believe. This is particularly true in the case of [Hohlfeld14],
>     which finds what I would call a significant amount of delay for a
>     significant proportion of users:
>         "Only 2.8% (1%) experience excessive queueing delays of more
>     than 500ms (1000ms)"
>     but it concludes that:
>     bufferbloat is modest. Our results [...] further substantiate
>     these findings."
>
>     measurements focused on one direction and did not include the
>     portion of the path in the FttH network, so it was looking for
>     bloat in the /downlink/ buffers of peers who were doing p2p file
>     transfers /from/ these FttH users. Therefore it was not set up to
>     look for bufferbloat in any uplink. While searching just now, I
>     found this posting at the time from Keith Winstein who obviously
>     had similar concerns to mine:
>     <http://mailman.postel.org/pipermail/end2end-interest/2013-January/008580.html>
>
>     results - altho it's hard to read off accurately). However, it
>     uses the range of the smoothed RTT metric from the Linux kernel to
>     estimate queueing delay. This metric (deliberately) includes delay
>     between packets due to TCP's delayed ACK mechanism, so it's not
>     appropriate for estimating queuing delay (it measures RTT samples
>     from the oldest outstanding packet not the newest). [Hohlfeld14]
>     also only reports the mean of this metric (which is already a
>     moving average internally). So, on the one hand, perceived delay
>     is over-estimated, and on the other it is under-estimated. But
>     there's no reason to believe that the two errors happen to be
>     equal and therefore cancel.
>
>     So I'm afraid we're still without a scientifically rigorous answer
>     to the question of how prevalent bufferbloat is.
>
>     BTW, in [Hohlfeld14], only §3 is intended to characterize queuing
>     delay over the real Internet. The large majority of the paper (§4
>     onwards) is devoted to deriving QoE metrics for three applications
>     (VoIP, streamed video and Web) from a testbed loaded with
>     synthetic traffic over varying buffer sizes. None of that is
>     intended to characterize real traffic, so it's not relevant to the
>     current conversation (but it's a very worthwhile read, altho IMO
>     there are flaws in that part too).
>
>
>           References
>
>
>     [Allman12] Allman, M. "Comments on Bufferbloat" Computer
>     Communication Review, ACM, 43(1):30–37 (Jan 2012)
>
>     [Hohlfeld14] Hohlfeld, O.; Pujol, E.; Ciucu, F.; Feldmann, A. &
>     Barford, P. A "QoE Perspective on Sizing Network Buffers" In Proc.
>     Internet Measurement Conf (IMC'14), ACM, 333-346 (2014)
>
>     [Chirichella13]  Chiara Chirichella and D. Rossi  "To the Moon and
>     back: are Internet bufferbloat delays really that large" In IEEE
>     INFOCOM TMA Workshop (2013).
>
>
>     -- 
>     Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>
>

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 29 Oct 2021 16:03:42 -0400

On Wed, Oct 27, 2021 at 5:19 PM Bob Briscoe <[email-masked]> wrote:

> * Were CUBIC and BBR sessions run at different times to the same users? Or
> were some users always CUBIC and others always BBR? Or was the RTT in each
> case normalized to the min RTT during the session?
>

In this experiment, a small, random subset of users was assigned to always
use CUBIC, and a different small, random subset of users was assigned to
always use BBRv1.

> * Was it that some percentage of the streams saw a lot lower RTT with BBR,
> while others saw no difference? Or was it a fairly even reduction across
> the board? More insight into the distribution of the reductions would give
> insight into how frequently users experience bufferbloat (unless some other
> effect is confusing us as well).
>

I don't have that data analysis at hand, unfortunately.

> packets? Or some percentile? Or excluding some start-up perod? Or what? Are
> long streams weighted in the average more than short streams?
>

Here what I meant was:

(1) Each YouTube playback saved the TCP RTT from tcp_info (which is the
Linux
    TCP srtt) at the end of each playback chunk.

(2) Each YouTube playback computed the mean of those RTT samples for the
   playback chunks in that playback, and that is the "Mean RTT" for that
playback.

(3) Given BBRv1 had a 32% lower "Mean RTT" metric, this implies the mean of
the
    "Mean RTT" metrics for all BBRv1 playbacks in the experiment was 32%
lower than
    the mean of the "Mean RTT" QoE metrics for all CUBIC connections in the
    sample.

Some more on YouTube chunks:

https://youtube-eng.googleblog.com/2018/04/making-high-quality-video-efficient.html

> [BB] To be clear, I wasn't questioning whether something needed to be done
> about bufferbloat. I am saying we still need proper scientific evidence of
> how prevalent the impact of bufferbloat is, by percentage of users and
> percentage of time online of each user, and what the distribution of actual
> delay increases are, not just the horror stories... *in order to convince
> doubters*, particularly those who operate the companies that we expect to
> shoulder the cost of remedying it.
>
> Facebook who said he had looked and found v little evidence of bufferbloat.
> And I was saying it's sad that, with all the banging on about bufferbloat,
> we don't have a good scientific study that we can just throw at a question
> like that to say "Look this is the evidence of how prevalent the problem
> is." Instead, AFAICT the three research studies I have found about
> bufferbloat all say it's not prevalent.
>
> occurs when a long flow coincides with other traffic, but few if any people
> have an accurate model in their heads of how often that happens. Due to
> these omissions, many people think that this bufferbloat problem must be
> impacting them all the time. Then when they discover it 'only' has an
> impact 2-3% of the time (say), they are inclined to think that bufferbloat
> is not such a problem, and nothing needs to be done about it.
>
> And I think #2 partly explains #1. Researchers are saying this problem is
> not as bad as they had been led to believe, probably because no-one told
> them it was intermittent. So when they discover the truth, they think
> someone has been crying wolf and dismiss it all.
>

I see. Makes sense. Thanks!

best,
neal

>> the path in the FttH network, so it was looking for bloat in the /downlink/
>> buffers of peers who were doing p2p file transfers /from/ these FttH users.
>> Therefore it was not set up to look for bufferbloat in any uplink. While
>> searching just now, I found this posting at the time from Keith Winstein
>> who obviously had similar concerns to mine:
>>