Subject: Re: [aqm] aqm Digest, Vol 24, Issue 18

From: [email-masked]
Date: Fri, 27 Feb 2015 15:06:29 -0500 (EST)

Having done many experiments in many locations, I can tell you it is a real problem and a severe problem, in the real world.

I can observe it every time I fly from Boston to San Jose and back (which I do once per month).  If you use Gogo (the provider that supports several airlines' network access), all you need to do is measure either the uplink or downlink queue, and you find the typical symptom: a buffer that has multiple seconds of backlog, and no packet loss whatsoever (using either ICMP or UDP).  That's pretty severe - what it means is that the probability that "ssh" times out is non zero, not because of lost packets or lost carrier, but because of the default timeout in ssh being exceeded.

Now why is this a serious problem?  Well, think about it.  If packets sit in queues for 4 seconds, what impact does that have on people using HTTP?  Well, they think a website is down, so within a couple of seconds, they switch to surfing some other web site.  Meanwhile, the downloaded packets (pictures or whatever, which are in transit because of a relatively long TCP window that has built up in HTTP 1.1/TCP/IP, which supports all consecutive HTTP requests to the same server, because the service was temporarily OK), are filling up the download queue - even though the user(s) have moved on and probably even closed their connections on their end).  Does this happen?  Of course it does.  Since I'm in the airplane with these guys, I've run wireshark on the WiFi network, recorded TCP packet streams (headers only), and observed exactly this behavior - very delayed HTTP 1.1 packets (many seconds of delay between request and response).

I don't intend to publish this data for one key reason: it does not meet the standards of academic research publication. Why? because I can't get the data that only Gogo has in its network administration.  Either they will measure and publish or they will not. I suspect they will do the latter. Only Comcast has been willing to publish at all - and that is in a CableLabs tech report by a Sr. VP I know at Comcast, who has the authority to make the decision to reveal Comcast "trade secrets" to some extent.  ATT Wireless had this problem on half of its HSPA+ access networks, but it will never publish - it's embarrassing (and would make people switch to an alternate provider, probably, too).  Instead they have blamed "wireless interference" and "badly designed iOS apps", anything other than their network operations.  (I told ATT about this, not under non-disclosure, when I first measured it.  Their network operations people actually measured the problem, I was told, but I don't know if they fixed it. All I know is that they blamed a subcontractor, without naming that subcontractor in my phone call).

So there's nothing for an academic to cite, unfortunately. Sorry.

It's actually quite hard to do research on whether a problem is real - if it is a problem the operator or vendor will not allow you to publish, or worse, will claim that you didn't measure the right thing or are lying or misleading people.  Thank goodness for Comcast's engineers, who were honest. (after their top management started an aggressive attack on those who observed that their initial solution, an attempt to disrupt "pirate traffic" by TCP RST injection, caused the US FCC to slap them on the wrist.

But the problem is real, and quite prevalent.  I carry around gear and software that I use whenever I am in a new location to detect both the "overbuffering" that may be there, and ALSO, by running tests for 24 hours every hour, how it affects usability over a 24 hour period.

I assure you it is a real problem in practice, and also that it affect others than "gamers".  HTTP based services suck badly when the problem arises. In the ATT case, when I was measuring the problem in a Chicago office building where I was doing other consulting work, every iPhone user was complaining that their iPhone data services were not working.

On Friday, February 27, 2015 2:30pm, [email-masked] said:

> Send aqm mailing list submissions to
> 	[email-masked]
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://www.ietf.org/mailman/listinfo/aqm
> or, via email, send a message with subject or body 'help' to
> 	[email-masked]
> 
> You can reach the person managing the list at
> 	[email-masked]
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of aqm digest..."
> Today's Topics:
> 
>    1. Re: Is bufferbloat a real problem? (Curtis Villamizar)
>    2. Re: Is bufferbloat a real problem? (Daniel Havey)
>    3. Re: Is bufferbloat a real problem? (Daniel Havey)
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 28 Feb 2015 20:12:20 -0500

In message <[email-masked]>
[email-masked] writes:
 
> Having done many experiments in many locations, I can tell you it is a
> real problem and a severe problem, in the real world.
>  
> I can observe it every time I fly from Boston to San Jose and back
> (which I do once per month).  If you use Gogo (the provider that
> supports several airlines' network access), all you need to do is
> measure either the uplink or downlink queue, and you find the typical
> symptom: a buffer that has multiple seconds of backlog, and no packet
> loss whatsoever (using either ICMP or UDP).  That's pretty severe -
> what it means is that the probability that "ssh" times out is non
> zero, not because of lost packets or lost carrier, but because of the
> default timeout in ssh being exceeded.
>  
> Now why is this a serious problem?  Well, think about it.  If packets
> sit in queues for 4 seconds, what impact does that have on people
> using HTTP?  Well, they think a website is down, so within a couple of
> seconds, they switch to surfing some other web site.  Meanwhile, the
> downloaded packets (pictures or whatever, which are in transit because
> of a relatively long TCP window that has built up in HTTP 1.1/TCP/IP,
> which supports all consecutive HTTP requests to the same server,
> because the service was temporarily OK), are filling up the download
> queue - even though the user(s) have moved on and probably even closed
> their connections on their end).  Does this happen?  Of course it
> does.  Since I'm in the airplane with these guys, I've run wireshark
> on the WiFi network, recorded TCP packet streams (headers only), and
> observed exactly this behavior - very delayed HTTP 1.1 packets (many
> seconds of delay between request and response).
>  
> I don't intend to publish this data for one key reason: it does not
> meet the standards of academic research publication. Why? because I
> can't get the data that only Gogo has in its network administration.
> Either they will measure and publish or they will not. I suspect they
> will do the latter. Only Comcast has been willing to publish at all -
> and that is in a CableLabs tech report by a Sr. VP I know at Comcast,
> who has the authority to make the decision to reveal Comcast "trade
> secrets" to some extent.  ATT Wireless had this problem on half of
> its HSPA+ access networks, but it will never publish - it's
> embarrassing (and would make people switch to an alternate provider,
> probably, too).  Instead they have blamed "wireless interference" and
> "badly designed iOS apps", anything other than their network
> operations.  (I told ATT about this, not under non-disclosure, when I
> first measured it.  Their network operations people actually measured
> the problem, I was told, but I don't know if th ey fixed it. All I
> know is that they blamed a subcontractor, without naming that
> subcontractor in my phone call).
>  
> It's actually quite hard to do research on whether a problem is real -
> if it is a problem the operator or vendor will not allow you to
> publish, or worse, will claim that you didn't measure the right thing
> or are lying or misleading people.  Thank goodness for Comcast's
> engineers, who were honest. (after their top management started an
> aggressive attack on those who observed that their initial solution,
> an attempt to disrupt "pirate traffic" by TCP RST injection, caused
> the US FCC to slap them on the wrist.
>  
> But the problem is real, and quite prevalent.  I carry around gear and
> software that I use whenever I am in a new location to detect both the
> "overbuffering" that may be there, and ALSO, by running tests for 24
> hours every hour, how it affects usability over a 24 hour period.
>  
> I assure you it is a real problem in practice, and also that it affect
> others than "gamers".  HTTP based services suck badly when the problem
> arises. In the ATT case, when I was measuring the problem in a Chicago
> office building where I was doing other consulting work, every iPhone
> user was complaining that their iPhone data services were not working.

Yes [unnamed person] you are certainly citing a specific service which
is severely broken as a result of having no clue whatsoever about
bufferbloat.  I will also point out that some frame relay switches of
ancient times also buffered for tens of seconds.  Not so good for
people in Ameritec land that had Internet service over FR in the late
1990s or anyone using PSI prior to the demise of PSI.

Hopefully severe cluelessness is still the exception (though there
seem to be a few repeat offenders).

My caution in
Message-Id: <[email-masked]>
was:

  You are convinced.  So am I but we need to temper the message to avoid
  going too far in the opposite direction - too little buffer.

  [...]

  So excuse the length of this but solving bufferbloat is *not* a silver
  bullet.  Not understanding that point and just making buffers really
  small could result in an even worse situation than we have now.

There was a brief (well years actually) push for tiny buffers or none
at all.  Some wanted to push this to make optical burst switching
seems more feasible.  This was an equally clueless idea and I'm hoping
we don't emphasize bufferbloat so much that some not so cluefuls go
too far in the direction of tiny buffers rather than reasonably sized
buffers with AQM or better yet some flavor of FQ and AQM.

Curtis