Subject: [113attendees] Network Responsiveness at IETF meeting

From: Stuart Cheshire <[email-masked]>
Date: Wed, 23 Mar 2022 15:54:11 +0100

On Monday morning in IPPM (IP Performance Measurement) Christoph Paasch, Randall Meyer (Apple) and William Hawkins (University of Cincinnati) presented a new testing methodology for testing “Responsiveness under Working Conditions”.

(Full disclosure: I am not unbiased here; I have also been involved with this work.)

For those who haven’t heard about this already, instead of recording the best-case ping time on an idle network (like so many network tests do) the “Responsiveness under Working Conditions” test measures the end-to-end application-layer round trip time while the network is also being used to transfer a realistic amount of data.

In this context, “a realistic amount of data” means enough to saturate typical residential upstream and downstream links, such that the intelligence of the buffer management at the respective bottleneck links is exposed. (Saturating upstream and downstream links is not that unusual -- a single smartphone should be able to do that easily. If you send a photograph or a video to a friend, and your smartphone is *not* sending that as fast as your upstream connection can carry it, then send a bug report to the vendor. Unless you have multi-gigabit Internet service to your home, your smartphone or computer should be easily capable of fully using your network capacity.)

Also, the round trip time is not measured using ICMP “ping” messages, because that risks encouraging vendors to implement special-case optimizations for ICMP packets, which might artificially improve benchmarks numbers but would not actually improve the quality of user experience in any useful way. This test measures the application-layer round trip time because that is what affects user experience. (Every time you are watching a streaming video, and you skip ahead to a different part of that video, or to a different video entirely, and you’re then watching a spinning “buffering” indicator before the new video starts playing, that’s mostly because the new video data you want is stuck at the back of a queue somewhere in the network, waiting behind the previously-requested video data that you no longer want. Avoiding excessive queue buildup in the network avoids the long wait for new data to arrive.)

Finally, rather than reporting round trip time in milliseconds, which is a fairly abstract unit to many people, the test instead reports “round-trips per minute” (RPM). This has the nice property of being a metric where bigger is better (like upload and download throughput) which normal people can relate to. It has a typical range from a few hundred to a few thousand RPM, expressed as a three- or four-digit integer, with no fractions or decimal places required.

While here at the IETF meeting in Vienna, I decided to test the various Wi-Fi networks we have available, and report the Responsiveness scores for each.

Presented without comment, here are the results I measured this afternoon while sitting in tcpm in Grand Klimt Hall 2:

ietf            229 RPM
ietf-nat64      225 RPM
ietf-hotel       30 RPM
Hilton Honors   731 RPM

You can try this test for yourself, either here at the IETF meeting, or on your own home or company network.

On macOS 12 (“Monterey”) you can run the networkQuality command-line tool in a Terminal window to see the network’s RPM score.

On iOS 15 you can enable the RPM test by following the instructions here:
<https://support.apple.com/en-us/HT212313>

On other platforms you can use William Hawkins’s Network Responsiveness test written in Go, available here:
<https://github.com/network-quality/goresponsiveness>

Note that these tools are still under development. One area we are still working on is improving the consistency and repeatability of the results. Ideally repeated runs of the test should give comparable results within ±10%; we are not there yet.

The Internet Draft describing how the responsiveness test works is here:
<https://datatracker.ietf.org/doc/html/draft-ietf-ippm-responsiveness>

There’s a video from the Apple developer conference last year which explains this:
<https://developer.apple.com/videos/play/wwdc2021/10239/>

Monday’s IPPM presentation is here:
<https://datatracker.ietf.org/meeting/113/materials/slides-113-ippm-responsiveness-under-working-conditions-00>

Stuart Cheshire

---

From: Stuart Cheshire <[email-masked]>
Date: Thu, 24 Mar 2022 11:58:57 +0100

I want to make it clear that my email yesterday was not intended as any criticism of the hardworking volunteers who run the IETF meeting networks or the generous companies that donate the equipment we use. Sincere thanks to both those groups.

The problem is that bufferbloat is everywhere, and we still have a limited selection of tools to detect bufferbloat, and limited awareness of those tools. Without routine use of bufferbloat detection tools, it is not at all surprising that bufferbloat remains a problem, even a decade after Jim Gettys identified the problem and coined the term “bufferbloat”.

What we experience is that sometimes network connections work well and our subjective experience using our computers and devices is that everything is “snappy” and fast. Other times everything is sluggish and slow, with no clear reason why. We can run an Internet speed test to see how many megabits of throughput we have, but that usually doesn’t tell us anything useful. What we’ve found is that above about 10 Mb/s there is little correlation between throughput rates and the experienced responsiveness of most day-to-day computing activities. If you have a 10 Gb/s Internet connection with a 1-second round trip time, then your once-a-year download of your operating system software update might be very fast, but for the other 364 days of the year everything else you do is going to feel sluggish and slow. High throughput alone is not enough for a good user experience. A very low round trip time when the network is idle is not enough for a good user experience, if the round trip time goes through the roof whenever the networking is being used. What we need is a good round trip time that remains good even when the network is being used. The good news is that we believe we know how to achieve this (e.g., L4S and similar), but unfortunately those solutions are not yet deployed.

Hence my email yesterday. I wanted to highlight Monday’s IPPM presentation by Christoph Paasch, Randall Meyer and William Hawkins, to highlight the need for development of more diagnostic tools, and to highlight the need for more awareness and routine usage of those tools.

For anyone interested to learn more, I also recommend this excellent report “Latency Explained” recently published by the Broadband Internet Technical Advisory Group. If you have people asking you what all this fuss about “latency” is about, and you don’t have the energy to explain it yourself every time, refer them to this report. In very clear terms it explains network latency (sometimes called “lag”), why it matters, what causes it, how to measure it, and what we can do about it.

<https://www.bitag.org/latency-explained.php>

Stuart Cheshire

P.S. When I took my measurements yesterday sitting in tcpm in Grand Klimt Hall 2 at IETF 113 in Vienna, I was surprised by the results, and hesitated about sending the email in case the results were bogus. But then I checked using the Bufferbloat Test provided by Waveform.com, and the data that showed confirnmed the Network Quality results.

<https://www.waveform.com/tools/bufferbloat>