Subject: [tsvwg] draft minutes from interim

From: Wesley Eddy <[email-masked]>
Date: Wed, 4 Mar 2020 22:04:12 -0500

Draft minutes from the interim are available at:

https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/materials/minutes-interim-2020-tsvwg-01-202002200900

Please send any corrections or improvements.

We are planning to get the WebEx recording posted to the IETF YouTube 
channel shortly, but need to do a little bit more work on that first.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Mar 2020 17:27:34 +0000

Wes,

> https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/materials/minutes-interim-2020-tsvwg-01-202002200900 
Thx for these. Look like a pretty good summary.

For those who might want to find these meeting materials in future 
without saving this specific URL, interim meetings are linked off the 
data tracker IETF proceedings page:
https://www.ietf.org/how/meetings/interim/

> Please send any corrections or improvements.
During Jonathan's slides, there was a conversation that seems to be 
completely missing where Greg asked for SCE scenarios to not just show 
experiments as selected individual time series that could be "cherry 
picked", but to usually give results over a range of realistic link 
rates and RTTs and a range of traffic scenarios. Also there was a 
request to present summary "figures of merit" e.g. queue delay, 
utilization, rate ratio, packet drop, ECN marking, normalized flow 
completion time, etc. Also to show "whiskers", e.g. 99-%ile as well as 
mean, across a number of experiment runs.

(The conversation started by audio, then continued on the chat). Time 
series are useful to dig into particular anomalous behaviours, but not 
really suitable for checking the operating range or to compare solutions 
over a range of scenarios. Higher percentile figures are essential for 
queue delay to determine how a real-time app would cope.

> channel shortly, but need to do a little bit more work on that first.
Thank you.
And the chat log hopefully?

Now I've seen some hints at the conversation I couldn't hear during the 
chairs slides (audio problems), I'll start following-up on the tsvwg list.

Cheers

Bob

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 7 Mar 2020 10:39:25 +0200

> On 6 Mar, 2020, at 7:27 pm, Bob Briscoe <[email-masked]> wrote:
> 
> During Jonathan's slides, there was a conversation that seems to be completely missing where Greg asked for SCE scenarios to not just show experiments as selected individual time series that could be "cherry picked", but to usually give results over a range of realistic link rates and RTTs and a range of traffic scenarios. Also there was a request to present summary "figures of merit" e.g. queue delay, utilization, rate ratio, packet drop, ECN marking, normalized flow completion time, etc. Also to show "whiskers", e.g. 99-%ile as well as mean, across a number of experiment runs.

I don't think the conversation was *completely* missed in the minutes:

> - Greg White asked if data is anecdotal cases and time series, rather than wider range, CDFs.

At the time, I remember pointing out that for a half-hour talk, we of course had to be selective, but that we could discuss specific tests and presentation formats offline.  Our code is also available for you to perform your own tests and reproduce ours.

The reason we prefer to present time-series charts is because they show the evolution behaviour of the congestion control in a way that makes behavioural anomalies obvious.  In particular, this was the best way to show the robustness of SCE to a shift from an SCE-aware bottleneck to an otherwise identical RFC-3168 bottleneck and back, a test which had been specifically requested by one of the Chairs; we have not yet seen a demonstration of an equivalent scenario in L4S.  We do this to project confidence that SCE works in a wide variety of circumstances, not just the ones actually presented.

Our choices of scenario to present were in part directed towards cases which are considered "difficult", either traditionally with conventional congestion, or through experience with high-fidelity congestion in particular.  We wanted to show that we could handle these "difficult" cases cleanly.  At least one of those cases was one that we have previously had difficulty with, and had been asked to find ways to improve on - so we did, and showed the improvement.

We are now in a process of generating more comprehensive results and with additional forms of data presentation.  These should be ready to show at Vancouver, though it looks increasingly likely that we will have to participate remotely this time.  And of course we will specifically address the question over ECT(1) proposed by the Chairs.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 7 Mar 2020 10:50:58 +0200

> On 5 Mar, 2020, at 5:04 am, Wesley Eddy <[email-masked]> wrote:
> 

Following up on this point from the minutes:

> - Bob says all code has been available for some time.

We had checked on the morning of the call to find that no commits had apparently been made to the previously publicised L4S codebase since November.  Nevertheless, the L4S team's presentations and commentary appeared to rely on new work which clearly post-dated that.  It was subsequently established that the paced-chirping and classic-detection code in particular had *not* yet been published anywhere, and we have also determined that the queue-protection code is not implemented in the public Linux version of the DualQ qdisc.

Since these appear to be important components of the L4S system, can you point us towards where this code is available to evaluate?

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 7 Mar 2020 16:03:15 +0000

Joanathan,

For all L4S code, as said many times, pls always see links from the L4S 
landing page.
https://riteproject.eu/dctth/#code

Also, normally, if someone is really interested in using some code and 
cannot find it, they just ask.

See inline for specifics...

> We had checked on the morning of the call to find that no commits had apparently been made to the previously publicised L4S codebase since November.  Nevertheless, the L4S team's presentations and…
==Paced Chirping==
Paced Chirping has been published code since Mar 2019 at least (when we 
explained it at netdev). The link was in the netdev presentation, video, 
paper, etc. as well as off the L4S landing page, as well as IRTF iccrg 
presentations, mailing lists, etc:
https://github.com/JoakimMisund/PacedChirping

The reason the Paced Chirping repo is separate from TCP Prague is that 
it's intended to be useful for any congestion control (it solely uses 
delay), altho you'll probably have to do some work to integrate it into 
each CC.

Under there, you'll find that, in Feb Joakim integrated it into TCP 
Prague (rather than DCTCP):
https://github.com/JoakimMisund/net-next/tree/tcp_prague

==Classic ECN AQM detection==
As I said during the presentation, the code was not quite available for 
the interim, but we were going to publish it shortly. Olivier is now 
testing its inclusion in an integrated version of TCP Prague. And we are 
preparing publication of a large set of test results, so they should be 
rather more self-explanatory than they are right now.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Pete Heist <[email-masked]>
Date: Sun, 8 Mar 2020 12:07:38 +0100

> On Mar 7, 2020, at 5:03 PM, Bob Briscoe <[email-masked]> wrote:
> 
> Joanathan,
> 
> For all L4S code, as said many times, pls always see links from the L4S landing page.
> Also, normally, if someone is really interested in using some code and cannot find it, they just ask.

Any additional tests we provide will continue to use the ‘testing’ (default) branch at:

https://github.com/L4STeam/linux

This repo helps us identify what is ready for testing, and provides a single kernel where all the required pieces have been integrated together.

Pete

> Paced Chirping has been published code since Mar 2019 at least (when we explained it at netdev). The link was in the netdev presentation, video, paper, etc. as well as off the L4S landing page, as well as IRTF iccrg presentations, mailing lists, etc:
> The reason the Paced Chirping repo is separate from TCP Prague is that it's intended to be useful for any congestion control (it solely uses delay), altho you'll probably have to do some work to integrate it into each CC.
> 
> Under there, you'll find that, in Feb Joakim integrated it into TCP Prague (rather than DCTCP):
> As I said during the presentation, the code was not quite available for the interim, but we were going to publish it shortly. Olivier is now testing its inclusion in an integrated version of TCP Prague. And we are preparing publication of a large set of test results, so they should be rather more self-explanatory than they are right now.
> 
> 
> 
> 
> Bob
> 
>>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 25 Mar 2020 09:11:24 +0100

Hi All,

Summary: Focussing only on raw time courses or aggregate statistics is not going to cut it, looking at both/all levels is required.

more below,

> On Mar 6, 2020, at 18:27, Bob Briscoe <[email-masked]> wrote:
> 
> Wes,
> 
> 
> For those who might want to find these meeting materials in future without saving this specific URL, interim meetings are linked off the data tracker IETF proceedings page:

	[SM] I want to add, that comprehensive tests are obviously a good thing, but that by only looking at aggregate statistics and log-scales it is quite easy to miss catastrophic failures like the dualQ fairness breakdown at short RTTs. This result has been there pretty much from the beginning and yet it has not received enough attention, presumably because it was buried in big tomb of data and tests.
	Why do I keep harping on that so much? Because this failure indicates that dualQ can be gamed, any sufficiently unresponsive paced flow will be able to effectively slow everything else down to a trickle, and flows in the non-LL queue will not even be able to "fight back" with an equal amount of unresponsiveness. And IMHO that is considerably worse than the status quo, where equally responsive flows will end up sharing roughly equitable. Insisting that the failure as currently observed only happens in a "corner-case" completely ignores that (in adddition to the fact that that corner-case, short RTT flows is exactly where the CDNd internet has been moving closer and closer to).

	In short both comprehensive presentation of aggregate summary statistics for different cases as well as detailed presentations of time-resolved identified problematic cases seem appropriate.
	Regarding statistical reporting, in box whisker plots, as appropriate for expected non-normal data like delays, the mean is far less relevant than the median... also great idea to define what the whiskers are supposed to represent. The "figures of merit" idea, I believe is not unproblematic, as it tempts to only look and compare aggregate 2nd level statistics which can occlude what is actually happening at the "raw" data level. As always, one needs to also look at the raw time courses if only to avoid falling into common GIGO traps.

> 
> (The conversation started by audio, then continued on the chat). Time series are useful to dig into particular anomalous behaviours, but not really suitable for checking the operating range or to compare solutions over a range of scenarios. Higher percentile figures are essential for queue delay to determine how a real-time app would cope.

	[SM] Let's be clear, CDF type plots for delay are relevant and have merit, but the idea that you seem to be pushing here, that L4S will allow to deliver arbitrary tight real-time demands over the wider best-effort internet needs to be quashed, because it will not. Sure L4S might push down the average queueing delay over a short path down far enough to allow some tighter real-time demands than before... but it will not the end-all of queueing delay and jitter. The moment you replace the traditional bad >100ms queueing delay under load traffic shaper at the end-user/ISP transition all other variances < 100ms will start to matter...

> 
>> 
>> We are planning to get the WebEx recording posted to the IETF YouTube channel shortly, but need to do a little bit more work on that first.
> Thank you.
> Now I've seen some hints at the conversation I couldn't hear during the chairs slides (audio problems), I'll start following-up on the tsvwg list.
> 
> Cheers
> 
> 
> Bob
>> 
>> 
>> 
>> 
> 
> --