Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency: defaults ready for testing

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Tue, 17 Nov 2020 13:32:12 +0000

Hi all,

The RTT-independence was implemented, available and demonstrated several meetings ago already and as presented working very well according to our tests. The following parameters are now set as default, so can be tested out of the box:

All Prague flows with an RTT below 25ms will now converge to the same rate, independent of their real base RTT. This means that flows with a bigger RTT than 25 ms will never have to compete against smaller than 25ms RTT flows. 

Now the defaults are set, I'm looking forward to independent evaluations.

Hope this helps??
Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Lars Eggert
Sent: Tuesday, November 17, 2020 1:33 PM
To: Ingemar Johansson S <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency

Hi,

On 2020-11-17, at 13:46, Ingemar Johansson S <[email-masked]> wrote:
> I imagine that, as Prague is work in progress, then the original 
> authors of this mail thread could have considered to try out the 
> different options that address the RTT bias. Or at least consider to 
> re-run the experiment instead of digging down in long discussions on 
> what is default and not. Prague is work in progress after all, I don't 
> think that this is unreasonable to ask and it could have saved us a few roundtrips in in this mailthread ?

I've long ago given up trying to follow the blow-by-blow of this overall discussion. And maybe that means I've disqualified myself from having an opinion. But I don't quite know what to make of your statement.

If "Prague" (I guess you mean a TCP variant taking advantage of L4S?) is a "work in progress", why are we then pushing towards finalizing the L4S part of the overall scheme?

Maybe I'm naive here, but I always thought the queuing scheme and a TCP variant for it would necessarily go together. That TCP variant might be an initial design that maybe doesn't take full advantage of all the benefits the queueing scheme might offer, but surely it should demonstrate some significant benefit without significant downsides?

And we can of course have a discussion on whether RTT unfairness (or any other property or lack of one) is a significant downside or not, but I don't think we've had that discussion?

Basically, I'm trying to re-engage here for a bit to see if there is *any* hope for this entire area of work, if I should just ignore it again or even start arguing we should abandon it all, given how wedged the discussion between the camps has been for a long time.

Thanks,
Lars

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 17 Nov 2020 14:59:14 +0100

Hi Koen,

> On Nov 17, 2020, at 14:32, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi all,
> 

	[SM] Mmmh, so all my complaining about a 15ms hack resulted in you changing the number to 25ms... hardly an improvement, and hardly an increase in RTT-independence (which in general would imply a change that affects fairness at any RTTs).

> Now the defaults are set, I'm looking forward to independent evaluations.

	[SM] How is that going to meaningfully change the abysmal sharing between TCP Prage at 20 and at 160ms in DualPI2's shallow queue? Could you speculate how the imbalance is going to look in that case.

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 18 Nov 2020 02:28:05 +0200

> On 17 Nov, 2020, at 3:32 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

Since our tests are quite well automated, we were able to run a subset of them (all at 50Mbps) against the new defaults this evening.

I'll give you credit: there is some improvement in some of the tests.  However, we could still draw most of the same conclusions from the new data as we did from last week's data; the big-picture problems are still present and in some cases have actually deteriorated.

I'll focus on two major concerns in particular:

1: Prague outcompetes CUBIC in DualPI2, at a common baseline RTT.  This only stops being true when the BDP is large enough for Prague to have difficulty growing to steady state in a reasonable amount of time.

With the new code, the Jain's index improves from .823 to .987 at 10ms (the advantage in both cases being to Prague), but actually worsens from .880 to .838 at 20ms, and from .936 to .890 at 80ms.  All of these are sampled after allowing two minutes for the flows to converge to steady-state.

2: Prague vs Prague competition on differing RTTs.

Here is Figure 3 from the test report we recently posted, followed by an equivalent chart generated from the new data this evening.  Let's play spot the difference:

I can say that the throughput ratio for Prague vs Prague via DualPI2 is, in fact, slightly improved in the new data, but it is still significantly worse even than the 16:1 ratio expected from the baseline RTTs at identical average cwnd.  In a similar test with 80ms versus 20ms RTTs, the two Prague flows also have more than the expected 4:1 throughput ratio.  I don't have an immediate explanation for that.

Notice that with both the old and new code, CodelAF gets very close to parity in throughput with the same traffic load, and that even through DualPI2, a pair of CUBIC flows is closer to parity than a pair of Prague flows.  That is not, overall, an improvement in RTT independence from switching to TCP Prague and/or DualPI2.

However, we did find an improvement in fairness, compared to the older code, when comparing 20ms vs 10ms Prague flows.  That's what you were going for, wasn't it?  A shame that, in achieving that singular success, so many other things are left unresolved.

I'm sure we will have the opportunity to run more tests on your future efforts.  For the moment, with limited time on our hands, this will have to do.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 07:21:23 +0100

Hi Jonathan,

The changes that Oliver submitted to TCP Prague not only increase the fudge RTT to 25ms, but they also, as Koen mentioned, increase the time it takes for Prague to switch to the new fairness mode to 500 RTTs, or 5 seconds at 10ms RTT.
I really wonder how that affects fairness if the shallow queue is used by a bunch of short flows with 10ms RTT. As far as I can tell the consequences of this delayed engagement have not been described properly. 
@Koen, @Oliver, in the kernel code you hint at a paper about your RTT independence method. It would be great if you could post links to your analysis how this transition period affects sharing between flows inside the shallow queue and across queues. Ideally that data would show data for the old 100 RTT transition delay, as well for the new value of 500 RTTs.

My fear is that this will now give transient TCP Prague flows an undeserved advantage (for longer, that transition was part of the RTT independence code before), and just because that condition has not been tested by external parties yet, does not mean that it is no matter of concern. In fact that lack of external testing is rather cause for concern, as so far almost all external testing found problem spots in L4S almost immediately.
In fact the non-chalance in which these RTT independence parameters where changed apparently with an email to this list, and without any note that the functionality of these new parameters had actually been empirically verified.

Best Regards
        Sebastian

On 18 November 2020 01:28:05 CET, Jonathan Morton <[email-masked]> wrote:
>> On 17 Nov, 2020, at 3:32 pm, De Schepper, Koen (Nokia - BE/Antwerp)
><[email-masked]> wrote:
>> 
>> The RTT-independence was implemented, available and demonstrated
>several meetings ago already and as presented working very well
>according to our tests. The following parameters are now set as
>default, so can be tested out of the box:
>> 
>> All Prague flows with an RTT below 25ms will now converge to the same
>rate, independent of their real base RTT. This means that flows with a
>bigger RTT than 25 ms will never have to compete against smaller than
>25ms RTT flows. 
>> 
>Since our tests are quite well automated, we were able to run a subset
>of them (all at 50Mbps) against the new defaults this evening.
>
>I'll give you credit: there is some improvement in some of the tests. 
>However, we could still draw most of the same conclusions from the new
>data as we did from last week's data; the big-picture problems are
>still present and in some cases have actually deteriorated.
>
>1: Prague outcompetes CUBIC in DualPI2, at a common baseline RTT.  This
>only stops being true when the BDP is large enough for Prague to have
>difficulty growing to steady state in a reasonable amount of time.
>
>With the new code, the Jain's index improves from .823 to .987 at 10ms
>(the advantage in both cases being to Prague), but actually worsens
>from .880 to .838 at 20ms, and from .936 to .890 at 80ms.  All of these
>are sampled after allowing two minutes for the flows to converge to
>steady-state.
>
>Here is Figure 3 from the test report we recently posted, followed by
>an equivalent chart generated from the new data this evening.  Let's
>play spot the difference:
>
>
>
>
>I can say that the throughput ratio for Prague vs Prague via DualPI2
>is, in fact, slightly improved in the new data, but it is still
>significantly worse even than the 16:1 ratio expected from the baseline
>RTTs at identical average cwnd.  In a similar test with 80ms versus
>20ms RTTs, the two Prague flows also have more than the expected 4:1
>throughput ratio.  I don't have an immediate explanation for that.
>
>Notice that with both the old and new code, CodelAF gets very close to
>parity in throughput with the same traffic load, and that even through
>DualPI2, a pair of CUBIC flows is closer to parity than a pair of
>Prague flows.  That is not, overall, an improvement in RTT independence
>from switching to TCP Prague and/or DualPI2.
>
>However, we did find an improvement in fairness, compared to the older
>code, when comparing 20ms vs 10ms Prague flows.  That's what you were
>going for, wasn't it?  A shame that, in achieving that singular
>success, so many other things are left unresolved.
>
>I'm sure we will have the opportunity to run more tests on your future
>efforts.  For the moment, with limited time on our hands, this will
>have to do.
>
> - Jonathan Morton

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 18 Nov 2020 08:45:25 +0200

> On 18 Nov, 2020, at 8:21 am, Sebastian Moeller <[email-masked]> wrote:
> 

Yes, that engagement delay is clearly visible in a time-series plot.  I think it may even be more than 5 seconds in practice:

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 18 Nov 2020 06:58:52 +0000

Hi

The difference does not look extreme to me. Can't guarantee it but I suspect
that you'll find similar anomalies with e.g. rate based congestion controls.
We had all sorts of interflow fairness issues in the RMCAT work that ended
up in RFCs for NADA and SCReAM and L4S was not on the table then. 

Besides this, I don't want to kill this discussion, it is very useful. But.
it delves into congestion control research and perhaps ICCRG is a better
venue for this ? 

/Ingemar

From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
Sent: den 18 november 2020 07:45
To: Sebastian Moeller <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency:
defaults ready for testing

On 18 Nov, 2020, at 8:21 am, Sebastian Moeller <[email-masked]
<mailto:[email-masked]> > wrote:

The changes that Oliver submitted to TCP Prague not only increase the fudge
RTT to 25ms, but they also, as Koen mentioned, increase the time it takes
for Prague to switch to the new fairness mode to 500 RTTs, or 5 seconds at
10ms RTT.
I really wonder how that affects fairness if the shallow queue is used by a
bunch of short flows with 10ms RTT. As far as I can tell the consequences of
this delayed engagement have not been described properly. 

Yes, that engagement delay is clearly visible in a time-series plot.  I
think it may even be more than 5 seconds in practice:

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 18 Nov 2020 07:03:06 +0000

With a slight errata.. SCReAM is not rate based. It is more of an unholy bastard baby of window based and rate based congestion control 😐

/I

From: Ingemar Johansson S <[email-masked]> 
Sent: den 18 november 2020 07:59
To: Jonathan Morton <[email-masked]>; Sebastian Moeller <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>; Ingemar Johansson S <[email-masked]>
Subject: RE: [tsvwg] new tests of L4S RTT fairness and intra-flow latency: defaults ready for testing

Hi

The difference does not look extreme to me. Can’t guarantee it but I suspect that you’ll find similar anomalies with e.g. rate based congestion controls. We had all sorts of interflow fairness issues in the RMCAT work that ended up in RFCs for NADA and SCReAM and L4S was not on the table then. 

Besides this, I don’t want to kill this discussion, it is very useful. But… it delves into congestion control research and perhaps ICCRG is a better venue for this ? 

/Ingemar

From: tsvwg <[email-masked] <mailto:[email-masked]> > On Behalf Of Jonathan Morton
Sent: den 18 november 2020 07:45
To: Sebastian Moeller <[email-masked] <mailto:[email-masked]> >
Cc: tsvwg IETF list <[email-masked] <mailto:[email-masked]> >
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency: defaults ready for testing

On 18 Nov, 2020, at 8:21 am, Sebastian Moeller <[email-masked] <mailto:[email-masked]> > wrote:

 

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 18 Nov 2020 09:14:01 +0200

> On 18 Nov, 2020, at 8:58 am, Ingemar Johansson S <[email-masked]> wrote:
> 
> The difference does not look extreme to me.

It's a persistent and stable 2:1 throughput ratio for about 5-10 seconds after flow startup, followed by a reasonably sharp switch to near parity.  I didn't say it was extreme, only confirmed for Sebastian's benefit that it was clearly noticeable in the data.

> Can’t guarantee it but I suspect that you’ll find similar anomalies with e.g. rate based congestion controls. We had all sorts of interflow fairness issues in the RMCAT work that ended up in RFCs for NADA and SCReAM and L4S was not on the table then. 
>  

I'm inclined to agree, except that the RTT independence requirement appears in TSVWG adopted drafts, and TCP Prague is presently the only CC available to test compliance.

What we're showing is that DualPI2 is not sufficient by itself to implement that particular requirement, and arguably makes matters worse than either the typical status quo (dumb FIFO) or the present state of the art (FQ-AQM or AF-AQM).  If all discussion of TCP Prague is to move to ICCRG, then what we will discuss in TSVWG is DualPI2's many shortcomings instead.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 08:37:52 +0100

Hi Jonathan,

thanks a lot.

> On Nov 18, 2020, at 08:14, Jonathan Morton <[email-masked]> wrote:
> 

	[SM] Thanks for confirming my intuition. 1:2 might not be terrible in itself, but since it seems to not be random, but systematic, I am not confident that this will not be trivially exploitable to persistently send data with a noticeable rate advantage. On the otherhand, the whole issue could be marketed as intended boost for short flows, but that fails to take into account that this only help short flows with short RTT.

Best Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 18 Nov 2020 11:13:29 +0000

Hi Jonathan, Sebastian,

Thanks for sharing these tests results again.

Indeed, a detailed study can be performed to define the optimal behavior. It probably will also depend on the applications using it. I think it is commonly accepted that it is beneficial to prioritize short flows and that we don't want to spoil the benefit of having a lower latency  (being able to respond faster) we only enabled the RTT fairness after a currently reasonable and easy to implement number of RTTs.  Improvements could be to do this more gradually, ... Also if applications stop sending data, and start again, it can be considered to restart the (gradual) RTT independence.

I guess the basic issue (too aggressive Short RTT flows due to the 0 to 1ms L4S queue latency target) is solved with this, and I encourage everyone to share data/experience, further study and define more optimal behavior. We can easily change this.

Regards,
Koen.

From: Jonathan Morton <[email-masked]>
Sent: Wednesday, November 18, 2020 7:45 AM
To: Sebastian Moeller <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency: defaults ready for testing

On 18 Nov, 2020, at 8:21 am, Sebastian Moeller <[email-masked]<mailto:[email-masked]>> wrote:

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 18 Nov 2020 11:56:44 +0000

Hi Jonathan, Pete, Sebastian,

Thanks for also showing this case sharp again. The reason for this worse longer RTT behavior of Prague and DCTCP is that the smallest flow induces the 1-RTT-on/X-RTT-off burst pattern on the step threshold of the L4S queue, which is known to trigger the r=2/(p^2*RTT) DCTCP response (as defined in the original DCTCP paper). L4S started when we found that DCTCP behaves as r=2/(p*RTT), so the square on p disappears, when it gets the continuous per RTT-stable marking rate from a Classic AQM (when it gets coupled). Any in between burstiness will result in a value somewhere in between those equations.

So when the 1-RTT-on/X-RTTs-off bursts of the smallest flow become a steady per RTT marking rate for the biggest RTT flow, it will have in worse case an extra 1/p disadvantage. The second paper described this behavior as a 1/RTT^2 dependency, but I think it is more related due to this effect. Neal Cardwell recently suggested looking into the more fluid per packet implementation to eliminate the RTT-dependence behavior, which I thought was not relevant as it "only" solves the getting the 1/RTT^2 to 1/RTT, and so not a solution for 1/RTT^0 objective for RTT-independence. But it would actually solve this "Too big rate disadvantage for long >80ms RTTs" problem. So currently to be tried and further investigated. Further contributions are appreciated there as well (from anyone).

As I understood there is an expectation to have a CC that can be just enabled and autotunes to all conditions. As a fast work-around to make selecting Prague as a good alternative in the full range, we are considering switching to non-ETC or ECT(0) and Reno when the RTT is >80ms. Even better, but a bigger code impact, would be to switch to Cubic in that case. Olivier can push the fallback to Reno if RTT>80ms, which for now would solve this issue until there is any of the better solutions available.

Regards,
Koen.

From: Jonathan Morton <[email-masked]>
Sent: Wednesday, November 18, 2020 1:28 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Lars Eggert <[email-masked]>; Ingemar Johansson S <[email-masked]>; tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] new tests of L4S RTT fairness and intra-flow latency: defaults ready for testing

On 17 Nov, 2020, at 3:32 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]<http://labs.com>> wrote:

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 14:57:19 +0100

Hi Koen,

> On Nov 18, 2020, at 12:56, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> So when the 1-RTT-on/X-RTTs-off bursts of the smallest flow become a steady per RTT marking rate for the biggest RTT flow, it will have in worse case an extra 1/p disadvantage. The second paper described this behavior as a 1/RTT^2 dependency, but I think it is more related due to this effect. Neal Cardwell recently suggested looking into the more fluid per packet implementation to eliminate the RTT-dependence behavior, which I thought was not relevant as it “only” solves the getting the 1/RTT^2 to 1/RTT, and so not a solution for 1/RTT^0 objective for RTT-independence.

	[SM] I do not buy  this rationale at all. Going from 1/RTT^2 to 1/RTT would be a tremedous reduction in RTT dependence, to big to argue you left this on the floor as you aimed for 1/RTT^0 (and failed). On the other hand that approach would explain a lot of what I object in L4S.

> But it would actually solve this “Too big rate disadvantage for long >80ms RTTs” problem. So currently to be tried and further investigated. Further contributions are appreciated there as well (from anyone). 
>  
> As I understood there is an expectation to have a CC that can be just enabled and autotunes to all conditions. As a fast work-around to make selecting Prague as a good alternative in the full range, we are considering switching to non-ETC or ECT(0) and Reno when the RTT is >80ms.

	[SM] That is an open admission that L4S really is just a fast-priority-lane design. Thanks for confirming that though. It does severely reduce the "low latency for all" claim on which L4S is still marketed.

> Even better, but a bigger code impact, would be to switch to Cubic in that case. Olivier can push the fallback to Reno if RTT>80ms, which for now would solve this issue until there is any of the better solutions available.

	[SM] It is exactly this pilling up of hacks to solve individual undesirable outcomes one by one, that leave me unconvinced whether the core "coupled AQMs" idea actually is suitable for the real internet.

Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 15:00:08 +0100

Hi Koen,

> On Nov 18, 2020, at 12:13, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Jonathan, Sebastian,
>  
> Indeed, a detailed study can be performed to define the optimal behavior. It probably will also depend on the applications using it. I think it is commonly accepted that it is beneficial to prioritize short flows and that we don’t want to spoil the benefit of having a lower latency  (being able to respond faster) we only enabled the RTT fairness after a currently reasonable and easy to implement number of RTTs.  Improvements could be to do this more gradually, … Also if applications stop sending data, and start again, it can be considered to restart the (gradual) RTT independence.
>  
> I guess the basic issue (too aggressive Short RTT flows due to the 0 to 1ms L4S queue latency target) is solved with this,

	Is it? We still see a consistent bias towards Prague over CUBIC, all we need to do is set the low RTT to 26ms... IMHO you "solved" that one test case, but you did not solve the root cause of the issue.

> and I encourage everyone to share data/experience, further study and define more optimal behavior. We can easily change this.

	[SM] As expected I disagree. The proof will basically be whether any other protocol aiming for L4S compliance will pick up that hack or not. Since a WG last call seems inevitable we can revisit this discussion in say 2 years.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 15:23:00 +0100

Hi Jonathan,

to come back to this.

> On Nov 18, 2020, at 01:28, Jonathan Morton <[email-masked]> wrote:
> 

	[SM] Would be interesting to see 20/10, as well as 26/13 with the newly changed code... My hunch is that 26/13 should look more different from 20/10 than the additional 6ms (30%) RTT offset would lead on to believe.

Best Regards
	Sebastian