Subject: Re: [tsvwg] Question regarding acceptable sharing behavior between "normal traffic" and L4S traffic --> "see These L4S issues reported are not show stoppers"

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 18 Nov 2019 11:36:13 +0000

Hi Sebastian,

See mail "These L4S issues reported are not show stoppers". In short: the TCP-Prague requirement "less RTT dependency" covers this. 
This effect has been reported from the beginning. Not everyone is convinced this is even a problem, rather a feature, as it is the "Normal Internet behavior since 40 year".

Regards,
Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
Sent: Monday, November 18, 2019 10:18 AM
To: tsvwg IETF list <[email-masked]>
Subject: [tsvwg] Question regarding acceptable sharing behavior between "normal traffic" and L4S traffic

Dear All,

I have been making an argument repeatedly on this list in different threads to which I received inconsistent feed-back. So, I would very much like to raise this issue explicitly to the full list, as I believe this to be important to address in regards to L4S.

It has been observed by two independent measurements that the reference dual queue AQM demonstrates RTT-dependent unfairness between L4S and standard TCP traffic, the shorter the RTT the more of a bandwidth-advantage L4S gets.
At a nominal RTT of 0ms (which as far as I can tell just means no additional netem delay, so a real RTT in the <1ms range) TCP Prague gets ~5-times more bandwidth than TCP cubic:
The L4S team's measurement:
https://l4s.cablelabs.com/l4s-testing/key_plots/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_var.png
The SCE-team's measurement:
https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/l4s-s1-2/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_fixed.png

That leads to my questions:

A) Is/was everybody aware of this unequal sharing over a nominally equal path? 

I ask because the following two statements from the L4S-arch and the dualq-coupled I.D.'s seem to indicate a different kind of more equal sharing to me:

https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10 :
"This specification defines `DualQ Coupled Active Queue Management
  (AQM)', which enables these Scalable congestion controls to safely
  co-exist with Classic Internet traffic.
Analytical study and implementation testing of the Coupled AQM have
   shown that Scalable and Classic flows competing under similar
   conditions run at roughly the same rate."

and

https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-04 :
"Further, the network part is simple to
  deploy - incrementally with zero-config.  Both parts, sender and
  network, ensure coexistence with other legacy traffic."

In the limit, the dual queue AQM will share traffic between the L4S queue and the normal queue approximately 15:1. Unless coexistence is considered to mean "does not starve" I fail to understand how claims and observed data match up.

B) Does everybody here think that this what is implied by the above stanzas and that this is an acceptable guarantee to merit allowing/endorsing the use of dual queue AQM over the internet?

	Oliver Tilmans helpfully explained, that this behavior is draft-compatible but relies on a specific interpretation of "under similar conditions", once the queue under load is taken into consideration the RTT ration goes from 1:1 to 1:15 and hence the observed bandwidth sharing is considered acceptable due to conditions not being similar enough any more. 

I have two issues with that rationale:

1) this does not seem to be the natural way to read that claim (Oliver mentioned that he had the same question/observation when he started to look into dualq, so my reading of the text is not completely outlandish)

2) It is to a large degree a consequence of a design decision in the dual queue AQM that does not seem to have been sufficiently thoroughly considered.
	The point is that dual queue simply copies the recommendation to set the target for the "normal queue's" pie instance to 15 milliseconds (simply copying from QDELAY_REF in https://tools.ietf.org/html/rfc8033#appendix-B) without taking the consequence into consideration how this affects bandwidth sharing behavior between the two queues at short RTTs. According to codel's theory (https://tools.ietf.org/html/rfc8289#section-4.3) target should be set to ~5-10% of the interval, so instead of 15 ms, 5 ms probably should do to maintain bandwidth utilization while at the same time addressing part of the root cause for unequal sharing at low RTTs. It is well possible that 5ms will not wrk for a PIE derivative shaper like dualQ, but this should at least be explored before designing an undeserved advantage for L4S into the specifications.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 18 Nov 2019 15:17:42 +0100

Hi Koen,

> On Nov 18, 2019, at 12:36, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	Koen, I believe you might want to re-read my email. I am not arguing about RTT dependence per se, but rather whether L4S meets the consensus "coexisting conditions" to merit roll-out into the internet, given that effectively it will build a low latency, higher bandwidth route between end hosts at the expense of both latency and bandwidth of non-participating flows. 
	Due to the unfortunate design decision to employ a loose coupling between the two queues instead of stricter isolation the artificially inflated RTT for a for normal flows (that dualq introduces under saturating load) puts them a distinct disadvantage in regards to L4S flows, the question hence is:

Does the TSVW community believe that this is acceptable, to declare a new design as "safely coexisting" and "ensur[ing] coexistence" if all that is guaranteed is not to starve the existing traffic?
 
IMHO neither the observed behavior, nor arguing the objections away instead of looking at the fundamental root cause that behavior strikes me as the most efficient/productive possible way forward.
I do want to add, that I even gave a pointer about how the issue might be ameliorated.
	But is it a correct representation of your position that demonstrated unequal sharing  is as designed and hence will not be reconsidered for improvements?

Best Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 18 Nov 2019 15:33:06 +0000

Hi Sebastian,

>> But is it a correct representation of your position that demonstrated unequal sharing  is as designed and hence will not be reconsidered for improvements?

I did understand your worry, and to be clear this is exactly also my worry. So no, I'm not OK with this unequal sharing. My position is the same as yours. The only difference is that for L4S we need to solve this in the End-System.

>> I am not arguing about RTT dependence per se

Please read the following to understand the relevance:

A.1.5.  Reduce RTT dependence

   Description: A scalable congestion control needs to reduce or
   eliminate RTT bias over as wide a range of RTTs as possible, or at
   least over the typical range of RTTs that will interact in the
   intended deployment scenario.

   Motivation: Classic TCP's throughput is known to be inversely
   proportional to RTT, so one would expect flows over very low RTT
   paths to nearly starve flows over larger RTTs.  However, Classic TCP
   has never allowed a very low RTT path to exist because it induces a
   large queue.  For instance, consider two paths with base RTT 1ms and
   100ms.  If Classic TCP induces a 100ms queue, it turns these RTTs
   into 101ms and 200ms leading to a throughput ratio of about 2:1.
   Whereas if a Scalable TCP induces only a 1ms queue, the ratio is
   2:101, leading to a throughput ratio of about 50:1.

   Therefore, with very small queues, long RTT flows will essentially
   starve, unless scalable congestion controls comply with this
   requirement.

It is how the "Reduce RTT dependence" TCP-Prague requirement is described in the L4S-ID draft. Isn't it describing the exact test case that you also claim as an issue?

The source of this problem and (I'm happy to say) "our" worry, is RTT dependence, for which I mean that RTT is at the bottom part of the fraction in TCP's throughput equations. If flows get the same throughput independent of the RTT they experience, this issue disappears, right? This is the solution from an end-system perspective. Obviously we cannot make it as good as an FlowQueue-scheduler in the network (which is actually also not perfectly working under all circumstances). Also DualQ cannot change the behavior of Classic flows with a big RTT (which Cubic is solving partly).

>> given that effectively it will build a low latency, higher bandwidth route between end hosts at the expense of both latency and bandwidth of non-participating flows

This is indeed the problem that our experiments showed early, and as far as I remember this issue was discussed from the beginning even in meetings at the mike when I presented/warned for this issue.
My presentation pitch/insight was always that creating big buffers, is the current network's solution to solve the RTT-dependence of TCP congestion controls. By putting AQMs with smaller buffers we have to compromise on bigger RTT unfairness. The only reason why FQ-CoDel can get away with a 5ms threshold is indeed because it compensates the small buffer with FQ. PI2 could as well work with a target of 1ms wouldn't it be that this would also hurt extremely the longer RTT flows. If you want you can do an FQ_PI2 (a PI2 instance per flow-queue) with a target of 5ms or even 1ms. Never tried it, but I expect that it would work better than FQ_CoDel. Also I think you can setup an experiment with multiple bottlenecks where FQ_CoDel causes the same extreme effect. You need to make again combinations of FIFO and FQ_CoDel type nodes.

To solve this in congestion controls we should not expect perfection again. If you want (near) perfection then deploy FQ 😉. I'm happy with an approximation / good compromise. I think increasing RTT independence on the time the flow is getting frequent CE signals is a good goal. When a flow just starts, you want to benefit from the low RTT, to get fast up to speed and terminate quickly short flows. When you are at speed, you want to converge to the slower converging bigger RTT flows.

Regards,
Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Monday, November 18, 2019 3:18 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] Question regarding acceptable sharing behavior between "normal traffic" and L4S traffic --> "see These L4S issues reported are not show stoppers"

Hi Koen,

 

Best Regards
	Sebastian

> https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T09
> 0559-r2/l4s-s1-2/batch-l4s-s1-2-cubic-vs-prague-50Mbit-0ms_fixed.png
> 
> "Further, the network part is simple to  deploy - incrementally with 
> zero-config.  Both parts, sender and  network, ensure coexistence with 
> other legacy traffic."
> 
> 1) this does not seem to be the natural way to read that claim (Oliver 
> mentioned that he had the same question/observation when he started to 
> look into dualq, so my reading of the text is not completely 
> outlandish)
>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 19 Nov 2019 08:53:58 +0100

Hi Koen,

> On Nov 18, 2019, at 16:33, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] Thank you for the clarification, great that we agree that this is sup-optimal behavior. We differ, I assume in the assessment of the severity though. IMHO this is a failure to meet required behavior for release of dual queue AQM into the wider internet, in other words show stopper, plain and simple. Now, that is not my decision to make, and I do not want to claim otherwise, but IMHO this issue alone should send the dual queue AQM back to the drawing board.

> It is how the "Reduce RTT dependence" TCP-Prague requirement is described in the L4S-ID draft. Isn't it describing the exact test case that you also claim as an issue?

	[SM] I read this before and am not opposed to this, but IMHO this is orthogonal to the fact that the dual queue AQM fails to deliver its core functionality, robust isolation of the two queues and equitable sharing between them. As I understand the L4S arch internet draft, you basically agree that strict separation is a non-negotiable requirement.

> The source of this problem and (I'm happy to say) "our" worry, is RTT dependence, for which I mean that RTT is at the bottom part of the fraction in TCP's throughput equations. If flows get the sam…

	[SM] The last is important, dualQ can implement stricter more equitable sharing between the two classes. There are at least two obvious toggles to explore, both the target for the "normal TCP" queue as well as the scheduling weight between the two queues, given that this issue is not new, I am really puzzled that there should be no parametric exploration of how at least these two obvious parameters affect the worst-case sharing behavior. IMHO this should be fully explored before considering to advance the dualQ I.D. to experimental rfc...
	Now in the spirit of offering a way forward, I would propose you implement the isolation by a simple priority scheduler/shaper that gives up to 50% of bandwidth (hard limit) to L4S flows and the rest to "normal traffic". That will do the following:
a) it will make sure that even at short RTTs (as are quite common for normal internet usage due to proliferation of CDNs) L4S and normal traffic will share bandwidth equitable

b) and hence allows you to carry your experiment into the internet, giving you time to re-design the dual queue AQM to change the obvious wrong behavior.

I believe this to be a reasonable compromise in which a potential failure to find a solution is not paid by the rest of the internet, but by the "offending" party.

> This is indeed the problem that our experiments showed early, and as far as I remember this issue was discussed from the beginning even in meetings at the mike when I presented/warned for this issu…

	[SM] Well, similar to the choice of "classification heuristic" I am not convinced that the right compromise was found. How much works has been done to try to fix this issue in the mean time?

> My presentation pitch/insight was always that creating big buffers, is the current network's solution to solve the RTT-dependence of TCP congestion controls. By putting AQMs with smaller buffers we have to compromise on bigger RTT unfairness.

	[SM] I agree, but it is one thing to do this inside a traffic class (for lack of a better wird) and a completely different to do this between L4S two queues. The first one is har to avoid with any AQM, the second one purely a design choice.

> The only reason why FQ-CoDel can get away with a 5ms threshold is indeed because it compensates the small buffer with FQ.

	[SM] I disagree, I cited the codel RFC on purpose, not the fq_codel one, the set target 5-10% of interval rule follows from TCP behavior, and is not a property of codel or fq_codel (both by default use 5ms), so 5ms should work pretty much as well for PIE. I looked though the dualQ ID, the PIE rfc and the PIE research paper referenced in the PIE RFC, nowhere there is a stringent explanation why the reference delay is set like it is (5 and 20ms in the PIE paper, 15ms in the PIE rfc, 15ms in the dualQ id).
If you have data showing that 5ms will not work with PIE/dualQ please share. Otherwise, feel free to explore whethjer my proposal helps to remedy the sharing issue.

> PI2 could as well work with a target of 1ms wouldn't it be that this would also hurt extremely the longer RTT flows.

	[SM] Sure, granted. But again that is a design choice, nobody forces you to use one and the same target and interval for all flows, you could adopt that to the RTT of each flow. Sure that is not 100% realistic, but it shows that we are talking about design decisions here, that are open for re-design and exploration.

> If you want you can do an FQ_PI2 (a PI2 instance per flow-queue) with a target of 5ms or even 1ms. Never tried it, but I expect that it would work better than FQ_CoDel. Also I think you can setup an experiment with multiple bottlenecks where FQ_CoDel causes the same extreme effect. You need to make again combinations of FIFO and FQ_CoDel type nodes.

	[SM] Possible, but in reality using say HTB+fq_codel after a fifo bottleneck works pretty well in reality, I am not saying that there is no room to improve this, but it already works wonders, and that is basically with the fq AQM on the wromg end of the bottleneck. This is, to repeat myself one of the configurations latency sensitive users already use successfully today. But that is a bit besides the point, I am not arguing for an FQ solution, but just that your solution is made compliant with your own claims and requirements so that is is safe to be tested over the internet.
	I have severe doubts that L4S over the internet will work as desired, but I am fine to be convinced otherwise, assuming L4S plays nicely with the existing internet traffic.

> 
> To solve this in congestion controls we should not expect perfection again.

	[SM] Oh, I agree, the goal is not perfection, but simply good enough. IMHO neither TCP Prague nor the dual queue AQM as they stand today are good enough though and should be given time to mature.

> If you want (near) perfection then deploy FQ 😉. I'm happy with an approximation / good compromise. I think increasing RTT independence on the time the flow is getting frequent CE signals is a good goal. When a flow just starts, you want to benefit from the low RTT, to get fast up to speed and terminate quickly short flows. When you are at speed, you want to converge to the slower converging bigger RTT flows.

	[SM] I am not opposed to any of these arguments, but I am not sure that they have too much bearing on the fact that dual queue AQM does not work as intended to day.

Best Regards
	Sebastian