Subject: Re: [tsvwg] RTT-independence

From: <[email-masked]>
Date: Tue, 11 Feb 2020 07:57:02 +0000

Bob,

your argument to the point below is valid. There’s a “working point”, to which a queue-depth (and AQM, if present) of a network provider access router is configured. If conditions significantly differ from those assumed for the working point, either transport throughput for all or a share of flows will deteriorate (or buffer might be underutilized).

I’d appreciate if test buffer depths, AQM configuration (if present) and RTTs are chosen reasonable to the above regard. That doesn’t mean, RTTs or which aren’t prevalent today shouldn’t be considered. But RTTs should be picked or mixed so that the overall test design including queue-configuration is operating in a mode to be expected with such a working point in mind.

I don’t volunteer to pick one of these advanced test scenarios. I note that 5G comes with partially low RTT expectations (10 ms in the most demanding case, I guess). I’m not a wireless network or wireless architecture expert however.

Regards,

Ruediger

<snip>

The L4S proponents argue, that this is working as designed, as this is a side-effect of their under-explained choices of acceptable standing queue "targets" for the two queues (1ms for the L4S-queue and 15ms for the non-L4S queue), which for short path RTTs in the limit creates an RTT imbalance of 1:15. (I have also argued that 15ms is the wrong value in theoretical grounds and so far this has not been disproven with facts/data).
15ms comes from the amplitude of the sawteeth of typical traffic at typical Internet bottlenecks over typical paths. "Traffic" might be multiple flows or single flows, but an AQM has to work well for the cases where the user wants to use the bandwidth they are paying for, including for single long-lived flows (OS updates, video uploads, etc). "Traffic" might use Cubic, Reno or other CCs but the user doesn't control the CC the other end uses, so the AQM has to work for typical cases and not be awful for worst cases. "Bottlenecks" might be anywhere, but they are most often in access links where there is not much flow multiplexing. With Reno, Cubic etc, the sawtooth amplitude depends on the RTT of the path. So "paths" might be between a customer in a large urban conurbation and a CDN in the same or a nearby conurbation, or a path might be half way round the globe, but the common case these days is an urban customer and a CDN.

One can argue whether the number is 10ms, 15ms, 20ms or whatever, but there is an operating point below which more and more traffic badly underutilizes the link. Whatever, this is not a theoretical question, it is a commercial question for ISPs when setting the target parameter of their AQM. You have to look at what ISPs do, not what you think they should do.

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 11 Feb 2020 13:53:54 +0200

> 15ms comes from the amplitude of the sawteeth of typical traffic at typical Internet bottlenecks over typical paths. 

For this to be true, as a peak-to-peak measure, while maintaining 100% throughput, the baseline RTT would have to be 15ms (for basic Reno) or 35ms (for CUBIC as implemented in Linux).  This does not square with other research which has established typical Internet path RTTs in the region of 80ms.

NB: Codel has an even lower queue target of 5ms by default, but this is for controlling the *standing* queue, not peak nor average queue depth.  Some reduction in link utilisation is accepted in the design, in service of keeping the queue short.

In any case, from a scientific perspective, when testing a claim that a given system is less RTT-dependent than the status quo, it is of course necessary to choose a variety of test conditions which are capable of exposing this property.  Tests to date actually suggest *greater* RTT dependence of the L4S system compared to conventional TCPs with conventional AQM, largely because of the moderating effect (with a very short baseline RTT) of keeping a modest standing queue which L4S specifically avoids.

It is this evidence which the L4S team must defend against on this particular claim.  I think Sebastian is asking important questions here.

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Tue, 11 Feb 2020 20:32:53 +0000

[GW]  It's not a peak-to-peak measure.  The PIE or PI2 controller aims to keep the "average" queue delay at the target value of 15ms.  This means that for a single Reno flow to fully utilize the link, its baseline RTT needs to be 30ms or less, and cubic more like 50ms or less.  Keep in mind that cubic is also a bit less sensitive to under-buffering than Reno.  You can quibble with his wording if you like, but setting the latency target in existing AQMs comes down to (as Bob indicated) a tradeoff between minimizing occasions where the link is underutilized and minimizing delay.  I'm not aware of a specific goal of ensuring 100% utilization for a single 80ms base-RTT flow.

[GW] It has been argued that way, but I'm not sure that argument actually holds up.  CoDel only aims to ensure that the minimum queue delay over the last interval (100ms) is below the target value.  Van and Kathie's definition of "standing" queue (or "bad" queue) was queue delay that lasts for more than about one RTT, and their description of why they landed on that definition appears to relate to handling stretch ACKs and other sources of burstiness rather than dealing with congestion control dynamics.  The other definition of standing queue (the queue to eliminate if 100% utilization for a single flow is desired) is the remaining queue at the trough of the TCP sawtooth.  These two definitions only align when the sawtooth period is less than 100ms. As a result, CoDel only supports 100% link utilization in the single flow case when that is true.  Since the period of the sawtooth for Reno or Cubic can be measured in seconds (or minutes) for typical broadband speeds and RTTs today, this doesn't really work as described, and CoDel reverts to more-or-less acting as a short (5ms-ish) drop-tail queue.  For 100Mbps links, CoDel only supports 100% utilization for RTTs less than ~3.5ms.  For 1Gpbs links, CoDel only supports 100% utilization for RTTs less than ~1ms.  Sebastian has mentioned a few times that there is a theoretical basis for the 5ms target in CoDel.  I've not seen one, and wonder if someone could point me where to find it. 

    
     - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 11 Feb 2020 23:23:13 +0100

> On Feb 11, 2020, at 21:32, Greg White <[email-masked]> wrote:
> 
> [GW]  It's not a peak-to-peak measure.  The PIE or PI2 controller aims to keep the "average" queue delay at the target value of 15ms.  

	[SM] The peak-to-peak interpretation is trying to make some sense out of Bob's "15ms comes from the amplitude of the sawteeth of typical traffic at typical Internet bottlenecks over typical paths. [...] the sawtooth amplitude depends on the RTT of the path" but to me this highlights that the rationale for the choice of the numerical value of 15ms is missing in the DQCAQM draft

The "history" of the target delay//ref_del/RTT_typ goes something like this:

1) "The" PIE paper (Pan, et al., "PIE:  lightweight control scheme to address the bufferbloat problem", IEEE HPSR, DOI 10.1109/HPSR.2013.6602305, 2013, https://www.researchgate.net/publication/261134127_PIE_A_lightweight_control_scheme_to_address_the_bufferbloat_problem) introduced the parameter ref_delay as follows:
"We present here a lightweight design, PIE (Proportional Integral controller Enhanced), that can effectively control the average queueing latency to a reference value"
and showed data acquired with ref_del set to 20, 15 and 5ms. It gives no guidance as to how to select the actual value and makes no judgement about which is better.

2) The PIE RFC (https://tools.ietf.org/html/rfc8033) calls the parameter QDELAY_REF or AQM Latency Target and recommends a default of  15 milliseconds:
"The target latency value, QDELAY_REF, SHOULD be set to 15 milliseconds"
without any further justification for that recommendation

3) The DOCSIS PIE RFC (https://tools.ietf.org/html/rfc8034) says the following about the Latency Target:
"4.1.  Latency Target
  The latency target (a.k.a. delay reference) is a key parameter that
  affects, among other things, the trade-off in performance between
  latency-sensitive applications and bulk TCP applications.  Via
  simulation studies, a value of 10 ms was identified as providing a
  good balance of performance.  However, it is recognized that there
  may be service offerings for which this value doesn't provide the
  best performance balance.  As a result, this is provided as a
  configuration parameter that the operator can set independently on
  each upstream Service Flow.  If not explicitly set by the operator,
  the modem will use 10 ms as the default value."

4) The DQCAQM draft (https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-09) renames the parameter in its pseudo-code to RTT_typ and states:
"for the PI2 algorithm (Appendix A) the queuing delay target is set to the typical RTT;"
without expanding what a typical RTT denotes exactly, and how RTT and target delay depend on each other.
IMHO that makes not too much sense theoretically as that parameter describes the amount of queueing delay the PIE instance will aim for, which is only partly depending on a flow's RTT.

IMHO this "history" shows that there is no clear theoretical underpinning/justification of any of the values in these four documents. I am not saying that there isn't any, but it would behoof the PIE proponents to come forward with a theoretical rationale or to present test data demonstrating that 15ms is a magical number... superior to all others or even a just good compromise.

> This means that for a single Reno flow to fully utilize the link, its baseline RTT needs to be 30ms or less, and cubic more like 50ms or less.  Keep in mind that cubic is also a bit less sensitive to under-buffering than Reno.  You can quibble with his wording if you like, but setting the latency target in existing AQMs comes down to (as Bob indicated) a tradeoff between minimizing occasions where the link is underutilized and minimizing delay.  I'm not aware of a specific goal of ensuring 100% utilization for a single 80ms base-RTT flow.

	[SM] We all agree that utilization versus latency-under-load-increase are the trade-offs of selecting a target delay, the question is why is the L4S-team both adamant that 15ms is the correct number and at the same time incapable of giving a reasoned justification of how the numeric value was reached. Of all the descriptions, I find the one in the DQCAQM draft the most forced one, RTT_typ, typical RTT is simply not how ref_del works, this is similar to codel the amount of queue delay under load that PIE will aim for (yes there are differences between codel's target and PIE's ref_delay, but essentially both are setpoints that directly affect long term queue duration).

>    NB: Codel has an even lower queue target of 5ms by default, but this is for controlling the *standing* queue, not peak nor average queue depth.  Some reduction in link utilisation is accepted in…

	[SM] Question: with PIE the long term average median queue length will be ref-delay and with codel/fq_codel it will be close to target, so do you see any reason, why 5ms will not also work for PIE? Looking at the PIE paper it looks like PIE at a target of 5ms ref_delay has slightly higher utilization cost than Codel but we are talking PIE is 96.6% vs. CoDel is 98.4%, IMHO both well acceptable.

> 
> [GW] It has been argued that way, but I'm not sure that argument actually holds up.  CoDel only aims to ensure that the minimum queue delay over the last interval (100ms) is below the target value.  

	[SM] testing under load shows that for fq_codel the median latency-under-load-increase for an unrelated measurement flow is close to either one or, for bi-directionally saturating traffic, two target's worth of milliseconds. So empirically it looks like Codel/fq_codel under load will give you a queues size that is mostly bound by target. Under overload conditions that does not hold anymore, but see fq_codel for a viable way to ameliorate that issue to some degree.

> Van and Kathie's definition of "standing" queue (or "bad" queue) was queue delay that lasts for more than about one RTT,

	[SM] No, it is not standing queue per se that is bad, but standing queue in excess of target that triggers Codel's signaling, in effect with well-behaved flows this ends up with an median queueing latency of target time units. And fq_codel then solves the well-behaving requirement, by isolation well and badly behaving flows such, that each flow mostly reaps what it sows itself, same principle appliies to fq_pie, by the way..

> and their description of why they landed on that definition appears to relate to handling stretch ACKs and other sources of burstiness rather than dealing with congestion control dynamics.  

	[SM] Partly, the recommendation is to set interval to the expected RTT, and from the AQM's perspective it will take at least one RTT until its signal has been ping-ponged to the receiver, back to the sender and the sender adjust its behavior in a way that is visible for the AQM, so unless already overloaded it makes little sense to send more signals before giving a flow time to react... but that is not explicit in the RFC.
Ideally the interval would be deduced for each flow independently, but empirically it turned out that getting interval exactly right is not really required for acceptable performance.

> The other definition of standing queue (the queue to eliminate if 100% utilization for a single flow is desired) is the remaining queue at the trough of the TCP sawtooth.  These two definitions only align when the sawtooth period is less than 100ms. As a result, CoDel only supports 100% link utilization in the single flow case when that is true.

	[SM] Exactly, the default interval is 100ms exactly to cater to RTTs of (up to) 100ms, in practice it turned out that codel, especially fq_codel are relative forgiving abut not getting interval exactly right, so an interval off 100 will work quite well for RTT <= 200ms , and in addition interval is a user-controllable parameter that can be adjusted to longer or shorter RTTs, the same also seems to hold for PIE that the ideal ref-delay depends on the expected RTT distribution and delay tolerance of the traffic. See e.g. the recommendation if rfc8033 to set ref_delay in data centers to 0.15 ms: 
"If the target latency is reduced, e.g., for data-center
   use, the values of alpha and beta should be increased by the same
   order of magnitude by which the target latency is reduced.  For
   example, if QDELAY_REF is reduced and changed from 15 milliseconds to
   150 microseconds -- a reduction of two orders of magnitude -- then
   alpha and beta values should be increased to alpha * 100 and
   beta * 100.")"
Side-note: Codel does not aim for strict 100% link utilization though, just "close to 100%", consciously trading-off some utilization for lower latency under load.

>  Since the period of the sawtooth for Reno or Cubic can be measured in seconds (or minutes) for typical broadband speeds and RTTs today, this doesn't really work as described, and CoDel reverts to more-or-less acting as a short (5ms-ish) drop-tail queue.  For 100Mbps links, CoDel only supports 100% utilization for RTTs less than ~3.5ms.  For 1Gpbs links, CoDel only supports 100% utilization for RTTs less than ~1ms.  Sebastian has mentioned a few times that there is a theoretical basis for the 5ms target in CoDel.  I've not seen one, and wonder if someone could point me where to find it. 

	[SM] Happy to deliver, https://tools.ietf.org/html/rfc8289#section-3.2 is leaps and bounds beyond and above what the PIE literature cited above has to say about ref_delay, and I am sure I have cited that before here on the list. 

	But to repeat my position, while I welcome increasing RTT-independence of transport protocols (just as with most of the other TCP Prague goals I am all for it), I consider it to be a red herring in regards to DQCAQM failure to the one thing it exist for robustly and reliably (and without having to rely on well-behaving and cooperating flows). 
	My proposal was and still is for the L4S proponents to just change that 15 to 5 ms in the C-queues PIE instance and see how this affects worst case (short RTT) sharing between the two traffic classes and link utilization. Basically to test the hypothesis that RTT-independence might ameliorate DQCAQM systematic failure. Yes, I realize that the L4S plan is to improve RTT-independence of the L4S traffic and improve things from that side, but until there is demonstrably working transport protocol that archives sufficiently high RTT-independence it ld seem worthwhile to explore other ways to fix DQCAQM, especially if one is eager to roll the DQCAQM out ASAP, no?

Best Regards
	Sebastian