Subject: Re: [tsvwg] Technical review 5

From: Bob Briscoe <[email-masked]>
Date: Mon, 11 Mar 2019 20:17:55 +0000

Dave,

While looking for your name, I found your posting from Xmas Day, that I 
never replied to - sorry.

The main thing I think you've missed is that the WRR scheduler between 
the two queues doesn't actually do anything in all normal circumstances. 
The DualQ actually even works with a strict priority scheduler because 
of this.

The scheduling is done by the senders, just as if the 2 queues were a 
single FIFO. That's not obvious so let me explain...

When there's traffic in the Classic queue, the ECN marking in the L4S 
queue coupled over from the Classic AQM, causes the L4S sender to leave 
enough space between it's packets for the classic queue. When Koen first 
suggested this, I didn't think it would work, but it works amazingly 
accurately.

The only reason for using WRR instead of strict priority is to give the 
Classic queue a chance, in case of an unresponsive sender in the L 
queue. It's good to keep the weight high (we recommend 15/16) so L4S 
packets get served quickly.

Not using strict priority also ensures that an initial DNS or SYN packet 
in the Classic queue is released quickly if there's a continuous flow 
going in the L4S queue.

In the Linux implementation, we also provide a scheduler called 
time-shifted FIFO, but we recommend WRR (particularly in the DOCSIS 
case, cos TS-FIFO needs packet time-stamping, which existing cable modem 
hardware doesn't support). TS-FIFO just takes the packet that arrived 
first, but for one queue it first subtracts from the arrival time by a 
pre-configured amount (e.g. 30ms) to prioritize it.

Bob

On 25/12/2018 02:56, Dave Taht wrote:
> Bob Briscoe <[email-masked]> writes:
>
>> David,
>>
>> OK, you're right that we don't do a good job of justifying this, other
>> than hinting that WRR is already well-known.
>>
>> I've recorded much more of the thinking behind this choice. I've also
>> explained why the choice of WRR scheduler weight is not important, as
>> follows (additions in green):
>>
>>     o  A well-understood weighted scheduler such as weighted round robin
>>        (WRR) is recommended.  The scheduler weight for Classic should be
>>        low, e.g. 1/16.  Note that the coupled congestion signal typically
>>        make L4S sources leave roughly equal per-flow capacity available
>>        for classic flows.  So, if the weight is small, its exact value is
>>        unimportant because it does not normally determine capacity
>>        shares.  The weight is only important to prevent unresponsive L4S
>>        traffic starving classic traffic.
> I note that the "fq" portion of fq_codel (RFC8290) has better min/max properties
> than straight drr, as does qfq.
>
> I am saddened that there is seemingly no fq or aqm component intended for
> "classic" flows in this design, as if it didn't need it.
>
> The first deployment of fq_codel (free.fr's) had the following weighted
> queue structure:
>
> drr 1 priority fq_codel (diffserv markings)
> drr 2 best effort fq_codel
> drr 3 background fq_codel
>
> and nearly the entire qos related deployment of this has essentially the
> same structure, including sch_cake.
>
> In deprioritizing "classic" to 1/16 the total possible flow volume, any
> form of weighted fair queuing can be used, but I still don't know why
> you wish to punish classic flows so much by eliding aqm and fq on those.
>
>
>
>>     o  Alternatively, a time-shifted FIFO (TS-FIFO) could be used.  It
>>        works by selecting the head packet that has waited the longest,
>>        biased against the Classic traffic by a time-shift of tshift.  To
>>        implement time-shifted FIFO, the "if (scheduler() == lq )" test in
>>        line 3 of the dequeue code would simply be replaced by "if (
>>        lq.time() + tshift >= cq.time() )".  For the public Internet a
>>        good value for tshift is 50ms.  For private networks with smaller
>>        diameter, about 4*target would be reasonable.  TS-FIFO is a very
>>        simple scheduler, but complexity might need to be added to address
>>        some deficiencies (which is why it is not recommended over WRR):
>>
>>        *  TS-FIFO does not fully isolate latency in the L4S queue from
>>           uncontrolled bursts in the Classic queue;
>>
>>        *  TS-FIFO is only appropriate if time-stamping of packets is
>>           feasible;
>>
>>        *  Even if time-stamping is supported, the sojourn time of the
>>           head packet is always stale.  For instance, if a burst arrives
>>           at an empty queue, the sojourn time will only measure the delay
>>           of the burst once the burst is over, even though the queue knew
>>           about it from the start.  At the cost of more operations and
>>           more storage, a 'scaled sojourn time' metric of queue delay can
>>           be used, which is the sojourn time of a packet scaled by the
>>           ratio of the queue sizes when the packet departed and arrived.
>>
>> Bob
>>
>> On 18/12/2018 19:53, David Pullen wrote:
>>
>>      Hi Bob,
>>      
>>      
>>      At the end of Appendix A, WRR is recommended over time-shifted
>>      FIFO, but no real justification is given. Why is WRR recommended?
>>      
>>      
>>      
>>      
>>      
>>      
>>      
>>      
>>      
>>      
>>      
>>      Thanks,
>>      - Dave
>>      
>>      
>>          David Pullen Distinguished Engineer | Wired Networking, Cable
>>      Modems Broadcom office: 678.475.3143 | mobile: 678.477.4108 | fax:
>>      770.232.0211 4385 River Green Parkway | Duluth, GA 30096
>>      [email-masked] | broadcom.com

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Dave Taht <[email-masked]>
Date: Mon, 11 Mar 2019 16:31:07 -0700

Bob Briscoe <[email-masked]> writes:

> Dave,
>
> While looking for your name, I found your posting from Xmas Day, that
> I never replied to - sorry.

It would have helped.

> the two queues doesn't actually do anything in all normal
> circumstances. The DualQ actually even works with a strict priority
> scheduler because of this.
>
> queue coupled over from the Classic AQM, causes the L4S sender to
> leave enough space between it's packets for the classic queue. When
> Koen first suggested this, I didn't think it would work, but it works
> amazingly accurately.

I'll believe it only after y'all subject it to bufferbloat.net's
core benchmark suites - flent.org and irtt.

or, after I get a chance to try it myself...

Did you try any of those yet - notably the rrul series of tests?

I'd love to get some tests with the --socket-stats option enabled to be
able to look closely at the tcp rtts.

>
> The only reason for using WRR instead of strict priority is to give
> the Classic queue a chance, in case of an unresponsive sender in the L

Still sounds to me like normal (as opposed to what you call classic -
to me it's *normal*)

But look, we're finally at the point where I can take a good hard look
at this, at least at a black box level (due to the FRAND patent I simply
cannot look at the code)

>
> Not using strict priority also ensures that an initial DNS or SYN
> packet in the Classic queue is released quickly if there's a
> continuous flow going in the L4S queue.
>
> case, cos TS-FIFO needs packet time-stamping, which existing cable
> modem hardware doesn't support). TS-FIFO just takes the packet that
> arrived first, but for one queue it first subtracts from the arrival
> time by a pre-configured amount (e.g. 30ms) to prioritize it.
>
>
>
> Bob
>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 12 Mar 2019 14:05:54 +0000

David,

> Did you try any of those yet - notably the rrul series of tests?
[BB] Would your tools be able to give equal reporting weight to the 
latency of every packet in our L4S queue (which is designed to give low 
latency even when heavy and light flows are mixed together: ftp 
downloads, web, gaming, ping, HAS, HD interactive video). Or do they 
only report the latency of packets that the experimenter deems to be in 
light flows (ping, etc)?

When I read the documentation (some time ago) it appeared to be the 
latter, which presumes that high bandwidth flows never need to have low 
latency.

Our measurement tools take the qDelay of every packet (which we write 
into the IPID of the packet at the bottleneck queue) and report the 
collective statistical properties of the aggregate of flows in a queue. 
We also measure e2e RTT - again reporting the statistical properties of 
the aggregate for all flows.

> Still sounds to me like normal (as opposed to what you call classic -
[BB] I'm not sure you've understood me. I wouldn't expect someone who 
had understood me to say this. Understanding is as important as 
measurement.

> at this, at least at a black box level (due to the FRAND patent I simply
Cheers

Bob

---

From: Dave Taht <[email-masked]>
Date: Wed, 13 Mar 2019 15:03:38 -0700

Bob Briscoe <[email-masked]> writes:

> latency of every packet in our L4S queue (which is designed to give
> low latency even when heavy and light flows are mixed together: ftp
> downloads, web, gaming, ping, HAS, HD interactive video).

I happen to care about the performance of these latter flows a lot under
the l4s circumstances, and have no idea, how they actually get low
latency. Been dying to run at least some tests in my suite against the
l4s dualpi code for years, as I've said.

> Or do they
> only report the latency of packets that the experimenter deems to be
> in light flows (ping, etc)?

We have a variety of tests. The socket-stats option pools all the
available tcp statistics.

A typical run of that is

flent -H wherever -t whatever_setup_you_have -te=upload_streams=X tcp_nup

We also had a dctcp test, and it's really easy to write more, assuming
you have a set of tests we can (re) design to run. 

However we certainly do not measure the latency of every packet, that's
impossible to do, except in simulation, or via analyzing a tcpdump.

We do measure the impact of the underling buffering (for example, bql
and the bqlmon tools), and can measure the queuing delay anywhere else
on the path.

(there's way more options than just --socket-stats - and getting all
that additional data takes work and setup. And for example, we'd need
to write a parser for dualpi's statistical outputs, to do it on your stuff)

> latter, which presumes that high bandwidth flows never need to have
> low latency.

There are 100+ tests in the suite. The other thing that we do all the
time, is run one of these well defined tests at the same time as a
packet capture, and look at both outputs. 

> collective statistical properties of the aggregate of flows in a

We don't currently do that, we use netperf and irtt underneath.

Are you parsing packet captures also?

> We also measure e2e RTT - again reporting the statistical
> properties of the aggregate for all flows.

That we do.

> had understood me to say this. Understanding is as important as

Yep.

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 14 Mar 2019 18:44:26 +0000

David,

> l4s dualpi code for years, as I've said.
[BB] As you and I have discussed many times, Olga open-sourced the DualQ 
Coupled AQM code and it has been freely available since July 2016. You 
have been able to use DCTCP over it for testing ever since then.
https://mailarchive.ietf.org/arch/msg/tcpprague/Hphy-BPGW6w5ozNhIULjDM96AOo

> impossible to do, except in simulation, or via analyzing a tcpdump.
[BB] It's not impossible to do at all with real traffic (we never rely 
on simulations). If you don't measure the latency of every packet, I 
cannot see the point of latency tests. To prove whether you can support 
stuff like real-time video-based control, you need to look at the very 
high percentiles (e.g. 3 or 4 nines), which you cannot trust if you are 
sampling. Even if you use very large numbers of packets over very long 
test runs, you cannot trust that the outliers will appear at all in your 
sample, and you certainly cannot trust that their appearance will be 
representative.

With our Linux test framework, the egress of the queue encodes the 
sojourn time into the IP ID of each packet (in a bespoke binary floating 
point format). Nonetheless, as I said, we also measure the e2e RTT in 
all our tests.

BTW, also, in Low Latency DOCSIS, we have required that all 
implementations populate a histogram with operator-definable bin 
boundaries, so that the management system can output queue delay 
histograms if required.

If you don't do all these things during a project, it is hard to pull 
the team back together again later to add all these boring but critical 
necessities.

> Are you parsing packet captures also?
[BB] Yes
>>> at this, at least at a black box level (due to the FRAND patent I simply
[BB] You've been at that point for nearly 3 years. I don't understand 
why you keep implying you haven't been able to use the code until now. 
Admittedly until now it's not been ported against the tip of the Linux 
mainline, but surely that's not what has stopped you? Few people work at 
the tip anyway.

The DualQ code has been completely functional since Jul 2016 - the 
latest release has made no /significant/ changes. Indeed, with this 
latest release Olga re-ran the whole huge test set to compare with three 
main prior versions of the code to check performance had not regressed 
at all. After fixing some bugs, it hasn't. We didn't expect it to, 'cos 
all the changes were to do with non-performance stuff like independence 
between all the parameters in the API, extra classifier configuration 
options, etc.

Bob