Subject: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Thu, 17 Feb 2022 18:00:30 +0000

Hi Jonathan,

It seems that the following open issue identified by the chairs:

Non-L4S traffic abusing the L-queue
* 'DualQ gives a large throughput bonus to L queue traffic, ie. a "fast lane"'
* Is this a matter specific for DualQ that can be left for experimentation?

is based on the following experiment you performed:

>             simple two-flow competition test on a standard dumbbell topology,

>             with the bottleneck running a DualQ qdisc into a 50Mbps shaper.

>             Both flows were configured to use CUBIC congestion control with

>             ECN negotiated, but one was additionally tweaked to set ECT(1)

>             instead of ECT(0) on all data segments, and to pace its output at

>             40Mbps. This latter measure prevents the L queue from seeing any

>             need to apply congestion signals, because it is always empty.  These

>             tweaks allowed that flow to use 80% of the link capacity, gaining a

>             fourfold advantage over its competitor,

If there is capacity seeking traffic in the Classic queue, then it is even desired that the L4S queue does not add extra marks. The L4S marks should come only from the Classic coupling.
Before diving into details, can you first explain why in your experiment the coupling from the Classic Q has no effect on your paced and ECT(1) labeled Cubic flow?

I would expect that this ECT(1) labeled Cubic flow would get even less throughput than the Classic Cubic flow, as the first gets the doubled coupled CE marking probability (eg 2*10% = 20%) for L4S flows instead of the squared CE marking probability (10%^2 = 1%) which ECT(0) traffic would get.

Thanks,
Koen.

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 25 Feb 2022 09:28:48 +0000

Hi Jonathan,

Can you confirm that this test is done with "Cubic" traffic that is not responding at all to CE marks? So it is just like any other non-responding traffic (like UDP CBR). We don't see any other way to explain your results.

If so, we can/should remove this "issue" from the shepherd's write-up, as such unresponsive flows will get the same throughput on any single-Q bottleneck with or without AQM (taildrop/PI2/PIE/CoDel/STEP/RED/...) with a latency that matches the AQM strategy.

Koen.

From: tsvwg <[email-masked]> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Thursday, February 17, 2022 7:01 PM
To: tsvwg IETF list <[email-masked]>; Jonathan Morton <[email-masked]>
Subject: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

Hi Jonathan,

Thanks,
Koen.

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 25 Feb 2022 10:53:38 +0100

Hi Koen,

so I see theoretical problem when a sufficiently paced ECN-unresponsive CBR-Flow with a rate above its commensurate share of the capacity gets placed in the L-queue (a configuration that is fully within the recommendations of the L4S drafts*). I think in that case the C-queue traffic is going to suffer. Arguing that this is not worse that with a single queue AQM is not really helpful, as:
a) a bog standard dropping AQM might make the CBR-flow respond
b) a single queue AQM ECN-enabled is not an option for L4S, exactly because L4S does re-define the meaning of CE (or better the appropriate response)

Mind you I have not tested that configuration and it hence is possible that coupling will push back sufficient "drops" into the L-queue to reign in the CBR-flow, but I think it will not and the C-queue is going to bear the brunt here. I would not be amazed if it will only be the weighted scheduler configuration that will limit how much capacity that CBR flow can hog.
This also shows that mismarking CBR-flows (or sufficiently paced but unresponsive flows) with ECT(1) on purpose will be a way to gain an "unfair" advantage on an L4S path. I am also not sure whether a perfectly paced flow would actually trigger the queue protection as currently (I would love for QP to detect and counter this behaviour**), but since that is optional anyway is is somewhat irrelevant whether it would help or not.

*) https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-24#page-22
The above section requires unresponsive traffic to be 'safe' to mix
   with L4S traffic.  Ideally this means that the sender never sends any
   sequence of packets at a rate that exceeds the available capacity of
   the bottleneck link.  However, typically an unresponsive transport
   does not even know the bottleneck capacity of the path, let alone its
   available capacity.  Nonetheless, an application can be considered
   safe enough if it paces packets out (not necessarily completely
   regularly) such that its maximum instantaneous rate from packet to
   packet stays well below a typical broadband access rate.

Our paced CBR flow only needs to stay/pace below "link capacity" to qualify here, that means that e.g. pacing at 80% capacity is well within this recommendation, but will not result in the advertised equitable sharing between the L and C queue.

**) Anybody willing to share data on this? Giving that Low-Latency-DOCSIS seems to implement all the required parts here, can somebody involved in the LL-DOCSIS effort share some insight, please?

> On Feb 25, 2022, at 10:28, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Jonathan,
>  
> Can you confirm that this test is done with “Cubic” traffic that is not responding at all to CE marks? So it is just like any other non-responding traffic (like UDP CBR). We don’t see any other way to explain your results.
>  
> If so, we can/should remove this “issue” from the shepherd’s write-up, as such unresponsive flows will get the same throughput on any single-Q bottleneck with or without AQM (taildrop/PI2/PIE/CoDel/STEP/RED/…) with a latency that matches the AQM strategy.
>  
> Koen. 
>  
>  
> • ‘DualQ gives a large throughput bonus to L queue traffic, ie. a “fast lane”’
> • Is this a matter specific for DualQ that can be left for experimentation?
>

---

From: "Black, David" <[email-masked]>
Date: Fri, 25 Feb 2022 16:55:33 +0000

Koen,

I'll observe that "traffic that is not responding at all to CE marks" is not necessary to achieve the reported results if the experimental setup "prevents the L queue from seeing any
need to apply congestion signals, because it is always empty" as there would be no CE marks for the traffic in the L queue to respond to.

Please give that further consideration.

Thanks, --David (as an individual)

From: tsvwg <[email-masked]> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Friday, February 25, 2022 4:29 AM
To: tsvwg IETF list; Jonathan Morton
Subject: Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

[EXTERNAL EMAIL]
Hi Jonathan,

Koen.

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Thursday, February 17, 2022 7:01 PM
To: tsvwg IETF list <[email-masked]<mailto:[email-masked]>>; Jonathan Morton <[email-masked]<mailto:[email-masked]>>
Subject: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

Hi Jonathan,

Thanks,
Koen.

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 25 Feb 2022 12:57:39 -0500

On Fri, Feb 25, 2022 at 11:56 AM Black, David <[email-masked]> wrote:

> Koen,
>
>
>
> I'll observe that "traffic that is not responding at all to CE marks" is
> not necessary to achieve the reported results if the experimental setup "prevents
> the L queue from seeing any
>
> need to apply congestion signals, because it is always empty" as there
> would be no CE marks for the traffic in the L queue to respond to.
>

I think the key part here is "if". :-) The assertion "prevents the L queue
from seeing any need to apply congestion signals, because it is always
empty" is from:
  https://sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf
That assertion is inconsistent with the functioning of the Dual-Q
algorithm, as described in:
  https://www.ietf.org/id/draft-ietf-tsvwg-aqm-dualq-coupled-21.html

As Bob noted: "in the scenario shown, although the L queue is indeed always
empty, it will see a high level of congestion signals (~10% in this case)
via the coupling."
Here's Bob's e-mail for more context/details:
  https://mailarchive.ietf.org/arch/msg/tsvwg/joFr3sfOrxxkYhWdYrO2rLlCNUw/

thanks,
neal

> *From:* tsvwg <[email-masked]> *On Behalf Of * De Schepper, Koen
> (Nokia - BE/Antwerp)
> *Sent:* Friday, February 25, 2022 4:29 AM
> *To:* tsvwg IETF list; Jonathan Morton
> *Subject:* Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue"
> discussion during the interim
>
>
>
> [EXTERNAL EMAIL]
>
> Hi Jonathan,
>
>
>
> Can you confirm that this test is done with “Cubic” traffic that is not
> responding at all to CE marks? So it is just like any other non-responding
> traffic (like UDP CBR). We don’t see any other way to explain your results.
>
>
>
> If so, we can/should remove this “issue” from the shepherd’s write-up, as
> such unresponsive flows will get the same throughput on any single-Q
> bottleneck with or without AQM (taildrop/PI2/PIE/CoDel/STEP/RED/…) with a
> latency that matches the AQM strategy.
>
>
>
> Koen.
>
>
>
>
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *De Schepper, Koen
> (Nokia - BE/Antwerp)
> *Sent:* Thursday, February 17, 2022 7:01 PM
> *To:* tsvwg IETF list <[email-masked]>; Jonathan Morton <
> [email-masked]>
> *Subject:* [tsvwg] Related to "Non-L4S traffic abusing the L-queue"
> discussion during the interim
>
>
>
> Hi Jonathan,
>
>
>
> If there is capacity seeking traffic in the Classic queue, then it is even
> desired that the L4S queue does not add extra marks. The L4S marks should
> come only from the Classic coupling.
>
> Before diving into details, can you first explain why in your experiment
> the coupling from the Classic Q has no effect on your paced and ECT(1)
> labeled Cubic flow?
>
>
>
> I would expect that this ECT(1) labeled Cubic flow would get even less
> throughput than the Classic Cubic flow, as the first gets the doubled
> coupled CE marking probability (eg 2*10% = 20%) for L4S flows instead of
> the squared CE marking probability (10%^2 = 1%) which ECT(0) traffic would
> get.
>
>
>
> Thanks,
>
> Koen.
>
>
>
>
>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 25 Feb 2022 18:04:38 +0000

Hi David,

The design goal of the DualQ is that the L queue is always empty as long as there is capacity seeking Classic traffic. If there is capacity seeking Classic traffic, it builds a Classic queue which is creating marks for the L4S flow too (don't forget the AQM coupling here!). If the sender is using falsely ECT(1) it would be marked even more than if it was using ECT(0), and back off even more if it was responsive. So this makes us conclude that the traffic in Jonathan's experiment was non-responsive (and not using Cubic) or CE marks were cleared or not echoed by the receiver, or ...?

One of the confusions is probably that the draft specifies only "starvation". As-is, it can be interpreted as being caused by senders, but in fact, the only starvation case we noticed is because of an implementation detail of how the classic queue measures the Classic delay. If the classic queue delay (or throughput measurement in combination with queue size) only happens when a classic packet is processed (scheduled), you need to use a WRR scheduler that processes a packet at least every update time (to get a new measurement value for the PI_update algorithm).

Another reason for using a WRR instead of a strict priority scheduler is that it improves responsiveness for sparse non-capacity seeking classic flows. You could call this a "temporary" starvation (in the order of 150ms which is how long the PI2 controller would need to reach the drop-ECN threshold, if the L4S flow has not responded based on the CE marks; BTW, it takes 240ms to reach 100% drop).

If you think it is useful/necessary we still can clarify this in the draft?

Koen.

From: Black, David <[email-masked]>
Sent: Friday, February 25, 2022 5:56 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg IETF list <[email-masked]>; Jonathan Morton <[email-masked]>
Cc: Black, David <[email-masked]>
Subject: RE: Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

Koen,

[EXTERNAL EMAIL]
Hi Jonathan,

Koen.

Hi Jonathan,

Thanks,
Koen.

---

From: "Black, David" <[email-masked]>
Date: Fri, 25 Feb 2022 18:03:53 +0000

Hi Neal,

So, I saw that explanation – could someone check the "running code" to make sure that the coupling and marking occur even when the L queue is always empty?

Thanks, --David

From: Neal Cardwell <[email-masked]>
Sent: Friday, February 25, 2022 12:58 PM
To: Black, David
Cc: De Schepper, Koen (Nokia - BE/Antwerp); tsvwg IETF list; Jonathan Morton; Bob Briscoe
Subject: Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

[EXTERNAL EMAIL]

On Fri, Feb 25, 2022 at 11:56 AM Black, David <[email-masked]<mailto:[email-masked]>> wrote:
Koen,

I think the key part here is "if". :-) The assertion "prevents the L queue from seeing any need to apply congestion signals, because it is always empty" is from:
  https://sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf [sce.dnsmgr.net]<https://urldefense.com/v3/__https:/sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf__;!!LpKI!2-aGvIgLG5_UcfoDdFo5NR6FegDlw6w7v97l7dT8g-Or7eBQNEgIITlBB4xE-wzo$>
That assertion is inconsistent with the functioning of the Dual-Q algorithm, as described in:
  https://www.ietf.org/id/draft-ietf-tsvwg-aqm-dualq-coupled-21.html [ietf.org]<https://urldefense.com/v3/__https:/www.ietf.org/id/draft-ietf-tsvwg-aqm-dualq-coupled-21.html__;!!LpKI!2-aGvIgLG5_UcfoDdFo5NR6FegDlw6w7v97l7dT8g-Or7eBQNEgIITlBB6qmUFY4$>

As Bob noted: "in the scenario shown, although the L queue is indeed always empty, it will see a high level of congestion signals (~10% in this case) via the coupling."
Here's Bob's e-mail for more context/details:
  https://mailarchive.ietf.org/arch/msg/tsvwg/joFr3sfOrxxkYhWdYrO2rLlCNUw/ [mailarchive.ietf.org]<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/joFr3sfOrxxkYhWdYrO2rLlCNUw/__;!!LpKI!2-aGvIgLG5_UcfoDdFo5NR6FegDlw6w7v97l7dT8g-Or7eBQNEgIITlBByEN3xHA$>

thanks,
neal

[EXTERNAL EMAIL]
Hi Jonathan,

Koen.

Hi Jonathan,

Thanks,
Koen.

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 25 Feb 2022 18:30:26 +0000

Hi David,

To be sure, we re-did the overload tests recently, confirming the previous overload results. These results are available at: Overload results caused by non-responsive UDP traffic for PIE, DualPI2 and CoDel AQMs | l4steam.github.io<https://l4steam.github.io/overload-results/>

Specifically look at figure 8 at the end which shows that L4S traffic gets marks, up to 100% and appropriate drop if it reaches and exceeds the link capacity.

The test case of Jonathan is approximated by the 70Mbps non-responsive ECT(1) UDP traffic on a 100Mbps link on a DualPI2 (Prague+Cubic) test case. In Jonathan’s case it was 40Mbps on a 50Mbps link. We also evaluated in extreme when sending at 100Mbps non-responsive ECT(1) UDP traffic on a 100Mbps link, and even exceeding at 140Mbps and 200Mbps. You will see the results are as if it is on a Single Q PIE AQM. Note also that CoDel which never drops ECT packets, causes actually close to starvation and high tail-drop delay results as shown in figure 1, even with ECT(0). So I guess all the concerns about FQ_CoDel and tunnels/Hash-collisions are equally severe and not related to L4S alone (can just be exploited by ECT(0) traffic today already!!).

Koen.

From: Black, David <[email-masked]>
Sent: Friday, February 25, 2022 7:04 PM
To: Neal Cardwell <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg IETF list <[email-masked]>; Jonathan Morton <[email-masked]>; Bob Briscoe <[email-masked]>; Black, David <[email-masked]>
Subject: RE: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

Hi Neal,

Thanks, --David

From: Neal Cardwell <[email-masked]<mailto:[email-masked]>>
Sent: Friday, February 25, 2022 12:58 PM
To: Black, David
Cc: De Schepper, Koen (Nokia - BE/Antwerp); tsvwg IETF list; Jonathan Morton; Bob Briscoe
Subject: Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

[EXTERNAL EMAIL]

thanks,
neal

[EXTERNAL EMAIL]
Hi Jonathan,

Koen.

Hi Jonathan,

Thanks,
Koen.

---

From: Dave Taht <[email-masked]>
Date: Fri, 25 Feb 2022 16:02:58 -0500

I am happy to see some results of different scenarios here, and have
some suggestions as to others.

On Fri, Feb 25, 2022 at 1:30 PM De Schepper, Koen (Nokia - BE/Antwerp)
<[email-masked]> wrote:
>
> Hi David,
>
>
>
> To be sure, we re-did the overload tests recently, confirming the previous overload results. These results are available at: Overload results caused by non-responsive UDP traffic for PIE, DualPI2 and CoDel AQMs | l4steam.github.io

It really helps if you document what codebase you are testing. As one
example among many, fq_codel has a drop_batch facility now. An
enhancement to that has been, for 4? 5 years? is the codel count gets
incremented when drop_batch is hit.

The article sets flows=1 for fq_codel, and then makes claims that it
has no overload protection. The "overload protection" is in the fq
portion of the algorithm. I have long thought were ecn of any form to
become a thing in the real world, that it would be best to also modify
codel to do "drop and mark" in the RTT seeking portion of the algo.

Now, if you are trying to make the point that having a queue selector
that is kept from hop to hop and allows for differential treatment is
a win, I'd rather like to see results for a queue size that is
actually sufficient for pie/dualpi to operate at 10Gbit or above,
working at a mix of 10 to 60ms or greater RTTs. I assume the packet
limit for dualpi is 1000, shared between the two? What happens on this
test at queue 1000 at 10Gbit, or queue 10000 at 100Mbit? [1]

In terms of future scaling issues for all these new forms of tcp and
qdiscs it would help if y'all (and by this I mean all parties here)
were testing 1gbit, 10gbit, and beyond traffic, at this point. The
default packet limit in codel is tuned to 10Gbit+, and has largely
been supplanted by a memory limit (as in cake) due to the prevalance
of gso/gro traffic.

Also seeing the work on TCP BIG at google, how well is anything
working with 4k packets?

[1] I've longed for a good rrul (simultaneous up/down) test on an
asymmetric network. 1Gbit down/50mbit up being common now.

>   https://sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf [sce.dnsmgr.net]
>
>   https://www.ietf.org/id/draft-ietf-tsvwg-aqm-dualq-coupled-21.html [ietf.org]
>
>
>
>   https://mailarchive.ietf.org/arch/msg/tsvwg/joFr3sfOrxxkYhWdYrO2rLlCNUw/ [mailarchive.ietf.org]
>
>
>
> thanks,
>
> neal
>
>
>
>
>
>
>

-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

---

From: Dave Taht <[email-masked]>
Date: Fri, 25 Feb 2022 16:06:16 -0500

while I do not want to spend much time nitpicking this document...

"causing most of the time tail-drop" stood out. codel, fq_codel, cake
all do head drop, and always have.

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 25 Feb 2022 23:00:36 +0100

Hi Koen,

interesting test, now it would be excellent to see how the 50 and 70 steps would look with queue protection active.

> On Feb 25, 2022, at 19:30, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi David,
>  
> The test case of Jonathan is approximated by the 70Mbps non-responsive ECT(1) UDP traffic on a 100Mbps link on a DualPI2 (Prague+Cubic) test case. In Jonathan’s case it was 40Mbps on a 50Mbps link. We also evaluated in extreme when sending at 100Mbps non-responsive ECT(1) UDP traffic on a 100Mbps link, and even exceeding at 140Mbps and 200Mbps. You will see the results are as if it is on a Single Q PIE AQM. Note also that CoDel which never drops ECT packets,

	Is that actually true? fq_codel has a batch drop mode it employs to digg itself out of severe overload, see:
https://elixir.bootlin.com/linux/latest/source/net/sched/sch_fq_codel.c#L137
not sure this code path is avoided for ECT(0/1) flows, so on massive overload fq_codel will drop min(64, num_packets_in_fattest_flow * 0.5) from the "fattest" flow, in single queue mode that should still trigger... so yes, in normal operation fq_codel will not drop ECT(0/1) packets, but if push comes to shove it will do so in batches. If I read the code correctly this mode will trigger if during enqueue of a new packet either:
a) all queued packets exceed the memory_limit or 
b) there are more packets queued than limit
in both conditions fq_codel will drop packets in the fattest hash-bucket independent on the ECN bits.

> causes actually close to starvation and high tail-drop delay results as shown in figure 1, even with ECT(0).

	Yes, if you overload the queues you end up triggering the stress mode dropper, by adjusting limit/memory_limit you can tune this behavior closer to your liking. But even in stress drop mode it seems to do head drops. What evidence do you have for tail-drop? 

> So I guess all the concerns about FQ_CoDel and tunnels/Hash-collisions are equally severe and not related to L4S alone

	Yes, we know that the advantages of flow queueing disappear if flows can not be identified anymore... In this specific case though, fq-odel with multiple queues will do fine as long as the unresponsive flow will not manage to already crowd out the other flows before hitting the AQM bottleneck. In full fq-mode most of the fall-out of the unresponsive flow will be restricted to its hash-bucket (including innocent bystanders in the same bucket).

> (can just be exploited by ECT(0) traffic today already!!).

	Subtle difference, today in a FIFO or codel  our offending flow will see actual drops commensurate to its share of capacity, but in the L-queue all it will see are inconsequential CE marks, in other the words the L-queue delivers better service for hostile traffic. I do predict that this will be exploited if we actually deploy such systems, the question is when not if...
	I really do not think this is a good argument to make, if you go and introduce a scheduler into the network, it would be a wasted opportunity not to actually squash a whole class of abuse while at it, if all you claim is to not be worse than a FIFO, than you aim too low.

Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 25 Feb 2022 23:14:23 +0100

Hi Dave,

> On Feb 25, 2022, at 22:06, Dave Taht <[email-masked]> wrote:
> 

	I think that codel (not fq_codel configured for a single hask-bucket) on overload actually drops the new packet in the enqueue routine. I would guess this is because most users followed Van's advise to use fq_codel instead... so in a sense sch_codel might be more documentation than a qdisc hardened by exposure to real usage. The linked page seems to indicate that fq_codel(1) was used, but it does not seem to say so for the figures we are discussing here.

Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 25 Feb 2022 23:31:38 +0100

Hi Koen,

from your link:

"Only when non-responsive traffic is below the link capacity it can fully use that share, making the responsive flows share the rest of the capacity (as usual for any AQM on the Internet on a shared queue)." 

This pretty much confirms what I predicted, except the drop plots look incomplete, the highly interesting drops for Not-ECT UDP (for CoDel) seem to be missing (generally the labeling for figure 8 is imprecise, e.g. showing Drops ECT(1)-UDP in the first set PIE with ECT(0)-UDP). My hunch is that comparing drops ECT(1)-UDP in DualPi2 with the missing drops Not-ECT UDP in Codel, that in the latter we see much more drops than in the former, et voila, exploitable way to gain more throughput by marking traffic ECT(1)... as long as that traffic is reasonably paced and does not exceed the capacity of the L4S AQM (not an unlikely scenario even for capacity seeking traffic, my measly 100Mbps access link will not saturate the BNG uplink of my ISP or the link to the DSLAM)

Regards
	Sebastian

---

From: Luca Muscariello <[email-masked]>
Date: Sat, 26 Feb 2022 00:16:08 +0100

Hi David

I'm just reading the draft and cannot comment on the code, but it would be
intuitive that a coupled queuing systems of two queues would decouple when
one of the two queues is empty.

Reading the details of the pseudo code in the draft, if lq is empty p'L
should be 0 and pL=pCL=kp' which may be !=0.

Additional text below the pseudo code provides details about a Linux
implementation that marks packet IFF queue length is >= 2 MTU. Which may
suggest that decoupling happens when the L queue is empty.

I did not check the code though.

I will read the document more carefully beginning of next week but I admit
this part is ambiguous.

Luca

On Fri, Feb 25, 2022, 19:05 Black, David <[email-masked]> wrote:

> Hi Neal,
>
>
>
> So, I saw that explanation – could someone check the "running code" to
> make sure that the coupling and marking occur even when the L queue is
> always empty?
>
>
>
> Thanks, --David
>
>
>
> *From:* Neal Cardwell <[email-masked]>
> *Sent:* Friday, February 25, 2022 12:58 PM
> *To:* Black, David
> *Cc:* De Schepper, Koen (Nokia - BE/Antwerp); tsvwg IETF list; Jonathan
> Morton; Bob Briscoe
> [sce.dnsmgr.net]
> <https://urldefense.com/v3/__https:/sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf__;!!LpKI!2-aGvIgLG5_UcfoDdFo5NR6FegDlw6w7v97l7dT8g-Or7eBQNEgIITlBB4xE-wzo$>
>
> [ietf.org]
> <https://urldefense.com/v3/__https:/www.ietf.org/id/draft-ietf-tsvwg-aqm-dualq-coupled-21.html__;!!LpKI!2-aGvIgLG5_UcfoDdFo5NR6FegDlw6w7v97l7dT8g-Or7eBQNEgIITlBB6qmUFY4$>
>
>
>
> As Bob noted: "in the scenario shown, although the L queue is indeed
> always empty, it will see a high level of congestion signals (~10% in this
> case) via the coupling."
>
> [mailarchive.ietf.org]
> <https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/joFr3sfOrxxkYhWdYrO2rLlCNUw/__;!!LpKI!2-aGvIgLG5_UcfoDdFo5NR6FegDlw6w7v97l7dT8g-Or7eBQNEgIITlBByEN3xHA$>
>
>
>
> thanks,
>
> neal
>
>
>
>
>
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 26 Feb 2022 13:44:57 +0000

Dave,

I will keep reminding everyone that this shift of topic to FQ-CoDel is 
distracting from the task at hand:
     "Is Jonathan going to confirm that his 'throughput bonus' and 'fast 
lane' accusations against DualQ are baseless because his experiment was 
broken?"

Nonetheless, I've changed the subject line and responded on FQ-CoDel 
below, tagged [BB]...

On 25/02/2022 21:02, Dave Taht wrote:
> portion of the algorithm.

[BB] Surely CoDel code is meant to be robust to different deployments 
and configurations, e.g.
1. a VPN classified into one flow-queue:
     then an unresponsive flow within the VPN that is not a problem for 
the whole link could overwhelm the capacity being scheduled for the 
flow-queue
2. With this problem in CoDel, it should not be claimed that FQ-CoDel 
can be repurposed for one queue per user (e.g. for within aggregation 
networks)

> I have long thought were ecn of any form to

[BB] Indeed, you often say you've "long thought" this. Pity you've never 
"short done" it ;)
Indeed, your primary concern about ECN only applies as long as "drop and 
mark" under overload is not implemented.

It should have been done when ECN was first added to CoDel, and it's 
still not done.
* The ECN spec [RFC3168; §7] has recommended it since 2001, so there's 
never been an excuse for not doing it.
* AQM Recommendations and Best Current Practice [RFC7567; §4.2.1] 
repeated this advice when the IETF first adopted it in Oct 2013:
https://datatracker.ietf.org/doc/html/draft-ietf-aqm-recommendation-00#section-4.2.1
     https://datatracker.ietf.org/doc/html/rfc7567#section-4.2.1

Bob

PS. The rest of the email strays off to yet another topic in the vein of 
'Bigger cages, Longer chains' Someone else might want to go there, but 
not me.

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 26 Feb 2022 13:45:24 +0000

Dave,

Nonetheless, response on FQ-CoDel is below, tagged [BB]...

> all do head drop, and always have.

[BB] For the list, we're talking about Figure 5 here: 
https://l4steam.github.io/overload-results/

I'm nearly certain that the cap at 600 ms is tail drop.
Cause: The control law increases head drop so slowly that the flow-queue 
containing the unresponsive flow eventually fills the buffer allocated 
to the whole qdisc. Then I believe it moves into what Jonathan calls 
'tallest sunflower' drop mode (tail drop focused on the longest flow-queue).

To help prove this, here's an experiment Asad ran for me last Oct on 
FQ-CoDel with an unresponsive flow rate just greater than the link rate.
https://bobbriscoe.net/projects/latency/CoDel-delta-bug.pdf#page=4
We were testing very slight overload, so it would stay in head drop 
mode, without hitting the need for tail drop. The plot shows a similar 
series of humps in the queue, but without the cut-off due to tail drop. 
So it's fairly conclusive that Koen's Fig 5 is showing tail drop.

I'll answer your question (on the SANE list) about why the humps repeat, 
but that's a trivial bug compared to the time CoDel takes in the first 
place.
It's a design flaw, not a bug.
The so-called 'control' law never even measures the queue it is meant to 
be controlling.
Here's some history:

* On 12-Nov-2013 I reported that to Kathie and Van as CoDel designers, 
cc the AQM list:
https://mailarchive.ietf.org/arch/msg/aqm/l4H1QdRl8B-E5FWpJh4w50B_nQE/
* No response by anyone for over 18 months, until...
* 07-Jun-2015: Toke confirmed my analysis empirically (see it, via same 
thread above)
    Toke's plot: 
https://kau.toke.dk/ietf/codel-drop-rate/codel-drop-rate.svg
* On 30-Sep-2015 you (DaveT) said "cake uses a better curve for CoDel 
but we still need to do more testing in the lab"
    As far as I understand it, that missed the point: CAKE's curve is 
still extremely slow, but somewhat faster than CoDel.
    But, CAKE's control law still never measures the queue it is meant 
to be controlling.
* 25-Feb-2022: You say you don't want to spend much time nitpicking 
Koen's experiment.
     If not you, someone needs to grasp this nettle, given FQ-CoDel is 
the default qdisc in the Linux mainline.

Bob

---

From: Dave Taht <[email-masked]>
Date: Sat, 26 Feb 2022 12:13:26 -0500

At one level you are interpreting an observed behavior as "tail drop"
- which may well be possible somewhere in the stack,
but it's not clear if you were running a post 2016 kernel which is
what added the drop_batch facility.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=9d18562a227

This drops from the head, not the tail.

I was not satisfied with this solution btw, and in some later patch
added an increment to the codel count in drop_batch so as to pass "bad
things are happening elsewhere" back over to the main portion of the
algorithm. I'm still very unsatisfied with the concept of a fixed and
user configurable drop_batch length, rather than something that
autotuned.

elsewhere in the fq_codel_fast repo I experimented with eliminating
the queue search, but accepting that small but constant cpu overhead
for a optimizing for what is perceived to be (and may not be!) a
rarely hit condition, or accepting the cost of the search when it
happens, remains to be seen.

So, while trying to disregard your conclusion this was tail drop, I am
happy that you have clearly identified (with a kernel version), and
described a test (yay!) that tickles a count caching problem and
proposed some solutions here:

https://bobbriscoe.net/projects/latency/CoDel-delta-bug.pdf

cc-ing the codel list.

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 26 Feb 2022 19:16:11 +0000

Dave,

> This drops from the head, not the tail.

[BB] Yes, sry, I should have said 'head drop on arrival', not tail drop. 
I was trying to say this is not drop driven by the AQM, rather it's drop 
'cos there's no more room in buffer allocated to the qdisc.

> https://bobbriscoe.net/projects/latency/CoDel-delta-bug.pdf

[BB] OK.

And pls would someone also take note that the much more major design 
flaw in the control law needs attention nearly 9 years after I reported 
it (see the last slide in the link above).

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 26 Feb 2022 23:17:49 +0000

David, it might help to be crystal clear, see [BB]

On 25/02/2022 16:55, Black, David wrote:
>
> Koen,
>
> I'll observe that "traffic that is not responding at all to CE marks" 
> is not necessary to achieve the reported results if the experimental 
> setup "prevents the L queue from seeing any
>

[BB] If there's a Classic capacity seeking flow (as in this case), the 
coupling is designed to mark any L packets with the coupled probability, 
even if they are never queuing (e.g. if they are being paced). That is 
the normal way that the DualQ works, and has done for the last 10 years.

> Please give that further consideration.

[BB] No further consideration can be given to Jonathan's experiment, 
beyond that it was clearly broken in one of the ways we emailed to him 
on this list (17-Feb-22), without response so far.

Bob

> *From:* tsvwg <[email-masked]> *On Behalf Of * De Schepper, 
> Koen (Nokia - BE/Antwerp)
> *Subject:* Re: [tsvwg] Related to "Non-L4S traffic abusing the 
> L-queue" discussion during the interim
>
> [EXTERNAL EMAIL]
>
> Hi Jonathan,
>
> Can you confirm that this test is done with “Cubic” traffic that is 
> not responding at all to CE marks? So it is just like any other 
> non-responding traffic (like UDP CBR). We don’t see any other way to 
> explain your results.
>
> If so, we can/should remove this “issue” from the shepherd’s write-up, 
> as such unresponsive flows will get the same throughput on any 
> single-Q bottleneck with or without AQM 
> (taildrop/PI2/PIE/CoDel/STEP/RED/…) with a latency that matches the 
> AQM strategy.
>
> Koen.
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *De Schepper, 
> Koen (Nokia - BE/Antwerp)
> *To:* tsvwg IETF list <[email-masked]>; Jonathan Morton 
> <[email-masked]>
> If there is capacity seeking traffic in the Classic queue, then it is 
> even desired that the L4S queue does not add extra marks. The L4S 
> marks should come only from the Classic coupling.
>
> Before diving into details, can you first explain why in your 
> experiment the coupling from the Classic Q has no effect on your paced 
> and ECT(1) labeled Cubic flow?
>
> coupled CE marking probability (eg 2*10% = 20%) for L4S flows instead 
> of the squared CE marking probability (10%^2 = 1%) which ECT(0) 
> traffic would get.
>
> Thanks,
>
> Koen.
>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 27 Feb 2022 17:51:40 +0000

Hi Dave,

We are using the l4steam kernel, which was not that long ago upgraded and merged with the BBRv2 kernel which was 5.10. The code you referenced that does the head-drop when the queue is full is part of this code. You can see it in action because the sojourn time reduces when the arrival overload rate increases. At 200Mbps it is 600ms instead of the 1200ms expected with taildrop for a 10000p limit at 100Mbps. This is because the queue gets served at 200Mbps too, 100Mbps to forward and 100Mbps to drop at dequeue.

That code is not actually AQM code, rather taildrop which drops packets at the head. In the meantime (luckily in this case only after at least 100 seconds) the CoDel algorithm is dropping all non-ECT packets while still leaving the ECT(0) packets untouched. Since the queue doesn't shrink below 5ms, it stays dropping in this case, killing "all" non-ECT traffic. This is a "real" attack vector for (FQ-)CoDel and needs an immediate fix for all (FQ-)CoDel deployments. At the same time it can be upgraded to support L4S too ;-) and everyone is happy.

Regards,
Koen.

-----Original Message-----
From: Dave Taht <[email-masked]> 
Sent: Friday, February 25, 2022 10:06 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Black, David <[email-masked]>; Neal Cardwell <[email-masked]>; tsvwg IETF list <[email-masked]>; Bob Briscoe <[email-masked]>
Subject: Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

"causing most of the time tail-drop" stood out. codel, fq_codel, cake all do head drop, and always have.

On Fri, Feb 25, 2022 at 4:02 PM Dave Taht <[email-masked]> wrote:
> > To be sure, we re-did the overload tests recently, confirming the 
> > previous overload results. These results are available at: Overload 
> > results caused by non-responsive UDP traffic for PIE, DualPI2 and 
> > CoDel AQMs | l4steam.github.io
>
> > Cc: De Schepper, Koen (Nokia - BE/Antwerp) 
> > <[email-masked]>; tsvwg IETF list 
> > <[email-masked]>; Jonathan Morton <[email-masked]>; Bob 
> > Briscoe <[email-masked]>; Black, David <[email-masked]>
> > I'll observe that "traffic that is not responding at all to CE 
> > marks" is not necessary to achieve the reported results if the 
> > experimental setup "prevents the L queue from seeing any
> >

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 27 Feb 2022 17:57:46 +0000

Hi Sebastian,

The important plots to check are the throughput plots in figure 1, where PIE and DualPI2 have very similar throughput profiles (confirming a coupled DualQ works like a single Q), and figure 3 where PIE and DualPI2 have a queue latency at the target. This can only be achieved if the drop and marks are correctly set by the AQM. Only in the CoDel case the ECT(0) non-responding UDP flow takes practically all the link capacity (99.9% when sending unresponsive UDP-ECT(0) at 100Mbps, and really 100% when sending at the higher 140 and 200Mbps rates, probably killing "all" not-ECT traffic; see other mail to Dave Taht). These results speak undeniably for themselves. Your hunch and the issue which is now mentioned in the interim slides and the shepherd's writeup is just wrong. This non-issue should just be removed from the writeup and I suggest you make it an issue for the CoDel/FQ-CoDel RFCs instead. There we detected a "real" attack vector!

Koen.

BTW:
>> the highly interesting drops for Not-ECT UDP (for CoDel) seem to be missing (generally the labeling for figure 8 is imprecise, e.g. showing Drops ECT(1)-UDP in the first set PIE with ECT(0)-UDP)

These results are not missing, only the legend is indeed missing the brown colored "Drop Not-ECT" label, but the plots show these values (in brown). Also ECT(1) label in the legend should say ECT(0/1) which depends on whether ECT(0/1) is used in the experiment (as in the labels under the results).

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Friday, February 25, 2022 11:32 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Black, David <[email-masked]>; Neal Cardwell <[email-masked]>; tsvwg IETF list <[email-masked]>; Bob Briscoe <[email-masked]>
Subject: Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

Hi Koen,

from your link:

Regards
	Sebastian

> The test case of Jonathan is approximated by the 70Mbps non-responsive ECT(1) UDP traffic on a 100Mbps link on a DualPI2 (Prague+Cubic) test case. In Jonathan's case it was 40Mbps on a 50Mbps link. We also evaluated in extreme when sending at 100Mbps non-responsive ECT(1) UDP traffic on a 100Mbps link, and even exceeding at 140Mbps and 200Mbps. You will see the results are as if it is on a Single Q PIE AQM. Note also that CoDel which never drops ECT packets, causes actually close to starvation and high tail-drop delay results as shown in figure 1, even with ECT(0). So I guess all the concerns about FQ_CoDel and tunnels/Hash-collisions are equally severe and not related to L4S alone (can just be exploited by ECT(0) traffic today already!!).
>  
> Koen.
>  
> <[email-masked]>; Jonathan Morton <[email-masked]>; Bob Briscoe 
> <[email-masked]>; Black, David <[email-masked]>
> So, I saw that explanation - could someone check the "running code" to make sure that the coupling and marking occur even when the L queue is always empty?
>  
> Thanks, --David
>  
> [sce.dnsmgr.net] That assertion is inconsistent with the functioning of the Dual-Q algorithm, as described in:

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 27 Feb 2022 20:44:46 +0100

Dear Koen,

I have not spent the time to verify that I understand your claims an that I agree with them, (I have a hard time understanding how dropping packets that the head of a queue can be called "taildrop" with a straight face and hence am not willing to give you the benefit of the doubt, sorry).
	But on a purely logical level, how can a real attack vector for CoDel you might or might not have detected remedy the fact that DualQ essentially established a new fast-lane for unresponsive paced flows that manage to stay below L4S weighted schedulers L4S capacity? 

> On Feb 27, 2022, at 18:57, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> The important plots to check are the throughput plots in figure 1, where PIE and DualPI2 have very similar throughput profiles (confirming a coupled DualQ works like a single Q),

	As I said that is not enough, the whole premise of DualQ is that L4S and traditional traffic can not be mixed, so regressing to single queue behavior indicates that DualQ fails to meet its promise... 

> and figure 3 where PIE and DualPI2 have a queue latency at the target. This can only be achieved if the drop and marks are correctly set by the AQM.

	But is this actually showing more than the known fact that PIE and CoDel have different strictness when interpreting the reference target parameter? To me this looks not really unexpected.

> Only in the CoDel case the ECT(0) non-responding UDP flow takes practically all the link capacity (99.9% when sending unresponsive UDP-ECT(0) at 100Mbps, and really 100% when sending at the higher 140 and 200Mbps rates, probably killing "all" not-ECT traffic; see other mail to Dave Taht).
> These results speak undeniably for themselves.

	That is, as always, a contingent upon interpretation; I am not convinced that your interpretation is necessarily the best objective one available.

> Your hunch and the issue which is now mentioned in the interim slides and the shepherd's writeup is just wrong.

	Excuse me, there is no data indicating that a faked ECT(1) unresponsive-to-CE flow (especially one staying below the L-queues capacity share) will get the same amount of actual drops in the L-queue as it would in a singe-queue CoDel or FIFO. I might not be seeing the forrest for the trees here, so please explain how the existing data shows that my hunch is wrong. But with the additional information about figure 8, I object to your claim, my "hunch [...] is just wrong". The data confirms what I described as my hunch, for cases when when the "illegitimate" flow stays below the L4S AQM's L-queue capacity.
	As I implied before this situation is not as exotic as you might think, given that most access links are << than core links, and L4S apparently is targeted for deployment on core links, no? So no, my flow might not noticeably affect the L-2-C "fairness at a congested L4S AQM but I might still be able to gain an unfair and undeserved throughput advantage over other flows at that link that runs counter to the L4S claims of rough equitable sharing between flows.

> This non-issue should just be removed from the writeup and I suggest you make it an issue for the CoDel/FQ-CoDel RFCs instead. There we detected a "real" attack vector!

	Yeah, you essentially demonstrated that CoDel in itself is not as DOS resilient as it would be desirable. As Dave already mentioned fq_codel partly takes the sting out of this by restricting the fall-out mostly to the hash-bucket housing the offending flow. Also note how the stress-dropper takes care to find the largest queue (which will house the DOS flow) so even batch-emergency dropping does mostly the right thing. A sufficiently motivated attacker obviously will spoof/randomize at least SRC ports or addresses causing more problems for FQ, but my concern really is not so much DOS resistance but simple ways to gain fast-lane access for actual useful data transfer and there address spoofing will make things much harder, than just rate-limiting and fake-ECT(1) marking a flow...
As I see it if I would simply fake ECT(1) mark capacity-seeking non-ECN-honoring flows L4S will grant me a priority lane for data as long as I a) pace packets sufficiently well and b) stay below the L4S bottlenecks L-queue capacity.

> These results are not missing, only the legend is indeed missing

	Which for someone not intimately familiar with the data boils down to the same consequence, thanks for clearing this up.

> the brown colored "Drop Not-ECT" label, but the plots show these values (in brown). Also ECT(1) label in the legend should say ECT(0/1)

	I had figured that out from context ;)

> which depends on whether ECT(0/1) is used in the experiment (as in the labels under the results).

	Ah, okay with that additional information figure 8 now confirms my hunch just perfectly, no drops for unresponsive UDP in L-queue versus drops in C-queue as long as it stays below L-queue capacity. That is in direct conflict with your argument above that my "hunch [...] is just wrong", no?

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Feb 2022 13:17:49 +0000

Sebastian,

> 	But on a purely logical level, how can a real attack vector for CoDel you might or might not have detected remedy the fact that DualQ essentially established a new fast-lane for unresponsive paced…

[BB] Because Jonathan's claim in his write-up that this is a new 
fast-lane is false.
Also, in the heading, Jonathan says "This bonus is easily exploited by 
unscrupulous senders _without_ disabling congestion control."
But clearly, the flow's congestion control is not responding to the ECN 
markings coupled across from the C queue. So this statement must also be 
false.

Reasoning:
By deliberate design, the DualQ treats an unresponsive flow the same as 
a FIFO AQM (like PIE) would. If Jonathan ran a similar experiment but 
classified that unresponsive flow into the C queue, whether it was 
ECN-capable or not, it would have also got 40Mb/s.

So, there is no new fast-lane.  There is no new throughput bonus. This 
is just the same 40Mb/s that any unresponsive 40Mb/s flow has always got 
in a single queue (AQM or not, ECN or not).

This is the oft-stated deliberate design intent of the DualQ. Per-flow 
rate policing (and/or per-flow latency policing) can be added as an 
optional extra, but only if the operator wants it. The vast majority of 
the Internet currently works fine without per-flow policing.

So far, I am solely focusing on Jonathan's scenario (40Mb/s in a 50Mb/s 
link). Koen's experiments show CoDel falls down when unresponsive 
flow(s) send more than the link capacity, but I want to take this one 
scenario at a time.

Bob

>
>
>> On Feb 27, 2022, at 18:57, De Schepper, Koen (Nokia - BE/Antwerp)<[email-masked]>  wrote:
>>
>> Hi Sebastian,
>>
>> From: Sebastian Moeller<[email-masked]>  
>> To: De Schepper, Koen (Nokia - BE/Antwerp)<[email-masked]>
>> Cc: Black, David<[email-masked]>; Neal Cardwell<[email-masked]>; tsvwg IETF list<[email-masked]>; Bob Briscoe<[email-masked]>
>>> On Feb 25, 2022, at 19:30, De Schepper, Koen (Nokia - BE/Antwerp)<[email-masked]>  wrote:
>>>
>>> Hi David,
>>>
>>> From: Black, David<[email-masked]>
>>> To: Neal Cardwell<[email-masked]>
>>> <[email-masked]>; Jonathan Morton<[email-masked]>; Bob Briscoe
>>> <[email-masked]>; Black, David<[email-masked]>
>>> From: Neal Cardwell<[email-masked]>
>>> On Fri, Feb 25, 2022 at 11:56 AM Black, David<[email-masked]>  wrote:
>>> Koen,
>>>
>>> From: tsvwg<[email-masked]>  On Behalf Of De Schepper, Koen
>>> (Nokia - BE/Antwerp)
>>> From: tsvwg<[email-masked]>  On Behalf Of De Schepper, Koen
>>> (Nokia - BE/Antwerp)
>>> To: tsvwg IETF list<[email-masked]>; Jonathan Morton
>>> <[email-masked]>

---

From: Luca Muscariello <[email-masked]>
Date: Mon, 28 Feb 2022 14:32:24 +0100

On Mon, Feb 28, 2022 at 2:18 PM Bob Briscoe <[email-masked]> wrote:

> [BB] Because Jonathan's claim in his write-up that this is a new fast-lane
> is false.
> unscrupulous senders *without* disabling congestion control."
> By deliberate design, the DualQ treats an unresponsive flow the same as a
> FIFO AQM (like PIE) would. If Jonathan ran a similar experiment but

Beyond the specific experiment, I do not think this is totally accurate. As
a stationary regime
a FIFO AQM and DualQ may converge to the same bandwidth sharing condition
wrt to unresponsive
traffic but in the FIFO AQM this would happen through a fight against, say
TCP. So the unresponsive flow would get packet losses during that fight. In
dualQ the unresponsive flow would get not a single loss to my
understanding, thanks to the prioritization behavior either via strict
priority or WFQ scheduling cycles.

>
> So, there is no new fast-lane.  There is no new throughput bonus. This is
> just the same 40Mb/s that any unresponsive 40Mb/s flow has always got in a
> single queue (AQM or not, ECN or not).
>
> optional extra, but only if the operator wants it. The vast majority of the
> Internet currently works fine without per-flow policing.
>
>
> link). Koen's experiments show CoDel falls down when unresponsive flow(s)
> send more than the link capacity, but I want to take this one scenario at a
> time.
>
>
> Bob
>
> On Feb 27, 2022, at 18:57, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> <[email-masked]> wrote:
>
> Hi Sebastian,
>
> From: Sebastian Moeller <[email-masked]> <[email-masked]>
> To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> <[email-masked]>
> Cc: Black, David <[email-masked]> <[email-masked]>; Neal Cardwell <[email-masked]> <[email-masked]>; tsvwg IETF list <[email-masked]> <[email-masked]>; Bob Briscoe <[email-masked]> <[email-masked]>
> On Feb 25, 2022, at 19:30, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> <[email-masked]> wrote:
>
> Hi David,
>
> From: Black, David <[email-masked]> <[email-masked]>
> To: Neal Cardwell <[email-masked]> <[email-masked]>
> Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> <[email-masked]>; tsvwg IETF list <[email-masked]> <[email-masked]>; Jonathan Morton <[email-masked]> <[email-masked]>; Bob Briscoe <[email-masked]> <[email-masked]>; Black, David <[email-masked]> <[email-masked]>
> From: Neal Cardwell <[email-masked]> <[email-masked]>
> On Fri, Feb 25, 2022 at 11:56 AM Black, David <[email-masked]> <[email-masked]> wrote:
> Koen,
>
> From: tsvwg <[email-masked]> <[email-masked]> On Behalf Of De Schepper, Koen
> (Nokia - BE/Antwerp)
> From: tsvwg <[email-masked]> <[email-masked]> On Behalf Of De Schepper, Koen
> (Nokia - BE/Antwerp)
> To: tsvwg IETF list <[email-masked]> <[email-masked]>; Jonathan Morton <[email-masked]> <[email-masked]>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 28 Feb 2022 14:02:17 +0000

Hi David, Luca,

>> I'm just reading the draft and cannot comment on the code, but it would be intuitive that a coupled queuing systems of two queues would decouple when one of the two queues is empty.

The coupled probability MUST NOT take the L4S minimum queue size into account as it would indeed disable the coupling under normal conditions (the L4S queue is empty most of the time if there is more than 10% classic traffic).

Our Linux running code is checking the total backlog (from both queues) to disable drop, so only if the sum of bytes in “both” queues is less than 2 MTUs, drop is disabled, otherwise everything dropped/marked as defined by the AQMs, see: linux/sch_dualpi2.c at testing · L4STeam/linux (github.com)<https://github.com/L4STeam/linux/blob/testing/net/sched/sch_dualpi2.c#L273>. This is allowed of course, as it disables drop for all traffic before the coupling. The STEP marking on the other hand is only disabled when at enqueue more than 1 packet is in the L queue, see: linux/sch_dualpi2.c at testing · L4STeam/linux (github.com)<https://github.com/L4STeam/linux/blob/testing/net/sched/sch_dualpi2.c#L395>.

In the draft, the pseudo code does the >1 packet queued check only for marking L4S traffic with the L4S STEP/RAMP (see lines 5a-d in figure 7), not for the Coupled drop/mark (line 6 in figure 7) which is not impacted by that check.

Testing would immediately make it clear though that something is wrong as it would always give a high throughput to L4S traffic even with normal responsive L4S flows, leaving Classic responsive flows with little more than the 10% share given by the WRR. So it wouldn’t even work for responsive flows, so even if an implementation would never be tested for non-responsive flows: if the coupling works correctly for responsive flows, it will also work correctly for non-responsive flows.

If you think any corrections or additional (more explicit) text is needed to make sure people don’t mistakenly disable the coupled mark and drop when the L-queue is empty, let us know.

Thanks,
Koen.

From: tsvwg <[email-masked]> On Behalf Of Luca Muscariello
Sent: Saturday, February 26, 2022 12:16 AM
To: Black, David <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>; Bob Briscoe <[email-masked]>
Subject: Re: [tsvwg] Related to "Non-L4S traffic abusing the L-queue" discussion during the interim

Hi David

I'm just reading the draft and cannot comment on the code, but it would be intuitive that a coupled queuing systems of two queues would decouple when one of the two queues is empty.

Reading the details of the pseudo code in the draft, if lq is empty p'L should be 0 and pL=pCL=kp' which may be !=0.

Additional text below the pseudo code provides details about a Linux implementation that marks packet IFF queue length is >= 2 MTU. Which may suggest that decoupling happens when the L queue is empty.

I will read the document more carefully beginning of next week but I admit this part is ambiguous.

Luca

On Fri, Feb 25, 2022, 19:05 Black, David <[email-masked]<mailto:[email-masked]>> wrote:
Hi Neal,

Thanks, --David

[EXTERNAL EMAIL]

thanks,
neal

[EXTERNAL EMAIL]
Hi Jonathan,

Koen.

Hi Jonathan,

Thanks,
Koen.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Feb 2022 14:35:02 +0000

Luca, Koen's just answered your first email. This is in response to your 
second on this thread. See [BB]

On 28/02/2022 13:32, Luca Muscariello wrote:
>     Also, in the heading, Jonathan says "This bonus is easily
>     exploited by unscrupulous senders _without_ disabling congestion
>     control."
>     But clearly, the flow's congestion control is not responding to
>     the ECN markings coupled across from the C queue. So this
>     statement must also be false.
>
>     Reasoning:
>     By deliberate design, the DualQ treats an unresponsive flow the
>     same as a FIFO AQM (like PIE) would. If Jonathan ran a similar
>     experiment but classified that unresponsive flow into the C queue,
>     whether it was ECN-capable or not, it would have also got 40Mb/s.
>
>
> Beyond the specific experiment, I do not think this is totally 
> accurate. As a stationary regime
> a FIFO AQM and DualQ may converge to the same bandwidth sharing 
> condition wrt to unresponsive
> traffic but in the FIFO AQM this would happen through a fight against, 
> say TCP. So the unresponsive flow would get packet losses during that 
> fight. In dualQ the unresponsive flow would get not a single loss to 
> my understanding, thanks to the prioritization behavior either via 
> strict priority or WFQ scheduling cycles.

[BB] The DualQ introduces drop of ECT packets when AQM marking exceeds 
an overload threshold (25% mark or drop in the C queue). It applies drop 
of ECT packets to /both/ queues with the same probability (IOW, the 
coupling for drop is equal, whereas for ECN marking it uses the square 
relationship).

See the 'else' at line 8b after the 'if' at line 4b in Fig 7 and the 
associated explanation in the DualQ draft.
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-aqm-dualq-coupled-21#appendix-A.2

If you meant something different by "beyond the specific experiment" 
please say. I've guessed you meant that they are still both ECN-capable 
flows but, due to transient events, the AQM marking probability 
temporarily becomes high enough for the AQM to introduce drop of ECT 
packets. I think you thought this only happens in the C queue, but the 
DualQ applies drop to both queues, as just explained above (and recently 
discussed on this list regarding the MUSTs/SHOULDs around this).

Classic AQMs (like PIE or even RED) don't start dropping ECT packets 
based on the instantaneous queue - they smooth their response. The DualQ 
is no different. It determines whether it starts introducing drop to ECT 
packets (in both queues equally) based on the probability from the Base 
AQM, which is already smoothed.

Bob

>
>
>
>     So, there is no new fast-lane.  There is no new throughput bonus.
>     This is just the same 40Mb/s that any unresponsive 40Mb/s flow has
>     always got in a single queue (AQM or not, ECN or not).
>
>     This is the oft-stated deliberate design intent of the DualQ.
>     Per-flow rate policing (and/or per-flow latency policing) can be
>     added as an optional extra, but only if the operator wants it. The
>     vast majority of the Internet currently works fine without
>     per-flow policing.
>
>
>     So far, I am solely focusing on Jonathan's scenario (40Mb/s in a
>     50Mb/s link). Koen's experiments show CoDel falls down when
>     unresponsive flow(s) send more than the link capacity, but I want
>     to take this one scenario at a time.
>
>
>     Bob
>
>>>     On Feb 27, 2022, at 18:57, De Schepper, Koen (Nokia - BE/Antwerp)<[email-masked]>  <mailto:[email-masked]>  wrote:
>>>
>>>     Hi Sebastian,
>>>
>>>     From: Sebastian Moeller<[email-masked]>  <mailto:[email-masked]>  
>>>     To: De Schepper, Koen (Nokia - BE/Antwerp)<[email-masked]>  <mailto:[email-masked]>
>>>     Cc: Black, David<[email-masked]>  <mailto:[email-masked]>; Neal Cardwell<[email-masked]>  <mailto:[email-masked]>; tsvwg IETF list<[email-masked]>  <mailto:[email-masked]>; Bob Briscoe<[email-masked]>  <mailto:[email-masked]>
>>>>     On Feb 25, 2022, at 19:30, De Schepper, Koen (Nokia - BE/Antwerp)<[email-masked]>  <mailto:[email-masked]>  wrote:
>>>>
>>>>     Hi David,
>>>>
>>>>     CoDel AQMs |l4steam.github.io  <http://l4steam.github.io>
>>>>
>>>>     From: Black, David<[email-masked]>  <mailto:[email-masked]>
>>>>     To: Neal Cardwell<[email-masked]>  <mailto:[email-masked]>
>>>>     <[email-masked]>  <mailto:[email-masked]>; tsvwg IETF list
>>>>     <[email-masked]>  <mailto:[email-masked]>; Jonathan Morton<[email-masked]>  <mailto:[email-masked]>; Bob Briscoe
>>>>     <[email-masked]>  <mailto:[email-masked]>; Black, David<[email-masked]>  <mailto:[email-masked]>
>>>>     From: Neal Cardwell<[email-masked]>  <mailto:[email-masked]>
>>>>     On Fri, Feb 25, 2022 at 11:56 AM Black, David<[email-masked]>  <mailto:[email-masked]>  wrote:
>>>>     Koen,
>>>>
>>>>     [sce.dnsmgr.net  <http://sce.dnsmgr.net>] That assertion is inconsistent with the functioning of the Dual-Q algorithm, as described in:
>>>>     [ietf.org  <http://ietf.org>]
>>>>
>>>>     https://mailarchive.ietf.org/arch/msg/tsvwg/joFr3sfOrxxkYhWdYrO2rLlCNU
>>>>     w/ [mailarchive.ietf.org  <http://mailarchive.ietf.org>]
>>>>
>>>>     thanks,
>>>>     neal
>>>>
>>>>
>>>>
>>>>     From: tsvwg<[email-masked]>  <mailto:[email-masked]>  On Behalf Of De Schepper, Koen
>>>>     (Nokia - BE/Antwerp)
>>>>     From: tsvwg<[email-masked]>  <mailto:[email-masked]>  On Behalf Of De Schepper, Koen
>>>>     (Nokia - BE/Antwerp)
>>>>     To: tsvwg IETF list<[email-masked]>  <mailto:[email-masked]>; Jonathan Morton
>>>>     <[email-masked]>  <mailto:[email-masked]>

---

From: Luca Muscariello <[email-masked]>
Date: Mon, 28 Feb 2022 15:54:29 +0100

On Mon, Feb 28, 2022 at 3:35 PM Bob Briscoe <[email-masked]> wrote:

> Beyond the specific experiment, I do not think this is totally accurate.
> As a stationary regime
> [BB] The DualQ introduces drop of ECT packets when AQM marking exceeds an
> overload threshold (25% mark or drop in the C queue). It applies drop of
> ECT packets to /both/ queues with the same probability (IOW, the coupling
> for drop is equal, whereas for ECN marking it uses the square
> relationship).
>
> flows but, due to transient events, the AQM marking probability temporarily
> becomes high enough for the AQM to introduce drop of ECT packets. I think
> you thought this only happens in the C queue, but the DualQ applies drop to
> both queues, as just explained above (and recently discussed on this list
> regarding the MUSTs/SHOULDs around this).
>

I was thinking about unresponsive traffic.
In maybe better English: In general, in presence of unresponsive traffic a
FIFO AQM and dualQ
may converge to the same stationary bandwidth sharing, i.e. the
unresponsive traffic gets all input rate while TCP gets the residual
capacity. Hope this is more explicit.

My remark was that in a FIFO AQM, unresponsive traffic may need to generate
quite a lot of queuing to make TCP backoff before gaining all the share of
bandwidth. In the case of dualQ, unresponsive traffic would get that share
immediately as it is obtained by packet prioritization and not by forcing
TCP to backoff.

>
> Classic AQMs (like PIE or even RED) don't start dropping ECT packets based
> on the instantaneous queue - they smooth their response. The DualQ is no
> different. It determines whether it starts introducing drop to ECT packets
> (in both queues equally) based on the probability from the Base AQM, which
> is already smoothed.
>
>
>
> Bob
>
>
>
>
>
>>
>> w/ [mailarchive.ietf.org]
>>
>> thanks,
>> neal
>>
>>
>>

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Feb 2022 17:51:25 +0000

Luca,

On 28/02/2022 14:54, Luca Muscariello wrote:
>     Luca, Koen's just answered your first email. This is in response
>     to your second on this thread. See [BB]
>
>>         [BB] Because Jonathan's claim in his write-up that this is a
>>         new fast-lane is false.
>>         exploited by unscrupulous senders _without_ disabling
>>         congestion control."
>>         But clearly, the flow's congestion control is not responding
>>         to the ECN markings coupled across from the C queue. So this
>>         By deliberate design, the DualQ treats an unresponsive flow
>>         the same as a FIFO AQM (like PIE) would. If Jonathan ran a
>>         similar experiment but classified that unresponsive flow into
>>         the C queue, whether it was ECN-capable or not, it would have
>>         also got 40Mb/s.
>>
>>
>>     traffic but in the FIFO AQM this would happen through a fight
>>     against, say TCP. So the unresponsive flow would get packet
>>     losses during that fight. In dualQ the unresponsive flow would
>>     get not a single loss to my understanding, thanks to the
>>     prioritization behavior either via strict priority or WFQ
>>     scheduling cycles.
>
>     [BB] The DualQ introduces drop of ECT packets when AQM marking
>     exceeds an overload threshold (25% mark or drop in the C queue).
>     It applies drop of ECT packets to /both/ queues with the same
>     probability (IOW, the coupling for drop is equal, whereas for ECN
>     marking it uses the square relationship).
>
>     See the 'else' at line 8b after the 'if' at line 4b in Fig 7 and
>     the associated explanation in the DualQ draft.
>     If you meant something different by "beyond the specific
>     experiment" please say. I've guessed you meant that they are still
>     both ECN-capable flows but, due to transient events, the AQM
>     marking probability temporarily becomes high enough for the AQM to
>     introduce drop of ECT packets. I think you thought this only
>     happens in the C queue, but the DualQ applies drop to both queues,
>     as just explained above (and recently discussed on this list
> In maybe better English: In general, in presence of unresponsive 
> traffic a FIFO AQM and dualQ
> My remark was that in a FIFO AQM, unresponsive traffic may need to 
> generate quite a lot of queuing to make TCP backoff before gaining all 
> the share of bandwidth. In the case of dualQ, unresponsive traffic 
> would get that share immediately as it is obtained by packet 
> prioritization and not by forcing TCP to backoff.

[BB] You're right that the dynamics will differ, but...
The longer it takes the TCP to back off, the /higher/ its share of 
capacity until it does. The priority scheduler serves more of the L 
packets earlier, but it doesn't change capacity shares (how many of each 
type of packet); only the sources can do that. At least, not unless 
there's unequal drop, but there isn't. Drop can only occur if the DualQ 
enters overload mode, and then the DualQ ensures the drop probability is 
equal in both queues, as I've explained. That's the same as a FIFO AQM.

Walking through this step-by-step:

==In the first RTT before any response, ==
1. The queue growth in the single FIFO queue case will be the same as 
the growth in the sum of the two Qs in the DualQ case. However, the C 
queue will grow more and the L queue less, because the L queue has 
higher scheduling priority.
2. So as the TCP traffic leaves the FIFO AQM it is more mixed-in with 
the unresponsive traffic. Whereas, in the DualQ, more of the TCP traffic 
leaves later.

These effects will lead to two opposing outcomes:
     a. #1 causes the Classic AQM to increase its mark/drop probability 
faster than a FIFO AQM would
     b. #2 causes the congestion signals on the Classic TCP packets to 
leave the queue later than they would have from a FIFO AQM, so TCP will 
receive them later.

It's not clear to me which will be the strongest effect. But it's not 
appropriate to say "the unresponsive traffic gets that share 
immediately" because the capacity share depends on what the sources 
send, and that hasn't changed yet.

==In subsequent rounds==
Classic TCPs only respond once per RTT and by the same reduction 
whatever the degree of queuing. So, if TCP responds slightly later in 
the DualQ case, it will remain slower to respond, and continue to hold a 
/higher/ share of the capacity for longer.

One would have to test the dynamics, but altho' I think the capacity 
share advantage might be the opposite of what you say, the difference 
will only last 2 or 3 rounds, so it's trivial either way.

Bob

>
>     Classic AQMs (like PIE or even RED) don't start dropping ECT
>     packets based on the instantaneous queue - they smooth their
>     response. The DualQ is no different. It determines whether it
>     starts introducing drop to ECT packets (in both queues equally)
>     based on the probability from the Base AQM, which is already smoothed.
>
>
>
>     Bob
>
>>
>>
>>
>>         So, there is no new fast-lane.  There is no new throughput
>>         bonus. This is just the same 40Mb/s that any unresponsive
>>         40Mb/s flow has always got in a single queue (AQM or not, ECN
>>         or not).
>>
>>         Per-flow rate policing (and/or per-flow latency policing) can
>>         be added as an optional extra, but only if the operator wants
>>         it. The vast majority of the Internet currently works fine
>>         without per-flow policing.
>>
>>
>>         So far, I am solely focusing on Jonathan's scenario (40Mb/s
>>         in a 50Mb/s link). Koen's experiments show CoDel falls down
>>         when unresponsive flow(s) send more than the link capacity,
>>         but I want to take this one scenario at a time.
>>
>>
>>         Bob
>>