Subject: Re: [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

From: Sebastian Moeller <[email-masked]>
Date: Sun, 1 Nov 2020 09:21:56 +0100

Hi Koen,

more below in-line, prefixed [SM2]

Gesendet: Sonntag, 01. November 2020 um 00:42 Uhr
Von: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
An: "Sebastian Moeller" <[email-masked]>, "Bob Briscoe" <[email-masked]>
Cc: "tsvwg IETF list" <[email-masked]>, "iccrg IRTF list" <[email-masked]>, "TCP Prague List" <[email-masked]>
Betreff: RE: [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

Hi Sebastian,
 
>> I appreciate that team L4S is finally admitting that it over-promised and under-delivered
 
I think the problem is that you interpret “requirements” as “promises”. 

        [SM2] I disagree, with promises, I typically address the claims about what L4S will achieve. The stated goal was that L4S will do no harm to non-L4S traffic (by e.g. adapting to rfc3168 bottlenecked paths by switching away from its CE interpretation) but the proposed text, basically changes that into a "the operator is free to do as she/he pleases". That is a considerable reduction in the promise to do no harm.
      Or are you saying that "do no harm" was never a design goal for L4S but only adopted as it was dragged in by external requirements? If true, it would explain a lot.

Most people understand the potential of L4S 

        [SM2] Most people have not even heard about L4S, and even in the group of people that have, the majority probably from reading your internet drafts without actually looking into the few papers presenting data tht allows to evaluation whether the L4S drafts are realistic. IMHO that means, that quite a lot of people that know about L4S understand its promises. Now, if by potential you admit uncertainty whether this potential can actually be realised in the real-world internet then I agree. I am just convinced that a more productive path forward would have been to work harder on figuring out whether L4S potential can be realised, before trying to accelerate the ratification proces.

and accept that special care needs to be taken to protect Single Q Classic ENC bottlenecks.

        [SM2] But the proposed change actually deemphasises this, from a mandate to protect, it now is a mandate to detect and if one pleases to protect. This seems like an odd way to protection.

 The draft needs to put the attention to (potential) problems that need to be solved or avoided by interesting parties that want to develop their L4S Prague-compliant CCs. So it is in the best interest for CC developers to push for feasible/realistic requirements. 

        [SM2] Well, the realistic part would have been something like "A scalable congestion control MUST do its best to react to ECN marking from a non-L4S but ECN-capable bottleneck in a way that will coexist with a TCP Reno congestion control" The change to react from "MUST react" to "SHOULD be capable (dependent on configuration) of automatically adapting", does not strike me as an improvement in either feasibility nor realism, it is simply a relaxation of a requirement, that proved harder to achieve than expected, no?

I heard quite some people (including you) being concerned about the feasibility of detecting a Single Queue Classic ECN AQM, 

        [SM2] Well, testing data indicated, that the proposed detector was/is riddled with false detections and missing detection, but the quality of the detector is orthogonal to what to do with the result of said detector, so please do not conflate these issues.

and said they would rather apply other mechanisms to avoid issues with these types of bottlenecks (ref operational guidelines that tries to collect all useful ideas that were shared during list discussions). 

        [SM2] To which, I have sent comments, against my better judgement of that being busy work.

So the changes announced by Bob should reflect this. I understood you are quite OK with the wording? If not, then your suggestions are very welcome.

        [SM2] No, I am not not okay with Bob's proposal, and I wonder how you could come to that conclusion. As long as that issue exists (because the meaning of CE is redefined in a rfc3168 incompatible fashion) L4S protocols, IMHO, "MUST adapt" which also includes a requirement that they MUST be capable. I hope that this is explicit enough to ne being mis understood as an endorsement for the proposed change. I explicitly stated "make both of these MUSTs" which clearly indicates that I disagree with the relaxation of the requirements, even though I admit, it does not cover my full criticism. Again, how you can interpret that as being " OK with the wording" puzzles me to no end.

>> "In the unlikely case of the L4S experiment being declared a failure, this replacement will need to be permanent, and the ECT(1) responsive elements in AQM nodes also needs to be disabled".
 
L4S will succeed if 1) L4S capable network nodes get deployed (including Classic ECN bottlenecks being upgraded) and 2) CCs are deployed that can meet the TCP Prague requirements (is not disrupting Classic flows and other L4S flows) AND can show great benefits. If one of those or both will not happen (like with Classic ECN deployment) it fails!! So I guess no need to worry about disabling permanently if it never gets deployed 😊.

        [SM2] Well, let' try logic, for a change, let's ignore the wishful thinking of "including Classic ECN bottlenecks being upgraded" as L4S needs to prove its merits in the existing internet, not a hypothetical one, if L4S nodes get deployed 1) and L4S compatible CCs also get deployed, that still leaves the point that they need to "show great benefits". If that important result will not be achieved than disabling the AQM and CC nodes already deployed seems an important measure to keep a failed experiment from a) wasting a precious IP header code-point ad infinitum and to keep complexity of TCP stacks low. Again I fail to understand how your argument can be considered logically convincing.

Best Regards

Sebastian

Thanks and regards,
Koen.

From: Sebastian Moeller <[email-masked]>
Sent: Saturday, October 31, 2020 3:54 PM
To: Bob Briscoe <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>; iccrg IRTF list <[email-masked]>; TCP Prague List <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Subject: Aw: [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

Dear Koen,

since Bob claims that he intends to correspond with my input, I am addressing this response to you, See my comments in-line prefixed [SM].

Tl;dr: I appreciate that team L4S is finally admitting that it over-promised and under-delivered, I am less happy about the solution to this issue by simply reducing the requirments ot match the unsatisfactory state of the L4S implementation. After years of basically adveritizing on these requirements, watering those down at the last minute does not qualify as a good faith effort in my book. It does confirm my negative view on team L4S' acumen and confirms my judgement that L4S "offers too little too late". Unfortunately I do not kid myself that this obvious fact is going to stop the current intenet drafts gaining RFC status without any meanigful changes. 

Gesendet: Samstag, 31. Oktober 2020 um 10:54 Uhr
Von: "Bob Briscoe" <[email-masked][mailto:[email-masked]]>
An: "tsvwg IETF list" <[email-masked][mailto:[email-masked]]>
Cc: "iccrg IRTF list" <[email-masked][mailto:[email-masked]]>, "TCP Prague List" <[email-masked][mailto:[email-masked]]>, "De Schepper, Koen (Koen)" <[email-masked][mailto:[email-masked]]>
Betreff: [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

Folks,

The co-authors of ECN L4S ID have been reviewing the correctness of the normative 'Prague' requirements.

        [SM] This seems less about correctness and more about whether you managed ot hit those targets with your implementation. Changing those requirements post-hoc might be justified, but pleas do not frame that as a correctness isssue.

    See https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3[https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3]
This is the second of 2 emails, about 2 of the requirements that we think ought to be reworded a little.

If you agree with the rationale, but think the new wording still doesn't fully capture the requirement, pls suggest sthg better.
If you disagree with the rationale, pls discuss.
4.3.  Prerequisite Congestion Response
...
CURRENT:
 
   o  A scalable congestion control MUST react to ECN marking from a
      non-L4S but ECN-capable bottleneck in a way that will coexist with
      a TCP Reno congestion control [RFC5681[https://tools.ietf.org/html/rfc5681]] (see Appendix A.1.4[https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#appendix-A.1.4] for
      rationale).
 
      Note that a scalable congestion control is not expected to change
      to setting ECT(0) while it falls back to coexist with Reno.

PROPOSED:
   o  A scalable congestion control MUST implement monitoring in order
      to detect a likely non-L4S but ECN-capable AQM at the bottleneck.
      On detection of a likely ECN-capable bottleneck it SHOULD be
      capable (dependent on configuration) of automatically adapting its
      congestion response to coexist with TCP Reno congestion controls
      [RFC5681] (see Appendix A.1.4 for rationale and a referenced
      algorithm).

        [SM] This is a significantly watering down, and it seems rather non-logical, IFF an implementation does not want/need to actually react to the result of a detection then performing that detection seems pure busy work. So make both of these MUSTs. Otherwise your requirments pretend to care about existing rfc3168 AQMs but in name only. 

      Note that a scalable congestion control is not expected to change
      to setting ECT(0) while it falls back to coexist with Reno.

RATIONALE:
1/ The requirement as currently written says what an omniscient sender MUST do. So there's an implied requirement that a sender MUST be omniscient, which is of course impossible.

2/ The requirement needs to be recast to require a sender to aim to be as knowledgeable as possible. Then, what it does as a result needs to take into account the a priori likelihood of there being a non-L4S bottleneck present.
3/ This includes the possibility that the operator of the host knows that the network it serves has not deployed any single queue classic ECN AQM (e.g. in a CDN case they're doing out of band testing, or they've asked the ISP). So we've included the possibility of fall-back being disabled by configuration.
4/ Nonetheless, as has been pointed out on the list, there is still a possibility that there is a Classic ECN AQM somewhere else on the path (to continue the CDN example, perhaps beyond the ISP in a home network). The 'MUST monitor' requirement still stands to ensure the operator doesn't miss these cases.
5/ Then, if the server operators have disabled fall-back for their deployment, they can reconsider their policy or at least do more focused testing if they are frequently detecting a single-queue Classic ECN AQM.

        [SM] And we are back to engineering/network administration by wishful thinking... Again with a syszem (L4S) that has not yet been proven to actually work over the existing internet with the sole exception of short RTT/low hop count paths (CDN to end-user fast-lanes copme to mind).

Items 3-5 are the "react via management" model that I've talked about on the list, given the unfairness doesn't amount to starvation, and it is possible that the prevalence of the problem is very low.

        [SM] Given that you never offered your own specific defintion of what starvation entails, I fail to see how that claim can b tre in a verifiable fashion?

Finally, after the bullet list of requirements in section 4.3, (which are prerequisites for setting the ECT1 codepoint), we propose to add the following requirement, as suggested on the tsvwg list:

      To participate in the L4S experiment, a scalable congestion control MUST
      be capable of being replaced by a Classic congestion control (by
      application and by administrative control). A Classic congestion control
      will not tag its packets with the ECT(1) codepoint.

        [SM] +1; I would appreciate if we could add a sentence like:

"In the unlikely case of the L4S experiment being declared a failure, this replacement will need to be permanent, and the ECT(1) responsive elements in AQM nodes also needs to be disabled".

I added the unlikely in spite of my own predictions, to account for the fact that the drafts are written on the hypothesis that L4S works as advertized.... 

Best

        Sebastian

Cheers

Bob
 
--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/[http://bobbriscoe.net/]

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 1 Nov 2020 14:02:28 +0100

Dear Christian,

I do not really want to drag you into the L4S protocol requirements issue, but since you seem to be contemplating making your protocol L4S-aware and -compliant, what is your interpretation of the "eliminate RTT bias" requirement in its original and in Bob's proposed modified version. Did you understand that this required you to account for 15ms higher sustained RTT under load in the classic queue of an L4S AQM by slowing down the CC response by ECT(1) flows by the same 15ms? If no, could you opinie whether my proposed change for this requirement is any clearer? 
I top post this as it does not address your specific issues (but I am quite intersted in learning team L4S' response to your questions), but seems sufficiently related to merit as response (past attempts at soliciting input from L4S implementers fell on deaf ears, so I will not cold-post any such questions anymore).

Thank you very much in advance!
 
Best Regards
        Sebastian

Gesendet: Sonntag, 01. November 2020 um 02:07 Uhr
Von: "Christian Huitema" <[email-masked]>
An: "Bob Briscoe" <[email-masked]>, "tsvwg IETF list" <[email-masked]>
Cc: "iccrg IRTF list" <[email-masked]>, "TCP Prague List" <[email-masked]>, "De Schepper, Koen (Koen)" <[email-masked]>
Betreff: Re: [tsvwg] [tcpPrague] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text
I am reading the L4S ECN-AQM proposal with an eye on responding to it in an implementation of QUIC, and I have a couple of questions regarding use of ECN marking with QUIC.
The document does not mention QUIC, yet QUIC is already used in a large fraction of Internet traffic. QUIC does specify support for ECN, and QUIC acknowledgements may carry counts of each category of ECN marks received from the peer -- three counters for ECT(0), ECT(1) and CE. In theory, QUIC implementations could take advantage of L4S -- in fact, at least one implementation supports DC-TCP like CC already. Is there interest in specifying L4S for QUIC?
My next question regards the interaction of the proposed L4S ECN-AQM with CC algorithms like BBR that attempt to discover the bottleneck packet rate for the connection, and use pacing to send packets at that rate. I observed that BBR is never mentioned in the draft, yet BBR is used in a sizeable part of Internet traffic. Do we have data on how a non-L4S aware implementation of BBR interacts with the proposed L4S AQM?
My last question regards potential use of ECT(1) marking. Most current implementations set ECT(0), but setting ECT(1) instead is trivial. This should elicit an L4S compatible response in L4S-AQM, and the BBR implementation might be modified to use the signals as part of the bottleneck bandwidth tracking. But there is a small issue there. With BBR, QUIC packets are supposedly paced at just under the bottleneck rate, except during "probe" periods in which they probe for 1 RTT at a slightly higher rate. The L4S AGM might degenerate in a form of ON-OFF control -- no feedback at all most of the time, then a bunch of CE marks if the probe rate exceeds the bottleneck bandwidth. As anyone experimented with that?
-- Christian Huitema

On 10/31/2020 2:54 AM, Bob Briscoe wrote:Folks,

The co-authors of ECN L4S ID have been reviewing the correctness of the normative 'Prague' requirements.
    See https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3
This is the second of 2 emails, about 2 of the requirements that we think ought to be reworded a little.

 
PROPOSED:

RATIONALE:

Cheers

Bob

_______________________________________________
tcpPrague mailing [email-masked][mailto:[email-masked]]https://www.ietf.org/mailman/listinfo/tcpprague

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 1 Nov 2020 13:30:59 +0000

Christian,

I've changed the subject line given it's no longer appropriate.
See inline tagged [BB]...

On 01/11/2020 01:07, Christian Huitema wrote:
>
> I am reading the L4S ECN-AQM proposal with an eye on responding to it 
> in an implementation of QUIC,
>

[BB] That's good news.

> and I have a couple of questions regarding use of ECN marking with QUIC.
>
> The document does not mention QUIC, yet QUIC is already used in a 
> large fraction of Internet traffic. QUIC does specify support for ECN, 
> and QUIC acknowledgements may carry counts of each category of ECN 
> marks received from the peer -- three counters for ECT(0), ECT(1) and 
> CE. In theory, QUIC implementations could take advantage of L4S -- in 
> fact, at least one implementation supports DC-TCP like CC already. Is 
> there interest in specifying L4S for QUIC?
>

[BB] Certainly.
Nonetheless, when I search for the string "QUIC" in L4S ECN-AQM 
proposal, I find it (assuming you mean draft-ietf-tsvwg-aqm-dualq-coupled ).
But you won't find much else about QUIC there, 'cos that doc is about 
the AQM.

There are three main L4S docs.
You need to be reading the doc with the requirements for L4S transports: 
draft-ietf-tsvwg-ecn-l4s-id 
<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id>
And the L4S architecture: draft-ietf-tsvwg-l4s-arch 
<https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch>
Both of which explain that QUIC already provides the necessary feedback.

Quentin De Coninck wrote an initial implementation of a "QUIC Prague" 
congestion control based on your picoQUIC code. It can be accessed via 
the L4S landing page here:
     https://riteproject.eu/dctth/#code
He started it at an IETF hackathon. I don't think it's been maintained 
since it was first written, but it might be a start.

> My next question regards the interaction of the proposed L4S ECN-AQM 
> with CC algorithms like BBR that attempt to discover the bottleneck 
> packet rate for the connection, and use pacing to send packets at that 
> rate. I observed that BBR is never mentioned in the draft, yet BBR is 
> used in a sizeable part of Internet traffic. Do we have data on how a 
> non-L4S aware implementation of BBR interacts with the proposed L4S AQM?
>

[BB] Again, if you're looking at the L4S AQM doc, you won't find much 
about CC.
For BBR, take a look at section 5.2 of l4s-arch 
<https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-07#section-5.2>.

When BBRv2 was released it included initial support for L4S ECN. Neal 
and Yuchung presented BBRv2's L4S ECN queue delay results when they 
first talked about BBRv2 in iccrg. I'm sure more work is needed on the 
L4S ECN part, but I guess that will rise in priority once ISPs start 
deploying L4S AQMs (e.g. Greg White reported that cable implementations 
were being interop tested some time ago, and Nokia released an L4S WiFi 
AP a while back).

So, if there is an L4S AQM at the bottleneck, the resulting ECN signals 
would make BBRv2 morph into an L4S CC.

However, your question is about how a non-L4S aware BBR interacts with 
an L4S AQM.
* For that, I'll have to take 'BBR' to mean something like BBRv2, but 
without any support for ECN - i.e. without the L4S ECN part.
* And just to be super-precise, I'll take the 'L4S AQM' to mean the 
DualQ Coupled AQM (not an FQ one).
Then the answer is, I think, only a few tests of that combination have 
been done. Am I right, Olga, Asad, Olivier, Neal, Yuchung, anyone? I 
suspect this mainly because there doesn't seem to be a reason why the 
L4S ECN part of BBRv2 would be disabled.

I can say what I suspect would happen in what I think is your 
hypothetical case. The DualQ Coupled AQM design relies on a flow that is 
classified into the Classic queue (e.g.. the non-L4S aware BBRv2 you ask 
about) responding to loss and/or delay in a way that would be friendly 
to Reno. The v2 changes to BBR were intended to be more friendly to 
other flows, in particular the response to >1% loss. Then, these non-L4S 
aware BBRv2 flows that you posit should co-exist with L4S flows in some 
way. But whether they would coexist well isn't something I've looked 
into, 'cos I think it's rather a hypothetical question (if I've 
understood you correctly).

Before BBRv2 when BBRv1 had no response to loss (and no ECN support), I 
can say that it usually starved L4S flows over the same DualQ AQM for 
the same reason it usually starved Cubic and other flows in a FIFO. But 
hopefully that is in the past now.

> My last question regards potential use of ECT(1) marking. Most current 
> implementations set ECT(0), but setting ECT(1) instead is trivial. 
> This should elicit an L4S compatible response in L4S-AQM, and the BBR 
> implementation might be modified to use the signals as part of the 
> bottleneck bandwidth tracking. But there is a small issue there. With 
> BBR, QUIC packets are supposedly paced at just under the bottleneck 
> rate, except during "probe" periods in which they probe for 1 RTT at a 
> slightly higher rate. The L4S AGM might degenerate in a form of ON-OFF 
> control -- no feedback at all most of the time, then a bunch of CE 
> marks if the probe rate exceeds the bottleneck bandwidth. As anyone 
> experimented with that?
>

[BB] Yes, once the IETF assigns the codepoint, BBRv2 will be 'allowed' 
to send packets over the Internet as ECT(1). Then, yes, an L4S DualQ 
Coupled AQM will classify BBRv2 packets into the L4S queue. This should 
have a very shallow ECN marking threshold (500us - 1ms), so even if the 
flow (whether QUIC or TCP) is flying just under the available capacity, 
bunching of packets means it is unlikely to completely avoid ECN marking 
between probes. If it could avoid ECN marking, you are right that it 
would get a bump of ECN marks during the probe. I haven't studied the 
code, but when it experiences ECN marking I believe it switches into an 
L4S ECN mode for a while, and uses ECN rather than delay probing to 
track available capacity. I assume it switches back to BBR's delay 
probing mode if it gets no ECN for a while (e.g. the bottleneck might 
have moved). But I haven't looked at BBRv2's ECN behaviour in detail.

I'm sure someone on one of the lists that this is posted to will be able 
to answer tho.

HTH

Bob

>>     See 
>> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3
>> If you agree with the rationale, but think the new wording still 
>> doesn't fully capture the requirement, pls suggest sthg better.
>>        a TCP Reno congestion control [RFC5681  <https://tools.ietf.org/html/rfc5681>] (seeAppendix A.1.4  <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#appendix-A.1.4>  for
>>        rationale).
>>
>> 1/ The requirement as currently written says what an omniscient 
>> sender MUST do. So there's an implied requirement that a sender MUST 
>> be omniscient, which is of course impossible.
>> 2/ The requirement needs to be recast to require a sender to aim to 
>> be as knowledgeable as possible. Then, what it does as a result needs 
>> to take into account the a priori likelihood of there being a non-L4S 
>> bottleneck present.
>> 3/ This includes the possibility that the operator of the host knows 
>> that the network it serves has not deployed any single queue classic 
>> ECN AQM (e.g. in a CDN case they're doing out of band testing, or 
>> they've asked the ISP). So we've included the possibility of 
>> fall-back being disabled by configuration.
>> 4/ Nonetheless, as has been pointed out on the list, there is still a 
>> possibility that there is a Classic ECN AQM somewhere else on the 
>> path (to continue the CDN example, perhaps beyond the ISP in a home 
>> network). The 'MUST monitor' requirement still stands to ensure the 
>> operator doesn't miss these cases.
>> 5/ Then, if the server operators have disabled fall-back for their 
>> deployment, they can reconsider their policy or at least do more 
>> focused testing if they are frequently detecting a single-queue 
>> Classic ECN AQM.
>>
>> Items 3-5 are the "react via management" model that I've talked about 
>> on the list, given the unfairness doesn't amount to starvation, and 
>> it is possible that the prevalence of the problem is very low.
>>
>>
>> Finally, after the bullet list of requirements in section 4.3, (which 
>> are prerequisites for setting the ECT1 codepoint), we propose to add 
>> the following requirement, as suggested on the tsvwg list:
>>
>> Bob Briscoehttp://bobbriscoe.net/
>>
>> tcpPrague mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/tcpprague
>
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 1 Nov 2020 19:00:09 +0200

> On 1 Nov, 2020, at 3:07 am, Christian Huitema <[email-masked]> wrote:
> 

In my view, the biggest question you should be asking is how QUIC will distinguish between CE marks applied by an L4S AQM on the path, and those applied by an RFC-3168 AQM on the path.  The latter will treat ECT(1) marked packets the same as ECT(0), and expects the same multiplicative-decrease congestion response, but L4S expects a qualitatively different linear response.

You should probably not do any substantial work on integrating L4S into QUIC until you have a good answer to that question.  Alternative approaches to high-fidelity congestion control exist which resolve that particular conundrum easily.  In particular, the ECN field feedback mechanism in QUIC can also support SCE.

 - Jonathan Morton

---

From: Christian Huitema <[email-masked]>
Date: Sun, 1 Nov 2020 11:48:09 -0800

> In my view, the biggest question you should be asking is how QUIC will distinguish between CE marks applied by an L4S AQM on the path, and those applied by an RFC-3168 AQM on the path.  The latter…

Yes, that's a significant issue. As it is, nodes can only work with L4S
on specific paths that are somehow guaranteed to not mix the "classic"
and "L4S" behavior. For now, the only way I would implement it is by
adding an "l4s" option, that would only be turned on by default when
experimenting with L4S behavior. If we want any large scale deployment,
we have to get an IETF consensus on the evolution of ECN -- something
prescriptive, not just the "may experiment" of RFC 8311. Currently,
implementations can only treat CE marks as an indication that they are
probably sending faster than the network allows.

> You should probably not do any substantial work on integrating L4S into QUIC until you have a good answer to that question.  Alternative approaches to high-fidelity congestion control exist which r…

Yes, QUIC implementations could indeed support SCE. But they could not
support both SCE and L4S, and that's a big bummer.

I wonder whether there is a way to unify the L4S and SCE behaviors.
Transport protocol developers are unlikely to widely develop either
option unless the IETF comes to a consensus. This is a shame, because
ECN based congestion control does not have the ambiguity of loss-based
CC: ECN marks are not ambiguous, they are clearly signals from the
network, the application does not need to build logic to differentiate
between congestion induced losses and random losses due to, for example,
radio event. Failing that ECN unification CC algorithms end up relying a
lot on bandwidth and delay measurements, but that is sub-optimal because
some radio links do have variable bandwidth and variable delays.

By the way, I do think that L4S makes too many hypotheses about the CC.
I wish the AQM was based on intrinsic properties, such as "this flow is
using more than its fair share of bandwidth" or "this link is congested
and the transport should back off" rather than reasoning as if
implementations were still using TCP-RENO, which neither Linux or
Windows or Chrome does.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Sun, 1 Nov 2020 12:02:25 -0800

> since it was first written, but it might be a start.

Care doing a PR so this could be added to the Picoquic distribution?

-- Christian Huitema

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 1 Nov 2020 20:16:51 +0000

On just this one point:

> On 1 Nov 2020, at 19:48, Christian Huitema <[email-masked]> wrote:
> 
> Yes, QUIC implementations could indeed support SCE. But they could not support both SCE and L4S, and that's a big bummer.

There is currently no intention to progress SCE within TSVWG, so support for SCE within a PS for QUIC is really is a non-issue.

Gorry

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sun, 1 Nov 2020 12:23:46 -0800 (PST)

Bob,

> See inline tagged [BB]...

And I am changing it again... as only addressing a statement that
I have great concerns about.

See inline ragged [RWG]

> On 01/11/2020 01:07, Christian Huitema wrote:
... [ much text deleted ] ...

> have moved). But I haven't looked at BBRv2's ECN behaviour in detail.

 [RWG] I have great concern about this 500uS to 1mS ECN marking threshold
given that I have recently learned the latest WiFi standards actually
run with a packet aggregation time of 4mS thus making it probably
impossible to have such traffic work in such a targeted low latency queue.

 [RWG] Has any consideration been given to what is already deployed on
the Internet as link layer technologies that would preclude the
operation of the L4S low latency queue?

Regards,
-- 
Rod Grimes                                                 [email-masked]

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 1 Nov 2020 21:53:53 +0100

Dear Christian,

more below in-line, prefixed with [SM]

> Gesendet: Sonntag, 01. November 2020 um 21:02 Uhr
> Betreff: Re: [tsvwg] [tcpPrague] L4S and BBR (was: [iccrg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text)
>

       [SM] A cursory look at prague.c at the repository indicates, that this code might not fulfil the "eliminate RTT bias requirement" yet (I might have missed the implementation, hence "might not fulfil"). Odd since the repository is linked from the L4S site at rite, obviously just an oversight that will be fixed in due time?
Or a sign, that requiring individual protocols to fix up deficiencies in core AQMs is simply an unsustainable approach, if even a team L4S endorsed implementation is allowed to fall out of requirements.

Best Regards
        Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 1 Nov 2020 21:53:43 +0000

Hi Christian,

>>By the way, I do think that L4S makes too many hypotheses about the CC. I wish the AQM was based on intrinsic properties, such as "this flow is using more than its fair share of bandwidth"

This last quote is exactly what the L4S marking is intended to do: the smoothed marking probability over a longer time interval (about 10 RTTs) indicates the fair target rate. Assuming we agree on 15ms being the reference RTT, then the average rate can be r=2/(p*0.015)=133/p packets per second. So you can evaluate yourself if you use more or less than the fair share. How you respond to this knowledge is up to your CC.

>>or "this link is congested and the transport should back off"

L4S marks also give a very accurate queue size/delay indication (the queue size/delay that the network prefers). So keeping track of the RTT of marked packets, especially when they change from unmarked to marked and v/v is a good indicator of the target RTT, without the need to assume a universal delay target.

>> rather than reasoning as if implementations were still using TCP-RENO, which neither Linux or Windows or Chrome does.
We only assume TCP-friendliness, which is typically based on the Reno/Cubic reference. I agree, there shouldn’t be any dependency on the implementation of the congestion control itself.

I would assume that BBR can more tightly follow the available BW if it receives CE marks, and that the BW probing increase/frequency/duration in that case can be adapted to trigger the right marking probability matching the current BW. If no (or insufficient) marks are received, the normal BBR BW probing can be used. If too many marks are received, the BW needs to be reduced and normal BW probing can be suppressed/reduced…

Regards,
Koen.

From: Christian Huitema <[email-masked]>
Sent: Sunday, November 1, 2020 8:48 PM
To: Jonathan Morton <[email-masked]>
Cc: iccrg IRTF list <[email-masked]>; Bob Briscoe <[email-masked]>; tsvwg IETF list <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; TCP Prague List <[email-masked]>
Subject: Re: [tcpPrague] [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

On 1 Nov, 2020, at 3:07 am, Christian Huitema <[email-masked]><mailto:[email-masked]> wrote:

Yes, that's a significant issue. As it is, nodes can only work with L4S on specific paths that are somehow guaranteed to not mix the "classic" and "L4S" behavior. For now, the only way I would implement it is by adding an "l4s" option, that would only be turned on by default when experimenting with L4S behavior. If we want any large scale deployment, we have to get an IETF consensus on the evolution of ECN -- something prescriptive, not just the "may experiment" of RFC 8311. Currently, implementations can only treat CE marks as an indication that they are probably sending faster than the network allows.

I wonder whether there is a way to unify the L4S and SCE behaviors. Transport protocol developers are unlikely to widely develop either option unless the IETF comes to a consensus. This is a shame, because ECN based congestion control does not have the ambiguity of loss-based CC: ECN marks are not ambiguous, they are clearly signals from the network, the application does not need to build logic to differentiate between congestion induced losses and random losses due to, for example, radio event. Failing that ECN unification CC algorithms end up relying a lot on bandwidth and delay measurements, but that is sub-optimal because some radio links do have variable bandwidth and variable delays.

By the way, I do think that L4S makes too many hypotheses about the CC. I wish the AQM was based on intrinsic properties, such as "this flow is using more than its fair share of bandwidth" or "this link is congested and the transport should back off" rather than reasoning as if implementations were still using TCP-RENO, which neither Linux or Windows or Chrome does.

-- Christian Huitema

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 1 Nov 2020 23:41:59 +0000

Christian, [Changed subject line again]

> Care doing a PR so this could be added to the Picoquic distribution?

[BB] Would need to ask Quentin.
This was some time ago; I have no idea what he's doing these days.
I meant to add him. I have now (don't know whether the @ will still 
reach him).
I've also included Olivier, who might be in touch with him.

Bob

---

From: Christian Huitema <[email-masked]>
Date: Sun, 1 Nov 2020 15:55:50 -0800

On 11/1/2020 1:53 PM, De Schepper, Koen (Nokia - BE/Antwerp) wrote:
>
> >>By the way, I do think that L4S makes too many hypotheses about the
> CC. I wish the AQM was based on intrinsic properties, such as "this
> flow is using more than its fair share of bandwidth"
>
> This last quote is exactly what the L4S marking is intended to do: the
> smoothed marking probability over a longer time interval (about 10
> RTTs) indicates the fair target rate. Assuming we agree on 15ms being
> the reference RTT, then the average rate can be r=2/(p*0.015)=133/p
> packets per second. So you can evaluate yourself if you use more or
> less than the fair share. How you respond to this knowledge is up to
> your CC.
>
I think that part of the complexity in L4S AQM comes from trying to
achieve fair sharing of resource without actually implementing fair
queuing. You end up expecting that implementations apply a formula and
derive their packet sending rate from a smoothed average of the packet
marking rate. It may work well in controlled environments, but I am
afraid that very few implementations are ever going to do that in practice.

Congestion Control algorithms are much more likely to treat packet
losses, markings, or delay increases as signals that they are sending
too fast, while considering their absence or lower frequency as signal
that they might be able to send faster. Implementations also are likely
to take into account the recent history, for example monitoring the min
RTT (LEDBAT, BBR), or monitoring the recent available bandwidth (BBR).
They also typically filter signals, e.g., consider only one packet loss
or ECN mark per RTT (New Reno, Cubic, etc.). This is expected, because
we do see packet losses arriving in batches, and can reasonably expect
ECN marks to have the same behavior. But implementations are inherently
selfish, do not really trust the signals from the network, and are very
unlikely to end up applying the formula that you suggest.

In any case, you have a scaling issue. Let's consider a 1.5Gbps link,
with 15 ms delay and 1500 bytes packets. The nominal sending rate is
125,000 packets per second. The marking rate with your formula shall be
p = 2/(r*0.015), which is about 0.0008%. Over the last 10 RTT, the
connection will on average see 0.14 marks -- that is, no mark over the
last 10 RTT 86% of the time. This falls well short of the requirement to
provide frequent feedback!

If you give up the formula, you could end up with a significantly
simpler event based congestion control, in which applications slow down
if they see too many CE marks, accelerate if they only see a very low
marking rate, and stay put if they see an infrequent marking rate. I can
see applications doing that, resulting in a fairly stable network and
low delays. Of course, you will have to do some form of fair share
enforcement to catch violators, but I am sure you can come up with
something.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Sun, 1 Nov 2020 16:10:49 -0800

On 11/1/2020 3:55 PM, Christian Huitema wrote:
>> This last quote is exactly what the L4S marking is intended to do:
>> the smoothed marking probability over a longer time interval (about
>> 10 RTTs) indicates the fair target rate. Assuming we agree on 15ms
>> being the reference RTT, then the average rate can be
>> r=2/(p*0.015)=133/p packets per second. So you can evaluate yourself
>> if you use more or less than the fair share. How you respond to this
>> knowledge is up to your CC.
>>
> that they might be able to send faster. Implementations also are
> likely to take into account the recent history, for example monitoring
> the min RTT (LEDBAT, BBR), or monitoring the recent available
> bandwidth (BBR). They also typically filter signals, e.g., consider
> only one packet loss or ECN mark per RTT (New Reno, Cubic, etc.). This
> is expected, because we do see packet losses arriving in batches, and
> can reasonably expect ECN marks to have the same behavior. But
> implementations are inherently selfish, do not really trust the
> signals from the network, and are very unlikely to end up applying the
> formula that you suggest.
>
> 125,000 packets per second. The marking rate with your formula shall
> be p = 2/(r*0.015), which is about 0.0008%. Over the last 10 RTT, the
> last 10 RTT 86% of the time. This falls well short of the requirement
> to provide frequent feedback!
>
Sorry, bug in my spreadsheet. The marking rate is actually 0.1%, about 2
packets per RTT. Still not frequent enough for my taste, but much better
than 0.0008%. Nevertheless, the inverse relation between marking rate
and data rate is not great.

> simpler event based congestion control, in which applications slow
> down if they see too many CE marks, accelerate if they only see a very
> low marking rate, and stay put if they see an infrequent marking rate.
> I can see applications doing that, resulting in a fairly stable
> network and low delays. Of course, you will have to do some form of
> fair share enforcement to catch violators, but I am sure you can come
> up with something.
>
> -- Christian Huitema
>
>
>
>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 2 Nov 2020 02:52:06 +0200

> On 2 Nov, 2020, at 2:10 am, Christian Huitema <[email-masked]> wrote:
> 
>> In any case, you have a scaling issue. Let's consider a 1.5Gbps link, with 15 ms delay and 1500 bytes packets. The nominal sending rate is 125,000 packets per second. The marking rate with your formula shall be p = 2/(r*0.015), which is about 0.0008%. Over the last 10 RTT, the connection will on average see 0.14 marks -- that is, no mark over the last 10 RTT 86% of the time. This falls well short of the requirement to provide frequent feedback!
> 
> Sorry, bug in my spreadsheet. The marking rate is actually 0.1%, about 2 packets per RTT. Still not frequent enough for my taste, but much better than 0.0008%. Nevertheless, the inverse relation between marking rate and data rate is not great.

A property of both Reno and DCTCP is that a particular CE marking probability results in a particular average cwnd, over some reasonably long period.  The relationship formula is of course very different for each one, but that qualitative rule happens to be the same.  It is a property that DualQ's coupled AQM design relies on.  CUBIC is a little different in its high-BDP regime, but acts like Reno otherwise.

What this means is that competing flows experiencing the same marking probability will converge to the same cwnd, and their relative throughputs will be inversely proportional to their effective RTTs.  This convergence is not necessarily very fast, but given a single queue and a single AQM, it's about as good as you can expect.

When you have an AQM per flow, you can do a bit better by applying a different marking probability to each flow.  This makes convergence faster, and you can make them converge to the same throughput, not merely the same cwnd.  When you have a separate queue per flow, you can additionally prevent one "fat" flow from affecting the latency of sparser flows, and *enforce* the throughput equality on short timescales, even with flows that are unresponsive to congestion signals.  This is all established practice.

And that is the key to achieving RTT independence with the minimum of added complexity.

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Mon, 2 Nov 2020 05:58:00 +0000

I think that part of the complexity in L4S AQM comes from trying to achieve fair sharing of resource without actually implementing fair queuing. You end up expecting that implementations apply a formula and derive their packet sending rate from a smoothed average of the packet marking rate. It may work well in controlled environments, but I am afraid that very few implementations are ever going to do that in practice.

Congestion Control algorithms are much more likely to treat packet losses, markings, or delay increases as signals that they are sending too fast, while considering their absence or lower frequency as signal that they might be able to send faster. Implementations also are likely to take into account the recent history, for example monitoring the min RTT (LEDBAT, BBR), or monitoring the recent available bandwidth (BBR). They also typically filter signals, e.g., consider only one packet loss or ECN mark per RTT (New Reno, Cubic, etc.). This is expected, because we do see packet losses arriving in batches, and can reasonably expect ECN marks to have the same behavior. But implementations are inherently selfish, do not really trust the signals from the network, and are very unlikely to end up applying the formula that you suggest.

[GW] But, this always works out to 2 marks per RTT or 2 marks per 15ms (whichever is slower), so in that sense it doesn’t depend on data rate at all.  The sender slows down if it sees more frequent marks than this, accelerates if it sees fewer marks than this, and stays put if it sees marking at this rate.

If you give up the formula, you could end up with a significantly simpler event based congestion control, in which applications slow down if they see too many CE marks, accelerate if they only see a very low marking rate, and stay put if they see an infrequent marking rate. I can see applications doing that, resulting in a fairly stable network and low delays. Of course, you will have to do some form of fair share enforcement to catch violators, but I am sure you can come up with something.

-- Christian Huitema

tcpPrague mailing list

[email-masked]<mailto:[email-masked]>

---

From: Christian Huitema <[email-masked]>
Date: Sun, 1 Nov 2020 22:50:39 -0800

On 11/1/2020 9:58 PM, Greg White wrote:
>         >>By the way, I do think that L4S makes too many hypotheses
>         about the CC. I wish the AQM was based on intrinsic
>         properties, such as "this flow is using more than its fair
>         share of bandwidth"
>
>         This last quote is exactly what the L4S marking is intended to
>         do: the smoothed marking probability over a longer time
>         interval (about 10 RTTs) indicates the fair target rate.
>         Assuming we agree on 15ms being the reference RTT, then the
>         average rate can be r=2/(p*0.015)=133/p packets per second. So
>         you can evaluate yourself if you use more or less than the
>         fair share. How you respond to this knowledge is up to your CC.
>
>     I think that part of the complexity in L4S AQM comes from trying
>     to achieve fair sharing of resource without actually implementing
>     fair queuing. You end up expecting that implementations apply a
>     formula and derive their packet sending rate from a smoothed
>     average of the packet marking rate. It may work well in controlled
>     environments, but I am afraid that very few implementations are
>     ever going to do that in practice.
>
>     losses, markings, or delay increases as signals that they are
>     sending too fast, while considering their absence or lower
>     frequency as signal that they might be able to send faster.
>     Implementations also are likely to take into account the recent
>     history, for example monitoring the min RTT (LEDBAT, BBR), or
>     monitoring the recent available bandwidth (BBR). They also
>     typically filter signals, e.g., consider only one packet loss or
>     ECN mark per RTT (New Reno, Cubic, etc.). This is expected,
>     because we do see packet losses arriving in batches, and can
>     reasonably expect ECN marks to have the same behavior. But
>     signals from the network, and are very unlikely to end up applying
>     the formula that you suggest.
>
>     In any case, you have a scaling issue. Let's consider a 1.5Gbps
>     link, with 15 ms delay and 1500 bytes packets. The nominal sending
>     rate is 125,000 packets per second. The marking rate with your
>     formula shall be p = 2/(r*0.015), which is about 0.0008%. Over the
>     last 10 RTT, the connection will on average see 0.14 marks -- that
>     is, no mark over the last 10 RTT 86% of the time. This falls well
>     short of the requirement to provide frequent feedback!
>
> Sorry, bug in my spreadsheet. The marking rate is actually 0.1%, about
> 2 packets per RTT. Still not frequent enough for my taste, but much
> better than 0.0008%. Nevertheless, the inverse relation between
> marking rate and data rate is not great.
>
> [GW] But, this always works out to 2 marks per RTT or 2 marks per 15ms
> (whichever is slower), so in that sense it doesn’t depend on data rate
> at all.  The sender slows down if it sees more frequent marks than
> this, accelerates if it sees fewer marks than this, and stays put if
> it sees marking at this rate.
>
Yes, there is that. But even if the rate is large enough, the CC
implementations can only program such a constant if they have assurance
that every bottleneck that they encounter will use the same time
constant. I don't think we are ready to make such assumptions outside of
controlled experiments. I would expect a mix of L4S-AQM, CODEL and PIE
variants, classic ECN, etc. The host implementation has to be ready to
handle all that, and trying to use a time constant there is very
challenging.

>     down if they see too many CE marks, accelerate if they only see a
>     very low marking rate, and stay put if they see an infrequent
>     marking rate. I can see applications doing that, resulting in a
>     fairly stable network and low delays. Of course, you will have to
>     do some form of fair share enforcement to catch violators, but I
>     am sure you can come up with something.
>
I still believe that this kind of "natural signaling" is easier to
handle than the formulaic behavior expected with L4S. It can easily
deliver the "low queing delay" goal of L4S. It cannot guarantee "fair
sharing of bandwidth", but my thesis is that such fair sharing requires
per-flow enforcement.

-- Christian Huitema

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 2 Nov 2020 08:30:15 +0000

Hi Rodney

I believe part of the answer to your question is found in
https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-07#section-6.3 . The
fact that we begin to see the scheduling jitter does not invalidate L4S as a
means to reduce the congestion related jitter.

Scheduling jitter exist also in cellular technology. Scheduling jitter is
however generally not evidence of congestion. L4S congestion marking should
therefore be designed to not trigger on this. The master thesis** the L4S
marking threshold was set sufficiently high to avoid this but it should be
understood that L4S marking in cellular can be implemented in smarter ways.
In the master thesis, the L4S marker only inspected the queue on the RLC
layer.
There are other causes to delay jitter in cellular, HARQ retransmissions and
handover delays are two examples. There is work in progress to reduce all
kinds on delay jitter, L4S is a tool to reduce congestion related jitter.
In any case, non-congestion related jitter is not an argument against L4S

/Ingemar

**
https://kth.diva-portal.org/smash/record.jsf?dswid=-6303&pid=diva2%3A1484466
&c=1&searchType=SIMPLE&language=en&query=brunello&af=%5B%5D&aq=%5B%5B%5D%5D&
aq2=%5B%5B%5D%5D&aqe=%5B%5D&noOfRows=50&sortOrder=author_sort_asc&sortOrder2
=title_sort_asc&onlyFullText=false&sf=all 

> -----Original Message-----
> From: Rodney W. Grimes <[email-masked]>
> Sent: den 1 november 2020 21:24
> Cc: Christian Huitema <[email-masked]>; tsvwg IETF list
> <[email-masked]>; iccrg IRTF list <[email-masked]>; TCP Prague List
> <[email-masked]>; De Schepper, Koen (Koen)
> <[email-masked]>
> Subject: [iccrg] Realistic Queue delay targets
> 
> Bob,
> 
> > Christian,
> >
> And I am changing it again... as only addressing a statement that I have
great
> concerns about.
> > Coupled AQM will classify BBRv2 packets into the L4S queue. This
> > should have a very shallow ECN marking threshold (500us - 1ms), so
> > even if the flow (whether QUIC or TCP) is flying just under the
> > available capacity, bunching of packets means it is unlikely to
> > completely avoid ECN marking between probes. If it could avoid ECN
> > marking, you are right that it would get a bump of ECN marks during
> > the probe. I haven't studied the code, but when it experiences ECN
> > marking I believe it switches into an L4S ECN mode for a while, and
> > uses ECN rather than delay probing to track available capacity. I
> > assume it switches back to BBR's delay probing mode if it gets no ECN
> > for a while (e.g. the bottleneck might have moved). But I haven't looked
at
> BBRv2's ECN behaviour in detail.
> 
> given that I have recently learned the latest WiFi standards actually run
with a
> packet aggregation time of 4mS thus making it probably impossible to have
> such traffic work in such a targeted low latency queue.
> 
>  [RWG] Has any consideration been given to what is already deployed on the
> Internet as link layer technologies that would preclude the operation of
the
> L4S low latency queue?

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 2 Nov 2020 09:46:36 +0100

Hi Ingemar,

more below in-line, prefixed [SM].

> Gesendet: Montag, 02. November 2020 um 09:30 Uhr
> Von: "Ingemar Johansson S" <ingemar.s.johansson=[email-masked]>
> An: "Rodney W. Grimes" <[email-masked]>, "Bob Briscoe" <[email-masked]>
> Cc: "tsvwg IETF list" <[email-masked]>, "iccrg IRTF list" <[email-masked]>, "Christian Huitema" <[email-masked]>, "TCP Prague List" <[email-masked]>, "De Schepper, Koen (Koen)" <[email-masked]>
> Betreff: Re: [tsvwg] [iccrg] Realistic Queue delay targets
>
> Hi Rodney
>

       [SM] As so often that section does contain some theoretical observations, but fails to empirically show which level of burstyness L4S can deal with gracefully and what the consequences are (on the bursty flow itself, other flows in the shallow-queue and and flows in the deep queue). That would be meaningful responses to Rodney's concern. As is that section basically just claims without supporting data, that even in the face of bursty flows, L4S should still be tried, but that must be compared to a FiFO, because compared to say fq_codel with a 5ms default target, traffic with > 5ms jitter will probably look much better after fq_codel than after L4S (L4S will conserve the burstyness much more, while fq_codel will smooth it out a bit).
        I wonder why such verbose but essentially non-actionable (almost useless) sections keep getting added to already  quite long and verbose internet drafts in the first place?

Regards
        Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 2 Nov 2020 09:50:55 +0000

Hi
I don't anymore know where all these discussions are heading. Seems like we 
always end up in infinite discussions that lead nowhere!.
Network implementers will implement whatever queue management they find 
useful, be it dualQ, fq-codel or whatever.

Yes, there are ways to deal with jitter, some are implemented, some are in the 
pipe, some are on the drawing board and as I mentioned several times: It is 
not very fruitful to take todays internet jitter as a litmus test on how good 
L4S will behave in the future.
As examples, 5G has a shorter HARQ retransmission loop than 4G, this reduces 
jitter, there is also work to reduce handover delay, shorter TTIs etc. It is 
therefore quite meaningless to produce data on how well L4S will fare in the 
precense of jitter. Leave that to the experiments.

/Ingemar

> Sent: den 2 november 2020 09:47
> To: Ingemar Johansson S <[email-masked]>
> Cc: Rodney W. Grimes <[email-masked]>; Bob Briscoe
> <[email-masked]>; tsvwg IETF list <[email-masked]>; iccrg IRTF list
> <[email-masked]>; Christian Huitema <[email-masked]>; TCP Prague List
> Subject: Aw: Re: [tsvwg] [iccrg] Realistic Queue delay targets
>
> Hi Ingemar,
>
> > Von: "Ingemar Johansson S"
> > <ingemar.s.johansson=[email-masked]>
> > Cc: "tsvwg IETF list" <[email-masked]>, "iccrg IRTF list"
> > <[email-masked]>, "Christian Huitema" <[email-masked]>, "TCP
> > Prague List" <[email-masked]>, "De Schepper, Koen (Koen)"
> > https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-07#section-6.3 .
> > The fact that we begin to see the scheduling jitter does not
> > invalidate L4S as a means to reduce the congestion related jitter.
>
>        [SM] As so often that section does contain some theoretical 
> observations,
> but fails to empirically show which level of burstyness L4S can deal with
> gracefully and what the consequences are (on the bursty flow itself, other
> flows in the shallow-queue and and flows in the deep queue). That would be
> meaningful responses to Rodney's concern. As is that section basically just
> claims without supporting data, that even in the face of bursty flows, L4S
> should still be tried, but that must be compared to a FiFO, because compared
> to say fq_codel with a 5ms default target, traffic with > 5ms jitter will 
> probably
> look much better after fq_codel than after L4S (L4S will conserve the
> burstyness much more, while fq_codel will smooth it out a bit).
>         I wonder why such verbose but essentially non-actionable (almost
> useless) sections keep getting added to already  quite long and verbose
> internet drafts in the first place?
>
> Regards
>         Sebastian
>
> >
> > Scheduling jitter exist also in cellular technology. Scheduling jitter
> > is however generally not evidence of congestion. L4S congestion
> > marking should therefore be designed to not trigger on this. The
> > master thesis** the L4S marking threshold was set sufficiently high to
> > avoid this but it should be understood that L4S marking in cellular can be
> implemented in smarter ways.
> > In the master thesis, the L4S marker only inspected the queue on the
> > RLC layer.
> > There are other causes to delay jitter in cellular, HARQ
> > retransmissions and handover delays are two examples. There is work in
> > progress to reduce all kinds on delay jitter, L4S is a tool to reduce
> congestion related jitter.
> &c=1&searchType=SIMPLE&language=en&query=brunello&af=%5B%5D&aq=
> %5B%5B%
> > 5D%5D&
> >
> aq2=%5B%5B%5D%5D&aqe=%5B%5D&noOfRows=50&sortOrder=author_sor
> t_asc&sort
> > Order2 =title_sort_asc&onlyFullText=false&sf=all
> >
> > > > to send packets over the Internet as ECT(1). Then, yes, an L4S
> > > > DualQ Coupled AQM will classify BBRv2 packets into the L4S queue.
> > > > This should have a very shallow ECN marking threshold (500us -
> > > > 1ms), so even if the flow (whether QUIC or TCP) is flying just
> > > > under the available capacity, bunching of packets means it is
> > > > unlikely to completely avoid ECN marking between probes. If it
> > > > could avoid ECN marking, you are right that it would get a bump of
> > > > ECN marks during the probe. I haven't studied the code, but when
> > > > it experiences ECN marking I believe it switches into an L4S ECN
> > > > mode for a while, and uses ECN rather than delay probing to track
> > > > available capacity. I assume it switches back to BBR's delay
> > > > probing mode if it gets no ECN for a while (e.g. the bottleneck
> > > > might have moved). But I haven't looked
> > at
> > >  [RWG] I have great concern about this 500uS to 1mS ECN marking
> > > threshold given that I have recently learned the latest WiFi
> > > standards actually run
> > with a
> > > packet aggregation time of 4mS thus making it probably impossible to
> > > have such traffic work in such a targeted low latency queue.
> > >
> > >  [RWG] Has any consideration been given to what is already deployed
> > > on the Internet as link layer technologies that would preclude the
> > > operation of
> > the
> > > L4S low latency queue?
> > >
> > > Regards,
> > > --
> > > Rod Grimes
> > [email-masked]
> > >
> >
> >

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 2 Nov 2020 11:27:02 +0100

Hi Ingemor,

more below in-line, prefixed [SM2].

> Gesendet: Montag, 02. November 2020 um 10:50 Uhr
> Von: "Ingemar Johansson S" <[email-masked]>
> An: "Sebastian Moeller" <[email-masked]>
> Cc: "Rodney W. Grimes" <[email-masked]>, "Bob Briscoe" <[email-masked]>, "tsvwg IETF list" <[email-masked]>, "iccrg IRTF list" <[email-masked]>, "Christian Huitema" <[email-masked]>, "TCP Prague List" <[email-masked]>, "De Schepper, Koen (Koen)" <[email-masked]>, "Ingemar Johansson S" <[email-masked]>
> Betreff: RE: Re: [tsvwg] [iccrg] Realistic Queue delay targets
>
> Hi
> I don't anymore know where all these discussions are heading. 

        [SM2] Ideally this process would lead to better designed and implemented L4S core methods and better written internet drafts. All that requires team L4S to be a bit more open, to a) support their assertions with real data and b) at least occasionally accept criticism as valid and change more than a few words in the draft as a consequence. As long as team L4S basically maintains L4S is perfect as it is, this is heading nowhere (at least it should, in reality we are heading straight towards ratifying the existing IMHO sub-optimal state of the L4S internet drafts as experimental track RFCs, if only based on the argument that after working on this for a the better part of a decade it is now time to show up something). This view is supported by the fact that the chairs stopped soliciting actual data and safety tests, and focus mainly on seeing proposals for text changes. Even though it is still unclear if L4S is actually suitable for wider release into the internet. The current drafts however are not written in a way to conduct an "is L4S safe to deploy into the internet" experiment, as they offer neither criteria on how to asses safety, nor a plan for how to resolve the experiment for either outcome. (On success it is simple, either do nothing and keep experimental infrastructure in use, or try to move the RFC to standards track, but on failure, how to disable all the network nodes and how to reclaim the ECT(1) codepoint for other experiments?)

Seems like we 
> always end up in infinite discussions that lead nowhere!.

        [SM2] Oh, no. we are making progress albeit at a worrisome slow rate. For example, your points confirmed that relaying on protocol designers to meet requirements that only have subtle safety consequences (like the 15ms hack) seems overly optimistic and not a viable path to guaranteeing safety. Yet, I am certain, that this assessment will a) not be shared universally on this list, nor lead to changes in how team L4S plans to deal with issue #28 (the strategy so far is to more or less ignore that issue).

> useful, be it dualQ, fq-codel or whatever.

        [SM2] Yes, and that as a consequence means, that the core L4S methods and components need to offer robust safety even when protocols might be (accidentally) cutting corners, no? Because if even you, a strong proponent of L4S, did not understand that section 4.3 was not intended to be optional, how can we expect other designers less involved in L4S get this right?

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 2 Nov 2020 12:33:01 +0000

Hi Christian,

Indeed a very important discussion we need to have, and agree on.

>> If you give up the formula, you could end up with a significantly simpler event based congestion control, in which applications slow down if they see too many CE marks, accelerate if they only see a very low marking rate, and stay put if they see an infrequent marking rate.

This is exactly what I expect most CCs will do: work event based. The formula is only to tune the events so it converges to the formula (eventually). If there is a lot of dynamism, obviously shorter RTTs can respond faster and will benefit from that. They will have memory over a shorter time interval, and on average more throughput than a longer RTT flow. But if things are stable, it is good to converge to a known rate. I don’t see any better solution, do you? If we don’t foresee such a mechanism how do you expect bandwidth to be shared over shared infrastructure? Then the whole discussion of fairness can be dismissed, and CCs should not be evaluated for this neither.

>> The marking rate is actually 0.1%, about 2 packets per RTT

Indeed, this is still 2 marks per 15ms, much better than 1 drop/mark per 1000 RTTs (15 seconds) for your example when using Reno (Cubic will also be seconds). Only if the RTT is much smaller than 15ms, it will get less marks per RTT, but in our Linux Prague we have an option to gradually take the RTT dependence into account, depending on how long the flow is running application unlimited (or by extension how long things are stable).

Also such a formula is just a reference to aim at. It should not be a continuous hard requirement. There are reasons when a “flow” can deviate from it (Less than best effort can take less, a tunnel of multiple flows will take as many times more as there are flows inside, and a real-time application might temporary take more when needed).

I’m in favor of specifying explicitly such a formula as a target for L4S CC’s to share BW in steady state situations (long flows and stable throughput). Up to now I don’t think we have such an explicit requirement in the drafts. Anyone else that would support this? At least CC control developers then have a target to aim for and a clear evaluation criterium.

Regards,
Koen.

From: Christian Huitema <[email-masked]>
Sent: Monday, November 2, 2020 1:11 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; Jonathan Morton <[email-masked]>
Cc: iccrg IRTF list <[email-masked]>; Bob Briscoe <[email-masked]>; tsvwg IETF list <[email-masked]>; TCP Prague List <[email-masked]>
Subject: Re: [tcpPrague] [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

-- Christian Huitema

tcpPrague mailing list

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 2 Nov 2020 12:42:40 +0000

Hi Jonathan,

Indeed, FQ is ensuring (actually enforcing) per 5 tuple equal BW. Do you suggest we need to have FQ per 5 tuple in every bottleneck? This would indeed be a simple solution for CC developers. But not sure if every NW box would like to implement this functionality. From the moment flows are treated as an aggregate, the rates that CC's will get is exactly based on their respective behavior (say their formula), and if they are vastly different, so will their rates be...

As long as there is no consensus about universal FQ, I think we need to discuss what the formula should be.

Regards,
Koen.

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Monday, November 2, 2020 1:52 AM
To: Christian Huitema <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; iccrg IRTF list <[email-masked]>; Bob Briscoe <[email-masked]>; tsvwg IETF list <[email-masked]>; TCP Prague List <[email-masked]>
Subject: Re: [tcpPrague] [tsvwg] ecn-l4s-id: Proposed Changed to Normative Classic ECN detection Text

 - Jonathan Morton