Subject: Re: [tsvwg] Prague requirements survey

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Fri, 16 Apr 2021 12:52:55 +0000

Hi all,

An update on the survey is available. We received an additional input from Apple which we could publicly share (thanks Vidhi for providing this input). I also updated the consolidated view v2 (available on https://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance).

I believe it is strongly in line with the previous survey conclusions as presented in last tsvwg. One main additional feedback was on "7. Measuring Reordering Tolerance in Time Units". There was disagreement that using time only and not packet count is a foolproof solution. As far as I understand the objection is to the current wording that a time based mechanism is the only/sufficient way to assure this.

The objective of this requirement is to allow a certain level of reordering for L4S traffic (actually avoid delaying packets in the network to guarantee correct order of packet delivery). I personally could support wording that expresses the core of the requirement, and not limit the text to one mechanism, which would allow alternative/more robust implementations. The requirement could be expressed as something like: "a scalable congestion control SHOULD  be resilient to reordering over an (adaptive) (time?) interval, which scales with / adapts to throughput, as opposed to counting only in (fixed) units of packets (as in the 3 DupACK rule of RFC 5681 TCP), which is not scalable". Let's further discuss here on the list what could be for all parties an acceptable wording.

Thanks,
Koen.

From: De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Sunday, March 7, 2021 1:57 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg IETF list <[email-masked]>
Cc: Bob Briscoe <[email-masked]>
Subject: RE: Prague requirements survey

Hi all,

The details of the consolidated view of all feedback received is available and can be found via following link: https://l4steam.github.io/PragueReqs/Prague_requirements_consolidated.pdf

The only strong objections were against the "MUST document" requirements, which will be removed from the next version of the draft. Some clarifications were asked and (will be) added.
For 2 requirements a big consensus was that they should be developed and evolved as needed during the experiment.
All other requirements had already implementations and if not, were seen feasible/realizable and were planned to be implemented.

We will present an overview during the meeting.

Regards,
Koen.

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Wednesday, March 3, 2021 2:20 PM
To: tsvwg IETF list <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] Prague requirements survey

Hi all,

We have received several surveys privately, for which I tried to get the approval for sharing those on the overview page: l4steam.github.io | L4S-related experiments and companion website<https://l4steam.github.io/#prague-requirements-compliance>

Thanks to NVIDIA for sharing their view and feedback for their GeforceNow congestion control. Their feedback was added to the above overview about a week ago. As we didn't get the explicit approval for the others, we will share and present a consolidated view of all feedback received later and during the meeting.

Note: pdf versions are now also available on the above page for easier reading.

Koen.

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: Monday, February 8, 2021 2:37 PM
To: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>; tsvwg IETF list <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tsvwg] Prague requirements survey

Hi Ingemar,

Thanks for your contributions. I linked your doc to the https://l4steam.github.io/#prague-requirements-compliance web page (and will do so for others).

I didn't see any issues or objections mentioned to the current requirements as specified in the draft. Does this mean you think they are all reasonable, valid and feasible?

Interesting observation (related to the performance optimization topic 1) that for the control packets "RTCP is likely not using ECT(1)". Why is this not likely? I assume this will impact the performance? Do we need to recommend the use of ECT(1) on RTCP packets in the draft?

Thanks,
Koen.

From: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>
Sent: Monday, February 8, 2021 10:59 AM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]<mailto:[email-masked]>>; tsvwg IETF list <[email-masked]<mailto:[email-masked]>>
Cc: Ingemar Johansson S <[email-masked]<mailto:[email-masked]>>
Subject: RE: Prague requirements survey

Hi
Please find attached (hopefully) a Prague requirements survey applied to SCReAM (RFC8298 std + running code)

Regards
Ingemar

From: tsvwg <[email-masked]<mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
Sent: den 6 februari 2021 23:20
To: tsvwg IETF list <[email-masked]<mailto:[email-masked]>>
Subject: [tsvwg] Prague requirements survey

Hi all,

To get a better understanding on the level of consensus on the Prague requirements, we prepared an overview document listing the L4S-ID draft requirements specific to the CC (wider Prague requirements), as a questionnaire towards potential CC developers. If you are developing or have developed an L4S congestion control, you can describe the status of your ongoing development in the second last column. If you cannot share status, or plan-to/would implement an L4S CC, you can list what you would want to support (see feasible). In the last column you can put any description/limitations/remarks/explanations related to evaluations, implementations and/or plans (will implement or will not implement). Any expected or experienced issues and any objections/disagreements to the requirement can be explained and colored appropriately.

The document can be found on following link: https://raw.githubusercontent.com/L4STeam/l4steam.github.io/master/PragueReqs/Prague_requirements_Compliance_and_Objections_template.docx<https://protect2.fireeye.com/v1/url?k=d16bc960-8ef0f066-d16b89fb-86ee86bd5107-080c65bfd839440d&q=1&e=7dbb7494-67c3-4315-88a6-325f32e4e8b1&u=https%3A%2F%2Fraw.githubusercontent.com%2FL4STeam%2Fl4steam.github.io%2Fmaster%2FPragueReqs%2FPrague_requirements_Compliance_and_Objections_template.docx>

As an example I filled it for the Linux TCP-Prague implementation on following link: https://l4steam.github.io/PragueReqs/Prague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx<https://protect2.fireeye.com/v1/url?k=f839c5f7-a7a2fcf1-f839856c-86ee86bd5107-29dabadc5d0e673d&q=1&e=7dbb7494-67c3-4315-88a6-325f32e4e8b1&u=https%3A%2F%2Fl4steam.github.io%2FPragueReqs%2FPrague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx>

Please send your filled document to the list (Not sure if an attachment will work, so I assume you also need to store it somewhere and send a link to it, or send to me directly).

We hope to collect many answers, understanding the position of the different (potential) implementers and come faster to consensus.

Thanks,
Koen.

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 16 Apr 2021 10:08:05 -0400

On Fri, Apr 16, 2021 at 8:53 AM De Schepper, Koen (Nokia - BE/Antwerp) <
[email-masked]> wrote:

> Hi all,
>
>
>
> An update on the survey is available. We received an additional input from
> Apple which we could publicly share (thanks Vidhi for providing this
> input). I also updated the consolidated view v2 (available on
> https://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance
> ).
>
>
>
> I believe it is strongly in line with the previous survey conclusions as
> presented in last tsvwg. One main additional feedback was on “7. Measuring
> Reordering Tolerance in Time Units”. There was disagreement that using time
> only and not packet count is a foolproof solution. As far as I understand
> the objection is to the current wording that a time based mechanism is the
> only/sufficient way to assure this.
>
>
>
> The objective of this requirement is to allow a certain level of
> reordering for L4S traffic (actually avoid delaying packets in the network
> to guarantee correct order of packet delivery). I personally could support
> wording that expresses the core of the requirement, and not limit the text
> to one mechanism, which would allow alternative/more robust
> implementations. The requirement could be expressed as something like: “a
> scalable congestion control SHOULD  be resilient to reordering over an
> (adaptive) (time?) interval, which scales with / adapts to throughput, as
> opposed to counting only in (fixed) units of packets (as in the 3 DupACK
> rule of RFC 5681 TCP), which is not scalable”. Let’s further discuss here
> on the list what could be for all parties an acceptable wording.
>

Thanks for posting this, Koen and Vidhi.

Regarding Apple's Prague response that you posted:

https://l4steam.github.io/PragueReqs/Apple_L4S_requirements_Compliance_and_Objections.pdf

And specifically regarding the response about a reordering tolerance in
time units:

     Disagree that using time only and not packet count is a
     foolproof solution. What if the time threshold value cause
     slow recovery in case of an actual packet loss event? Would
     it be better to use either packet count or time threshold? We
     currently don't support RACK - so time based loss detection
     wouldn't be possible.

I would agree that detecting losses purely based on a time
threshold value can cause slower recovery in case of an actual
packet loss event. And this can be needless delay if the path
actually has no reordering.

But it may be worth noting that in RACK-TLP loss detection [
https://tools.ietf.org/html/rfc8985 ] if no reordering has been
observed on the connection then the loss detection mechanism will
trigger based on packet count (DupThresh=3 segments SACKed).
RACK-TLP only uses time-based triggering of recovery if
reordering has been observed on the connection. This strikes a
balance between quick loss recovery for paths with no reordering,
and reordering tolerance for paths with reordering. So in effect
RACK-TLP does use the approach urged above ("Would it be better
to use either packet count or time threshold?")

I agree it is worth clarifying in the Prague requirements whether
it is required that loss detection *always* be based on time
units, or whether it is acceptable to use an adaptive approach
like the RACK-TLP approach outlined above (as specified in
RFC8985 and implemented in Linux TCP).

IMHO Koen's suggested alternate text for this requirement is
quite good. Here is a spin on Koen's text that sounds good to me
and explicitly mentions that RACK/RFC8985 qualifies as meeting
this requirement:

     a scalable congestion control SHOULD be resilient to
     reordering over an adaptive time interval that scales with
     throughput and adapts to reordering (as in [RFC8985]), as
     opposed to counting only in fixed units of packets (as in
     the 3 DupACK rule of [RFC5681] and [RFC6675], which is not
     scalable)

best,
neal

> Thanks,
>
> Koen.
>
>
>
>
>
> *From:* De Schepper, Koen (Nokia - BE/Antwerp)
> *Sent:* Sunday, March 7, 2021 1:57 AM
> *To:* De Schepper, Koen (Nokia - BE/Antwerp) <
> [email-masked]>; tsvwg IETF list <[email-masked]>
> *Cc:* Bob Briscoe <[email-masked]>
> *Subject:* RE: Prague requirements survey
>
>
>
> Hi all,
>
>
>
> The details of the consolidated view of all feedback received is available
> and can be found via following link:
> https://l4steam.github.io/PragueReqs/Prague_requirements_consolidated.pdf
>
>
>
> The only strong objections were against the “MUST document” requirements,
> which will be removed from the next version of the draft. Some
> clarifications were asked and (will be) added.
>
> All other requirements had already implementations and if not, were seen
> feasible/realizable and were planned to be implemented.
>
>
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *De Schepper, Koen
> (Nokia - BE/Antwerp)
> *Sent:* Wednesday, March 3, 2021 2:20 PM
> *To:* tsvwg IETF list <[email-masked]>
> *Subject:* Re: [tsvwg] Prague requirements survey
>
>
>
> Hi all,
>
>
>
> We have received several surveys privately, for which I tried to get the
> approval for sharing those on the overview page: l4steam.github.io |
> L4S-related experiments and companion website
> <https://l4steam.github.io/#prague-requirements-compliance>
>
>
>
> Thanks to NVIDIA for sharing their view and feedback for their GeforceNow
> congestion control. Their feedback was added to the above overview about a
> week ago. As we didn’t get the explicit approval for the others, we will
> share and present a consolidated view of all feedback received later and
> during the meeting.
>
>
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *De Schepper, Koen
> (Nokia - BE/Antwerp)
> *Sent:* Monday, February 8, 2021 2:37 PM
> *To:* Ingemar Johansson S <[email-masked]>; tsvwg IETF
> list <[email-masked]>
> *Subject:* Re: [tsvwg] Prague requirements survey
>
>
>
> Hi Ingemar,
>
>
>
> Thanks for your contributions. I linked your doc to the
> https://l4steam.github.io/#prague-requirements-compliance web page (and
> will do so for others).
>
>
>
> I didn’t see any issues or objections mentioned to the current
> requirements as specified in the draft. Does this mean you think they are
> all reasonable, valid and feasible?
>
>
>
> Interesting observation (related to the performance optimization topic 1)
> that for the control packets “RTCP is likely not using ECT(1)”. Why is this
> not likely? I assume this will impact the performance? Do we need to
> recommend the use of ECT(1) on RTCP packets in the draft?
>
>
>
> Thanks,
>
> Koen.
>
>
>
> *From:* Ingemar Johansson S <[email-masked]>
> *Sent:* Monday, February 8, 2021 10:59 AM
> *To:* De Schepper, Koen (Nokia - BE/Antwerp) <
> [email-masked]>; tsvwg IETF list <[email-masked]>
> *Cc:* Ingemar Johansson S <[email-masked]>
> *Subject:* RE: Prague requirements survey
>
>
>
> Hi
>
> *From:* tsvwg <[email-masked]> *On Behalf Of *De Schepper, Koen
> (Nokia - BE/Antwerp)
> *Sent:* den 6 februari 2021 23:20
> *To:* tsvwg IETF list <[email-masked]>
> *Subject:* [tsvwg] Prague requirements survey
>
>
>
> Hi all,
>
>
>
> To get a better understanding on the level of consensus on the Prague
> requirements, we prepared an overview document listing the L4S-ID draft
> requirements specific to the CC (wider Prague requirements), as a
> questionnaire towards potential CC developers. If you are developing or
> have developed an L4S congestion control, you can describe the status of
> your ongoing development in the second last column. If you cannot share
> status, or plan-to/would implement an L4S CC, you can list what you would
> want to support (see feasible). In the last column you can put any
> description/limitations/remarks/explanations related to evaluations,
> implementations and/or plans (will implement or will not implement). Any
> expected or experienced issues and any objections/disagreements to the
> requirement can be explained and colored appropriately.
>
>
>
> The document can be found on following link:
> https://raw.githubusercontent.com/L4STeam/l4steam.github.io/master/PragueReqs/Prague_requirements_Compliance_and_Objections_template.docx
> <https://protect2.fireeye.com/v1/url?k=d16bc960-8ef0f066-d16b89fb-86ee86bd5107-080c65bfd839440d&q=1&e=7dbb7494-67c3-4315-88a6-325f32e4e8b1&u=https%3A%2F%2Fraw.githubusercontent.com%2FL4STeam%2Fl4steam.github.io%2Fmaster%2FPragueReqs%2FPrague_requirements_Compliance_and_Objections_template.docx>
>
>
>
> As an example I filled it for the Linux TCP-Prague implementation on
> following link:
> https://l4steam.github.io/PragueReqs/Prague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx
> <https://protect2.fireeye.com/v1/url?k=f839c5f7-a7a2fcf1-f839856c-86ee86bd5107-29dabadc5d0e673d&q=1&e=7dbb7494-67c3-4315-88a6-325f32e4e8b1&u=https%3A%2F%2Fl4steam.github.io%2FPragueReqs%2FPrague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx>
>
>
>
> Please send your filled document to the list (Not sure if an attachment
> will work, so I assume you also need to store it somewhere and send a link
> to it, or send to me directly).
>
>
>
> We hope to collect many answers, understanding the position of the
> different (potential) implementers and come faster to consensus.
>
>
>
> Thanks,
>
> Koen.
>

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 16 Apr 2021 16:16:17 +0200

Hi Koen,

Thanks,.

Here is a question for Apple though:

"5. Reduce RTT dependence (A1.5)
Section 4.3: A scalable congestion control MUST eliminate RTT bias as much as possible in the range between the minimum likely RTT and typical RTTs expected in the intended deployment scenario.
Apple's comment:		
Again, agreed with the rationale behind this and the MUST compliance. This might be easy to implement as well based on heuristics but will require thorough testing."

If this is easy to implement, could you please propose a description of such a solution to the mailing list please? As far as I can tell RT- bias has been a topic of research for decades and still no general solution has beed presented, so I am quite interested to learn more about this comment. Even if the response is something like "for the expected range of RTTs from 1ms to 20 ms" a solution like TCP Pragues, pretend all RTTs are 20ms" I am quite interested in apple's thoughts.

Best Regards
	Sebastian

> On Apr 16, 2021, at 14:52, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi all,
>  
> An update on the survey is available. We received an additional input from Apple which we could publicly share (thanks Vidhi for providing this input). I also updated the consolidated view v2 (available onhttps://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance).
>  
> I believe it is strongly in line with the previous survey conclusions as presented in last tsvwg. One main additional feedback was on “7. Measuring Reordering Tolerance in Time Units”. There was disagreement that using time only and not packet count is a foolproof solution. As far as I understand the objection is to the current wording that a time based mechanism is the only/sufficient way to assure this.
>  
> The objective of this requirement is to allow a certain level of reordering for L4S traffic (actually avoid delaying packets in the network to guarantee correct order of packet delivery). I personally could support wording that expresses the core of the requirement, and not limit the text to one mechanism, which would allow alternative/more robust implementations. The requirement could be expressed as something like: “a scalable congestion control SHOULD  be resilient to reordering over an (adaptive) (time?) interval, which scales with / adapts to throughput, as opposed to counting only in (fixed) units of packets (as in the 3 DupACK rule of RFC 5681 TCP), which is not scalable”. Let’s further discuss here on the list what could be for all parties an acceptable wording.
>  
> Thanks,
> Koen.
>  
>  
> The only strong objections were against the “MUST document” requirements, which will be removed from the next version of the draft. Some clarifications were asked and (will be) added.
> From: tsvwg <[email-masked]> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
> To: tsvwg IETF list <[email-masked]>
> We have received several surveys privately, for which I tried to get the approval for sharing those on the overview page: l4steam.github.io | L4S-related experiments and companion website
>  
> Thanks to NVIDIA for sharing their view and feedback for their GeforceNow congestion control. Their feedback was added to the above overview about a week ago. As we didn’t get the explicit approval for the others, we will share and present a consolidated view of all feedback received later and during the meeting.
>  
> From: tsvwg <[email-masked]> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
> To: Ingemar Johansson S <[email-masked]>; tsvwg IETF list <[email-masked]>
> I didn’t see any issues or objections mentioned to the current requirements as specified in the draft. Does this mean you think they are all reasonable, valid and feasible?
>  
> Interesting observation (related to the performance optimization topic 1) that for the control packets “RTCP is likely not using ECT(1)”. Why is this not likely? I assume this will impact the performance? Do we need to recommend the use of ECT(1) on RTCP packets in the draft?
>  
> Thanks,
> Koen.
>  
> From: Ingemar Johansson S <[email-masked]> 
> Cc: Ingemar Johansson S <[email-masked]>
> From: tsvwg <[email-masked]> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
> To: tsvwg IETF list <[email-masked]>
> The document can be found on following link: https://raw.githubusercontent.com/L4STeam/l4steam.github.io/master/PragueReqs/Prague_requirements_Compliance_and_Objections_template.docx
>  
> As an example I filled it for the Linux TCP-Prague implementation on following link: https://l4steam.github.io/PragueReqs/Prague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx
>

---

From: Vidhi Goel <[email-masked]>
Date: Fri, 16 Apr 2021 14:14:02 -0700

>      the 3 DupACK rule of [RFC5681] and [RFC6675], which is not

Thanks Neal. This wording works for me.

> On Apr 16, 2021, at 7:08 AM, Neal Cardwell <[email-masked]> wrote:
> 
> On Fri, Apr 16, 2021 at 8:53 AM De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked] <mailto:[email-masked]>> wrote:
> Hi all,
> 
>  
> 
> An update on the survey is available. We received an additional input from Apple which we could publicly share (thanks Vidhi for providing this input). I also updated the consolidated view v2 (available on https://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance <https://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance>).
> 
>  
> 
> https://l4steam.github.io/PragueReqs/Apple_L4S_requirements_Compliance_and_Objections.pdf <https://l4steam.github.io/PragueReqs/Apple_L4S_requirements_Compliance_and_Objections.pdf>
> 
> And specifically regarding the response about a reordering tolerance in time units:
>   
> https://tools.ietf.org/html/rfc8985 <https://tools.ietf.org/html/rfc8985> ] if no reordering has been
> To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked] <mailto:[email-masked]>>; tsvwg IETF list <[email-masked] <mailto:[email-masked]>>
> Cc: Bob Briscoe <[email-masked] <mailto:[email-masked]>>
> The details of the consolidated view of all feedback received is available and can be found via following link: https://l4steam.github.io/PragueReqs/Prague_requirements_consolidated.pdf <https://l4steam.github.io/PragueReqs/Prague_requirements_consolidated.pdf>
>  
> 
> From: tsvwg <[email-masked] <mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
> To: tsvwg IETF list <[email-masked] <mailto:[email-masked]>>
> We have received several surveys privately, for which I tried to get the approval for sharing those on the overview page: l4steam.github.io | L4S-related experiments and companion website <https://l4steam.github.io/#prague-requirements-compliance>
>  
> 
> From: tsvwg <[email-masked] <mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
> To: Ingemar Johansson S <[email-masked] <mailto:[email-masked]>>; tsvwg IETF list <[email-masked] <mailto:[email-masked]>>
> Thanks for your contributions. I linked your doc to the https://l4steam.github.io/#prague-requirements-compliance <https://l4steam.github.io/#prague-requirements-compliance> web page (and will do so for others).
> 
>  
> 
> From: Ingemar Johansson S <[email-masked] <mailto:[email-masked]>> 
> To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked] <mailto:[email-masked]>>; tsvwg IETF list <[email-masked] <mailto:[email-masked]>>
> Cc: Ingemar Johansson S <[email-masked] <mailto:[email-masked]>>
> From: tsvwg <[email-masked] <mailto:[email-masked]>> On Behalf Of De Schepper, Koen (Nokia - BE/Antwerp)
> To: tsvwg IETF list <[email-masked] <mailto:[email-masked]>>
> The document can be found on following link: https://raw.githubusercontent.com/L4STeam/l4steam.github.io/master/PragueReqs/Prague_requirements_Compliance_and_Objections_template.docx <https://protect2.fireeye.com/v1/url?k=d16bc960-8ef0f066-d16b89fb-86ee86bd5107-080c65bfd839440d&q=1&e=7dbb7494-67c3-4315-88a6-325f32e4e8b1&u=https%3A%2F%2Fraw.githubusercontent.com%2FL4STeam%2Fl4steam.github.io%2Fmaster%2FPragueReqs%2FPrague_requirements_Compliance_and_Objections_template.docx>
>  
> 
> As an example I filled it for the Linux TCP-Prague implementation on following link: https://l4steam.github.io/PragueReqs/Prague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx <https://protect2.fireeye.com/v1/url?k=f839c5f7-a7a2fcf1-f839856c-86ee86bd5107-29dabadc5d0e673d&q=1&e=7dbb7494-67c3-4315-88a6-325f32e4e8b1&u=https%3A%2F%2Fl4steam.github.io%2FPragueReqs%2FPrague_requirements_Compliance_and_Objections_Linux_TCP-Prague.docx>
>  
>

---

From: Vidhi Goel <[email-masked]>
Date: Fri, 16 Apr 2021 14:21:11 -0700

Hi Sebastian,

> If this is easy to implement, could you please propose a description of such a solution to the mailing list please? As far as I can tell RT- bias has been a topic of research for decades and still…

I think the simple proposal in Linux (which you already know) is a good starting point. As a community, we might come up with more heuristics / tunable parameters to handle edge cases.
https://l4steam.github.io/PragueReqs/Linux_TCP_Prague_L4S_requirements_Compliance_and_Objections.pdf <https://l4steam.github.io/PragueReqs/Linux_TCP_Prague_L4S_requirements_Compliance_and_Objections.pdf>

Thanks,
Vidhi

> On Apr 16, 2021, at 7:16 AM, Sebastian Moeller <[email-masked]> wrote:
> 
> Hi Koen,
> 
> Thanks,.
> 
>> On Apr 16, 2021, at 14:52, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi all,
>>  
>> An update on the survey is available. We received an additional input from Apple which we could publicly share (thanks Vidhi for providing this input). I also updated the consolidated view v2 (available onhttps://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance <https://github.com/L4STeam/l4steam.github.io#prague-requirements-compliance>).
>>

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 17 Apr 2021 00:45:19 +0200

Hi Vidhi,

> On Apr 16, 2021, at 23:21, Vidhi Goel <[email-masked]> wrote:
> 
> Hi Sebastian,
> 
> I think the simple proposal in Linux (which you already know) is a good starting point. A

	Are we talking about the same Linux proposal here ((https://github.com/L4STeam/linux/commit/a2ef76f8da1c9d1b13fa941f55607f3e60d4112e)? Where TCP Prague is instructed to basically behave like ("pretend") there was a fixed lower bound RTT when growing the congestion window? IMHO that is not a good starting point for fixing RTT-bias, given that that over the internet RTTs easily range from low single to low triple digit ms. Unless we are prepared to set, say 100 ms as our lower bound RTT this approach will not work for the internet, but once we do that we are giving up one of the advantages that high frequency congestion signaling is promising, faster reactions to congestion signals. 

	I want to note that TCP Prague by default only tries to equalize RTTs up to 25 ms, which indicates that not even its developers consider it as a generic solution (or they believe that 25ms is a "magic" RTT on the internet). I also note that the root-cause for adding that feature to TCP Prague, was/is the fact that the dual queue coupled AQM failed to properly share capacity between its two queues at low RTTs. This failure is rationalized as being an effect of RTT-bias caused by the difference in queueing delay between the two queues (~1ms fir the LL queue, ~20ms for the classic queue) and the proposed solution is to make TCP Prague not grow its congestion window faster than a 25 ms RTT flow. In other words this is not meaningfully addressing RTT-bias, but is fixing a deficiency in L4S's reference AQM*.

> s a community, we might come up with more heuristics / tunable parameters to handle edge cases.

	Sorry, for the last decades people have worked on RTT-bias and no generic solution based solely on end-point actions has been found. I am not saying that this is impossible, but it it is quite unlikely that this is easy enough for the community to come up with a solution. And TCP Pragure is IMHO not a promising contender for a generic solution.
	IMHO, the problem is that the issue is not caused by the endpoints in the first place, but by the interaction of control loops of different "fidelity"/reaction times in bottleneck buffers. This can easily be seen in that a properly configured TCP flow can approach bottleneck capacity when run as the sole flow over a bottleneck, but will be suppressed if competing with TCP flows of shorter RTT in the same bottleneck. It hence seems clear that management of the bottleneck is at least as important to counter RTT-bias as the endpoints's control loops. The L4S approach of relegating the issue solely to the endpoints/protocols to fix, instead of also making the AQM part of the solution strikes me as short-sighted especially in the light of deployment of an AQM being one of the core pillars of the L4S design.

> https://l4steam.github.io/PragueReqs/Linux_TCP_Prague_L4S_requirements_Compliance_and_Objections.pdf

Best Regards
	Sebastian

*) And doing so before actual deployment, at a point in time when that AQM could actually still be fixed for good.

---

From: Vidhi Goel <[email-masked]>
Date: Sat, 17 Apr 2021 19:48:27 -0700

Hi Sebastian,

> 	I want to note that TCP Prague by default only tries to equalize RTTs up to 25 ms, which indicates that not even its developers consider it as a generic solution (or they believe that 25ms is a "m…

Yes, we are talking about the same proposal.
At the time I read the Linux Prague proposal, I didn’t realize the rationale behind it and now I understand it better with your reasoning. I agree that we should not fix RTT bias which is purely created by the L4S dual queue.

> 	IMHO, the problem is that the issue is not caused by the endpoints in the first place, but by the interaction of control loops of different "fidelity"/reaction times in bottleneck buffers. This ca…
The problem of RTT-unfairness arises from different ACK clocking speeds based on RTT. If the propagation delay is different for two flows, then there is nothing that AQM can do. OTOH, if the propagation delay is same for two flows, and it is really the buffering (queuing) delay that is causing RTT unfairness, then I agree with you that we should solve this problem at the bottleneck.

I believe you are concerned about the latter scenario and yes in this case, we should not try to solve the RTT bias at the endpoint as that could be counter productive to what we are trying to achieve with scalable congestion controllers.

Thanks,
Vidhi

> On Apr 16, 2021, at 3:45 PM, Sebastian Moeller <[email-masked]> wrote:
> 
> Hi Vidhi,
> 
>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 18 Apr 2021 08:41:31 +0000

Ok, then we’ll take that wording and we can close this topic. Thanks both.

Koen.

From: Vidhi Goel <[email-masked]>
Sent: Friday, April 16, 2021 11:14 PM
To: Neal Cardwell <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg IETF list <[email-masked]>; Christoph Paasch <[email-masked]>; Yuchung Cheng <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Randall Stewart <[email-masked]>; Priyaranjan Jha <[email-masked]>; Nandita Dukkipati <[email-masked]>
Subject: Re: [tsvwg] Prague requirements survey

On Apr 16, 2021, at 7:08 AM, Neal Cardwell <[email-masked]<mailto:[email-masked]>> wrote:

On Fri, Apr 16, 2021 at 8:53 AM De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]<mailto:[email-masked]>> wrote:
Hi all,

Thanks,
Koen.

Cc: Bob Briscoe <[email-masked]<mailto:[email-masked]>>

Hi all,

Regards,
Koen.

Hi all,

Koen.

Hi Ingemar,

Thanks,
Koen.

Hi

Regards
Ingemar

Hi all,

Thanks,
Koen.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 18 Apr 2021 11:54:55 +0200

Hi Vidhi,

thank you for response.

Question: Does this discussion actually change your/apple position in regards to the TCP Prague requirements? (I assume not*, but it would be helpful to make that explicit for the sake of further discussions)

*) Because reducing RTT bias is a worthy goal and "playing dumb" as TCP Prague does is at least a theoretical viable way forward, albeit an IMHO unattractive one.

> On Apr 18, 2021, at 04:48, Vidhi Goel <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] +1; as a side note figure 6 (https://abload.de/img/thegoodthebadandthewinpjmx.png) of Høiland-Jørgensen, Toke, Per Hurtig, and Anna Brunstrom. ‘The Good, the Bad and the WiFi: Modern AQMs in a Residential Setting’. Computer Networks 89 (4 October 2015): 90–106. https://doi.org/10.1016/j.comnet.2015.07.014. (https://www.sciencedirect.com/science/article/pii/S1389128615002479) shows the RTT fairness of different bottleneck management solutions. L4S dual queue coupled AQM is not unusual in making RTT-unfairness worse than the reference FIFO (see the ared, pie, and codel columns), but IMHO it does so even stronger than the traditional AQMs. Also note how AQMs can actually do equal or better than FIFO (sfq and fq_codel, as well as the pure FQ scheduler without AQM, fq_nocodel).
	In other words there are AQMs that can ameliorate RTT-bias, it is just that L4S's dual queue coupled AQM is not among them.
All of this is in addition to the fact that the dual queue coupled AQM at its core is a stricht 1:16 (classic:L4S) scheduler where the unfairness between the two queues is only limited to 1:16, with a bunch of clever heuristics to not cause this situation all too often. But it was clear from the beginning, that extremely shoer RTTs would not be covered by those heuristics. And instead of say fixing the issue by mandating a less extreme priority split (say 1:4) L4S opted to mandate protocols to fix up the short comings of the AQM.

> The problem of RTT-unfairness arises from different ACK clocking speeds based on RTT. If the propagation delay is different for two flows, then there is nothing that AQM can do.

	[SM] That is essentially what I tried to call "reaction time" above, but also see the examples referenced above showing how AQM can actually do something to make things better. Really, since the problem occurs by the interaction in the bottlenecks buffers an AQM is in a very good position to help, and that even without requiring cooperation of protocols! As a further example how AQM can actually help, have a look at https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667 the last 4 pairs of bars show an approximate fairness AQM in action, compare that to the first 4 pairs, with the dual queue coupled AQM. The L4S solution is clearly deficient compared to the state of the art, heck if you compare the pair 1 (2 cubic flows in L4S classic queue) and pair 5 (2 cubic flows in a FIFO) you see that L4S' AQM is considerably worse than the status quo in the internet the unmanaged FIFO.
	The same plot also shows why TCP Prague is not ready for internet scale deployment, as it fares quite badly if it has to compete with CUBIC in a bottleneck (pairs 6 and 7).

> OTOH, if the propagation delay is same for two flows, and it is really the buffering (queuing) delay that is causing RTT unfairness, then I agree with you that we should solve this problem at the bottleneck.

	[SM] Yes! Cleaning up an AQM's mess should be made a burden for protocols.

> I believe you are concerned about the latter scenario and yes in this case, we should not try to solve the RTT bias at the endpoint as that could be counter productive to what we are trying to achi…

	[SM] Exactly. BTW, I agree that making protocols less RTT-unfair is a worthy goal we should strive for, but it is not an easy problem to solve (judged by how long we know about RTT-bias and how little progress we made so far in tackling it in protocols).

Best Regards
	Sebastian

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Sun, 18 Apr 2021 10:11:38 +0000

Hi Sebastian, Vidhi,

Some background:

RTT (in)dependence is an end-point property, that can best be corrected in the endpoints. It could be solved in the network if:
- the network can identify flows and schedule then accordingly: FQ_x
- the endpoints add more info (RTT) in the packet headers: RCP, XCP, ...
- the network can identify flows and adapts the marking/drop to let flows converge to the same throughput: CSAQM, ...
- the network can identify flows and "estimate/guess" their RTT and adapt the marking/drop probability: I'm sure you will find research...
- ...

So unless we further expand headers or make it the responsibility to identify and schedule flows, measure rates or guess RTTs in the network, we better solve the problem in the entity that causes it. Assuming from here we restrict us to the latter:

Sebastian, as you mentioned, there has been a lot of research in the past, which resulted in a lot of solutions. The problem is not that these don't work, but that converging at a lower rate for the lower end RTTs requires a clean slate reset of all endpoints, as nobody would start doing that unilaterally. L4S is a clean slate starting point where we set new rules. During the first Prague meeting there was a lot of enthusiasm and proposals just because of this new opportunity. So if we want to introduce RTT independence, this is the moment, and all this previous research can now be used and be deployed.

As L4S removed the queue completely when competing with Classic over a DualQ and limits it to 1ms when not, the previous role of the (large) queue to middle out RTT unfairness completely disappears. So even in an L4S-only world, the unfairness would be unsustainable and needs a solution. As this also solves the DualQ created imbalance is part of the total concept (which otherwise could only be solved by setting both queues to the same RTT target, defying the purpose of the 2 queues and DualQ at all).

Then RTT independence, means that we need to converge to a (more) equal rate when RTTs are different. This means that based on the marking signal we need to agree on a common marking rate (which automatically emerges when a marker marks packets with equal probability and all flows have an equal rate). If such a marking rate in marks per second is defined, it can automatically be translated in a reference RTT when taking the marks per RTT of an existing congestion control into account. This is not a trick, or hack, it is just a result of the concept (RTT independence). We can discuss a reference in marks per second (marking rate) or in terms of a Reference RTT (DCTCP AIMD converging to a fixed 2 marks per RTT being a good reference base CC for that purpose). So "pretending" to be a 25ms is exactly what we do if we set the Reference RTT to 25ms, or set the marking rate to 80 marks per second. For evenly distributed marks this would be around 12.5ms per mark, which would be a very frequent signal for converging to a fair rate. 25ms also is a useful number, as it is a practical lower limit for Classic RTTs on congested links, and require no changes for the Classic flows. It is of course an additional opportunity for Classic CCs to also increase the rate for higher RTTs (but not necessarily for lower RTTs, although this wouldn't hurt them much, as under congestion they wouldn't see lower RTTs on the Internet anyway). CuBic was designed to be a fairness/performance compromise to Reno on the longer RTTs. If it would have been acceptable at that time, they would have set the compromise more towards the performance. Today most traffic comes from nearby datacenters, making most traffic experiencing less than 50ms latency, so I believe setting a reference RTT of 25ms also for higher RTTs would be completely acceptable today.

As a final remark, converging to a steady state rate in the past was always seen as a property of a single mechanism (AIMD of +1 and /2 for Reno, and +cubic(t) and *0.7 for Cubic). I believe we are past simple single ACK response mechanisms (see BBR, ...) where models based on measurements and different states adapt the response and selects appropriate mechanism. When we detect we are out of steady state (0% or 100% marking for a while), the selected mechanism can be RTT dependent (getting up to speed, avoiding latency, ...), once back in sync with the steady marking rate, the RTT independent response can be selected (whatever the mechanism is).

Hope this clarifies.

Koen.

-----Original Message-----
From: Vidhi Goel <[email-masked]> 
Sent: Sunday, April 18, 2021 4:48 AM
To: Sebastian Moeller <[email-masked]>
Cc: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; tsvwg IETF list <[email-masked]>
Subject: Re: [tsvwg] Prague requirements survey

Hi Sebastian,

Thanks,
Vidhi

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 18 Apr 2021 13:19:48 +0200

Hi Koen,

more below prefixed [SM].

Tl;dr: Both TCP Prague and the dual queue coupled AQM both independently increase the observable RTT-bias compared to both the status quo ("dumb" FIFOs) or the state of the art (mostly FQ or approximate FQ AQMs), neither supports the claim, that RTT-bias can and should be addressed in end-points only.

> On Apr 18, 2021, at 12:11, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian, Vidhi,
> 
> Some background:
> 
> RTT (in)dependence is an end-point property, that can best be corrected in the endpoints.

	[SM] History disagrees with you, there exis zero general RTT-fairness solutions that work purely from endpoints, but there are several solutions that work at the AQM level. I have posted references to plots showing that explicitly. So if you maintain this is fixable best from end-points, please post references demonstrating that fact.
	As I explained before, RTT-unfairness happens as an effect of interactions of the control-loops of different RTT flows INSIDE the bottleneck buffer, so unless you can fix the RTT differences, fixing it in the management of the bottleneck buffer is the second best (but more realistic) approach. If you disagree, aagin, please cite references of end-point only solutions that are actually palatable for internet scale RTTs (from 1ms (FTTH) to 500ms (geostationary satellites)). And no, TCP Pragues approach of playbing dumb seems not acceptable if the corrected RTT is set to a more realistic 100ms or realistic worst case of 500ms. If you disagree, I am waiting for TCP Pragues default change to >> 25ms.

> It could be solved in the network if:

	[SM] I agree, a control system that needs to arbitrate between different entities, needs to identify those entities.

> - the endpoints add more info (RTT) in the packet headers: RCP, XCP, ...

	[SM] Well, even without that additional information, sate of the art AQMs already do a much better job at RTT-fairness than L4S's reference AQM, heck, even a dumb FIFO seems superior here. So I agree, having per flow RTT information would theoretically helpful and good to have, it is by no means necessary to do better than L4S (given that even a FIFO does better).

> - the network can identify flows and "estimate/guess" their RTT and adapt the marking/drop probability: I'm sure you will find research...

	[SM] Again, for a perfect solution all of that seems to apply, but as the comparison with a FIFO, which does nothing of that sort demonstrates, it is well possible to do better than L4S even without all that trickery and sophistication. Let's aim for good enough, instead of perfect, and we see that the dual queue coupled AQM is simply defective in te RTT-bias department, and ameliorating that is possible without having to change end-point behavior.\

> So unless we further expand headers or make it the responsibility to identify and schedule flows, measure rates or guess RTTs in the network, we better solve the problem in the entity that causes i…

	[SM] Wrong root-cause and hence wrong solution. This looks also like an attempt to ignore that L4S noticeably increases RTT-bias at the AQM level, and trying to push the fix for that AQM deficiency into the end-points seems unduly onerous.

> 
> Sebastian, as you mentioned, there has been a lot of research in the past, which resulted in a lot of solutions.

	[SM] Except, we have no generally applicable solution that actually got deployed over the internet (okay, for some RTT/rate regimes CUBiC does better than say reno, but all of that is rather minor). So I agree there have been attempts at solutions, sure, but that is not the same as actual solutions, sorry.

> The problem is not that these don't work, but that converging at a lower rate for the lower end RTTs requires a clean slate reset of all endpoints, as nobody would start doing that unilaterally.

	[SM] Which means, these are not general solutions, sorry. But that is expected for end-point only solutions, these try to paper over the issue from the wrong place in the network, and hence can not solve the issue. And e.g. having all end-points agree to use an identical RTT independent cwnd-gowth function is just that, trying to paper over what happens in the bottleneck buffers. 

> L4S is a clean slate starting point where we set new rules. During the first Prague meeting there was a lot of enthusiasm and proposals just because of this new opportunity. So if we want to introduce RTT independence, this is the moment, and all this previous research can now be used and be deployed.

	[SM] Again, you are not doing that at all. all you defaulted TCP Prague to is to paper over the dual queue AQMs sub-optimal sharing behavior at short RTTs. If one repeated Pete's test with RTTs of 26 ms and 176 ms (Pete's 10/160 shifted to just fall outside of TCP Pragues "fixed" RTT range) one would still see considerably increased RTT bias as compared to a FIFO "solution". I fail to see, how you made any significant dent into RTT-bias at all, sorry.

> 
> As L4S removed the queue completely when competing with Classic over a DualQ and limits it to 1ms when not, the previous role of the (large) queue to middle out RTT unfairness completely disappears.

	[SM] Yes, you keep repeating that argument, but I keep telling you, since you are the party that inflicts ~20ms queueing delay on the classic queue, you can not blame the resulting unfairness as RTT-bias that neds to be fixed in the end-points. You broke it, you need to fix it.

> So even in an L4S-only world, the unfairness would be unsustainable and needs a solution.

	[SM] Yes, this needs a solution, but I do not think that you actually found one, given that TCP Prague over an L4S AQM shows stronger self RTT bias than CUBIC over an L4S AQM (see pairs 1 and 4, as well as 5 and 8 in https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667). This is one more data point for my hypothesis that trying to fix that in the end-points is a fool's errand. You are making things worse, and yet you market that under increasing RTT-independence, oh the irony.

> As this also solves the DualQ created imbalance is part of the total concept (which otherwise could only be solved by setting both queues to the same RTT target, defying the purpose of the 2 queues and DualQ at all).

	[SM] Please show data from a 26ms CUBIC flow competing with a 176 ms TCP Prague flow in an L4S AQM (as well as the opposite pair) as well as pairs of CUBIC, and TCP Prague flows at 26 and 176 competing with itself (all with the default 25ms value foe TCP Prague and a path RTT << 1 ms). I predict that L4S' RTT-bias is still going to be noticeably worse than the reference FIFO. If you manage to improve upon the default FIFO we can keep talking.

> 
> Then RTT independence, means that we need to converge to a (more) equal rate when RTTs are different. This means that based on the marking signal we need to agree on a common marking rate (which automatically emerges when a marker marks packets with equal probability and all flows have an equal rate).

	[SM] AS said before, that is a possible solution feasible in the end-points, but it is equivalent in trading in temporal fidelity in the response dynamics, which seems opposite to the goal behind high-frequency congestion signaling.

> If such a marking rate in marks per second is defined, it can automatically be translated in a reference RTT when taking the marks per RTT of an existing congestion control into account.

	[SM] Can you share data how well this would works if the reference RTT is shorter than the true path RTT of a flow. My intuition tells me this is going to fail unless refRTT >= pRTT, but I am happy to convinced otherwise by data.

> This is not a trick, or hack, it is just a result of the concept (RTT independence). We can discuss a reference in marks per second (marking rate) or in terms of a Reference RTT (DCTCP AIMD converging to a fixed 2 marks per RTT being a good reference base CC for that purpose). So "pretending" to be a 25ms is exactly what we do if we set the Reference RTT to 25ms, or set the marking rate to 80 marks per second.

	[SM] But you are not doing that by default in TCP Prague, you are simply set the cwnd growth dynamics to assume RTT = max(25ms, pathRTT), that is not what you claim above. Not a vote of confidence in your own solution, sorry.

> For evenly distributed marks this would be around 12.5ms per mark, which would be a very frequent signal for converging to a fair rate. 25ms also is a useful number, as it is a practical lower limit for Classic RTTs on congested links, and require no changes for the Classic flows.

	[SM] Except for the fact that the delay target of the classic AQM is a configurable number (e.g. in dual queue coupled AQM, see https://github.com/L4STeam/iproute2/blob/master/tc/q_dualpi2.c), now for this to make sense, the potentially user-configurable number in the end-point protocol needs to be kept in lock step with the operator configurable number of all L4S AQMs along the path... this does not appear to be a robust and reliable engineering solution to me, sorry, especially since there is no easy way for end-points to measure the value fro the network. But I have made this argument before. IFF you believe this to be the way forward, you would at least need to make these numbers only configurable at compile time to make it reasonably likely that these actually match.

> It is of course an additional opportunity for Classic CCs to also increase the rate for higher RTTs (but not necessarily for lower RTTs, although this wouldn't hurt them much, as under congestion they wouldn't see lower RTTs on the Internet anyway).

	[SM] Assuming all bottlenecks are L4S-compliant, which is not going to happen any time soon (if at all). So we are now talking about two separate code paths depending on the estimated behaviour of the bottleneck along the path. Not sure how realistic any such change is going to be for the duration of the L4S experiment.

> CuBic was designed to be a fairness/performance compromise to Reno on the longer RTTs.

	[SM] Yes, and it does so, mildly. Note how CUBIC did not fix RTT-bias for good.

> If it would have been acceptable at that time, they would have set the compromise more towards the performance.
> Today most traffic comes from nearby datacenters,

	[SM] Often, but not always, we need to make sure that longer RTT flows will not be starved by those common short RTT flows, otherwise we will considerable decrease the range of reasonably reachable internet. I am not convinced that e.g. European gamers playing by choice on an east coast server, will be happy if their "siblings" video streaming will kill their gaming sessions fidelity, by virtue of coming from close by CDNs. Short RTT flows will always have an edge, by virtue of their tighter control loops, no need to make that worse.

> making most traffic experiencing less than 50ms latency, so I believe setting a reference RTT of 25ms also for higher RTTs would be completely acceptable today.

	[SM] NO, as the consequence of that choice is strangling/starving longer RTT flows. This confirms my hypothesis that L4S really is just an exercise in building a short-RTT priority/fast-lane, but I do not believe that the internet as a whole was waiting for that, sorry.

Best Regards
	Sebastian

> As a final remark, converging to a steady state rate in the past was always seen as a property of a single mechanism (AIMD of +1 and /2 for Reno, and +cubic(t) and *0.7 for Cubic). I believe we are past simple single ACK response mechanisms (see BBR, ...)
> where models based on measurements and different states adapt the response and selects appropriate mechanism. When we detect we are out of steady state (0% or 100% marking for a while), the selected mechanism can be RTT dependent (getting up to speed, avoiding latency, ...), once back in sync with the steady marking rate, the RTT independent response can be selected (whatever the mechanism is).
> 
> Hope this clarifies.
> 
> Koen.
>

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 18 Apr 2021 16:00:22 +0200

Dear list:

> [...]

[SM] According to for example https://arxiv.org/pdf/1706.09115.pdf BBR actually suffers from a reverse RTT unfairness, in which longer RTT flows overwhelm shorter RTT flows. While this is intriguingly hinting the possibility of a control loop design that might fall in between the traditional shorter RTT wins and BBR's longer RTT wins, it also demonstrates that RTT-bias is still not a solved problem, even for modern TCP CC as sophisticated and clever as BBR. I take this as support for my hypothesis, that the end-points alone simply are not well suited to solve this problem; no matter how convenient that would be.

Best Regards
	Sebastian

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 18 Apr 2021 14:54:47 +0000

The link below shows results (by no means complete) from an early experiment
(2018) to make TCP BBR L4S capable
https://datatracker.ietf.org/meeting/101/materials/slides-101-iccrg-bbr-cong
estion-control-with-l4s-support-02

/Ingemar

> From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
> Sent: den 18 april 2021 16:00
> To: De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-
> bell-labs.com>
> Cc: tsvwg IETF list <[email-masked]>
> > On Apr 18, 2021, at 12:11, De Schepper, Koen (Nokia - BE/Antwerp)
> <[email-masked]> wrote:
> > [...]
> > As a final remark, converging to a steady state rate in the past was
always
> seen as a property of a single mechanism (AIMD of +1 and /2 for Reno, and
> +cubic(t) and *0.7 for Cubic). I believe we are past simple single ACK
> response mechanisms (see BBR, ...) where models based on measurements
> and different states adapt the response and selects appropriate mechanism.
> When we detect we are out of steady state (0% or 100% marking for a
while),
> the selected mechanism can be RTT dependent (getting up to speed,
> avoiding latency, ...), once back in sync with the steady marking rate,
the RTT
> independent response can be selected (whatever the mechanism is).
> 
> [SM] According to for example https://arxiv.org/pdf/1706.09115.pdf BBR
> actually suffers from a reverse RTT unfairness, in which longer RTT flows
> overwhelm shorter RTT flows. While this is intriguingly hinting the
possibility
> of a control loop design that might fall in between the traditional
shorter RTT
> wins and BBR's longer RTT wins, it also demonstrates that RTT-bias is
still not a
> solved problem, even for modern TCP CC as sophisticated and clever as BBR.
> I take this as support for my hypothesis, that the end-points alone simply
are
> not well suited to solve this problem; no matter how convenient that would
> be.
> 
> 
> Best Regards
> 	Sebastian