Subject: [tsvwg] L4S and the RACK requirement

From: Wesley Eddy <[email-masked]>
Date: Tue, 12 Feb 2019 13:00:47 -0500

In discussion among the TSVWG chairs, we are concerned about lack of 
consensus on the requirement currently in L4S ID draft ( 
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05 ) regarding 
the need for RACK-like behavior in a transport that uses the L4S queue.

The statement in the draft is:

     A scalable congestion control MUST detect loss by counting in units 
of time, which is scalable, and MUST NOT count in units of packets (as 
in the 3 DupACK rule of traditional TCP), which is not scalable (see 
Appendix A.1.7 for rationale).

By saying this, it seems to rule out DCTCP and some other existing code 
that might be used with L4S (and DCTCP discussed in the draft as an 
example scalable transport, even though it violates this rule (?)).  
This seems like a bit of a problem for making L4S usable.  I guess maybe 
TCP Prague code fixes this, but isn't as widely available yet?

The discussion in the appendix is good at explaining what I think the 
real goal is here, which is to enable major reduction in latency from 
link-layer (or other underlying transport network) re-ordering buffers.  
We want that in order to meet the low latency goals, which makes total 
sense.

So, my question is whether the "MUST" is really more appropriately 
turned into a "SHOULD" guidance?  Given that we expect reordering to be 
possible (and maybe normal) over hops supporting L4S, then the 
congestion control algorithm SHOULD have mechanisms that allow it to 
perform robustly.  If it doesn't, it only hurts itself, not any other 
traffic, so there seems to be no real reason to say "MUST" (someone 
violating it doesn't break the Internet or cause interop issues, etc).  
As I understand it, this would allow the examples like DCTCP to be 
relevant for use with L4S as well.

Does Bob or anyone else have thoughts on this?

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 12 Feb 2019 16:27:16 -0800

> example scalable transport, even though it violates this rule (?)).
I am missing something: how does DCTCP depend a specific loss
detection mechanism (e.g. DupACK based, RACK, etc)? DCTCP is only
mandated to react to packet losses.

Linux DCTCP uses RACK by default. There's no inherent dependency of
the two either.

---

From: "Black, David" <[email-masked]>
Date: Wed, 13 Feb 2019 03:34:15 +0000

> Linux DCTCP uses RACK by default. There's no inherent dependency of

See slide 8 from Bob's Bangkok presentation to TCPM (and the rest of that
presentation) - the requirement in the ecn-l4s-id draft is more restrictive
than RACK as currently implemented - current DCTCP with RACK doesn't
meet that requirement.
 
https://datatracker.ietf.org/meeting/103/materials/slides-103-tcpm-sessa-l4s-rack-00

We (TSVWG chairs) are looking for "running code" ... as we're having difficulty
figuring out how the L4S experiment (the L4S drafts are intended to become
 experimental RFCs) would be carried out without any transport protocol code.

Thanks, --David

> -----Original Message-----
> From: tsvwg [mailto:[email-masked]] On Behalf Of Yuchung Cheng
> Sent: Tuesday, February 12, 2019 7:27 PM
> To: Wesley Eddy
> Cc: [email-masked]
> Subject: Re: [tsvwg] L4S and the RACK requirement
> 
> 
> [EXTERNAL EMAIL]
>

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 14 Feb 2019 11:15:53 -0800

>  experimental RFCs) would be carried out without any transport protocol code.
Thanks for the pointer. I too think RACK seems a recommended
component, not a necessity based on the slides.

Perhaps Bob can shed more lights on how TCP L4S would break badly w/o RACK?

---

From: "Black, David" <[email-masked]>
Date: Thu, 14 Feb 2019 20:18:20 +0000

> Perhaps Bob can shed more lights on how TCP L4S would break badly w/o

Start by looking at the rest of that slide deck:

Thanks, --David

> From: Yuchung Cheng [mailto:[email-masked]]
> Sent: Thursday, February 14, 2019 2:16 PM
> To: Black, David
> Cc: Wesley Eddy; [email-masked]

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 15 Feb 2019 17:20:19 +0100

OTOH, for non-RACK (time-based) stacks, mechanisms like this
https://tools.ietf.org/html/draft-zimmermann-tcpm-reordering-detection-02

should also adjust to the reordering extent - the current wording of L4S 
seems to preclude any enhancement along those lines to qualify for use 
of L4S, even thoug a dynamic measurement of the reordering extent would 
implicitly have some time component as measured, correct?

Also, for the proposed RTT/8, what would be a default starting value for 
RACK? 1 sec / 8 = 125 ms?

Best regards,
   Richard

---

From: [email-masked]
Date: Mon, 18 Feb 2019 12:36:08 +0000 (UTC)

Am I the only person who remembers Savage attacks from ack splitting and packet counting?
http://cseweb.ucsd.edu/~savage/papers/CCR99.pdf 
there's the support for the MUST NOT, at least. I's not the lack of scale that is the major issue there, it's being open to abuse. ack counting didn't work...
Time is a tricky thing to handle with vatying scale, as delay-tolerant networking has shown.

Lloyd [email-masked]

On Wednesday, February 13, 2019, 5:00 am, Wesley Eddy <[email-masked]> wrote:

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 21 Feb 2019 10:16:34 +0000

David, Richard, Yuchung, all,

1/ Let me explain why I would still argue for MUST, even tho it strictly 
rules out existing DCTCP (cos of its initial use of 3DupACK).

Existing DCTCP does not comply with other requirements in the draft 
either. The requirements started life as a list of things to change in 
DCTCP. DCTCP has always been put forward as an interim solution that is 
sufficiently close to the requirements that it can be used for initial 
testing / demonstration.

The draft is experimental track. It states how the code of all L4S 
senders has to behave if you want to get meaningful results from an 
experiment. For an experimental draft, I would argue against saying "You 
SHOULD do X", if actually you MUST do X before you can expect to get the 
results claimed by the draft.

2/ To coexist safely with other Internet traffic?

Nonetheless, this discussion has helped me see a deeper problem with 
this list of L4S congestion control requirements. I think this is the 
root cause of David's concern.

The time-units requirement is about inter-operability with any links 
that do not maintain strict ordering. It is not about safe co-existence 
with other traffic. That's true for some of the other requirements too. 
So I propose that we split up the list into:

  * those requirements that really are about safe coexistence with other
    traffic
  * those requirements that are about interoperability with other parts
    of the L4S experiment (where safety is too strong a word)

Then code that does not meet any safety requirement cannot be used 
safely on the public Internet. Whereas code that doesn't interoperate 
with the rest of the L4S experiment can still be used on the Internet, 
it just won't necessarily give meaningful results in an L4S experiment.

For instance "MUST react to packet loss in a way that will coexist 
safely with a TCP Reno" is a safety requirement. So a DCTCP that does 
not satisfy this requirement cannot safely be used on the Internet For 
instance, Windows DCTCP is safe on the public Internet, but current 
mainline Linux DCTCP is not because there's bug where it intends to fall 
back to Reno - here's Koen's patch 
<https://www.spinics.net/lists/netdev/msg420916.html> from 2yrs ago.

Additional responses (to Richard) tagged [BB] inline...

On 15/02/2019 16:20, Scheffenegger, Richard wrote:
> should also adjust to the reordering extent - the current wording of 
> L4S seems to preclude any enhancement along those lines to qualify for 
> use of L4S, even thoug a dynamic measurement of the reordering extent 
> would implicitly have some time component as measured, correct?
[BB] If an approach implicitly uses time, surely that satisfies a 
requirement to use units of time?

>
>
> Also, for the proposed RTT/8, what would be a default starting value 
> for RACK? 1 sec / 8 = 125 ms?
[BB] No, I suggest:

(initial RTT from the 3WHS) / 8

Depending on the implementation, the RTT smoothing algo sometimes 
bootstraps using prior info as well, such as per-destination cache or 
TFO cookie.

Cheers

Bob

>>>> https://datatracker.ietf.org/meeting/103/materials/slides-103-tcpm-sessa- 
>>>>
>>> l4s-rack-00
>>>>
>>>>   experimental RFCs) would be carried out without any transport 
>>>> protocol
>>> code.

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 22 Feb 2019 09:35:25 +0000

Lloyd,

Yes, I'm v much aware of the attacks in that paper, incl. the spoofed 
DupACK attack. But I hadn't written them in as additional motivation, 
cos I'm trying to keep this draft's scope to requirements for interop.

I'll mention this in the informational appendix tho, and I guess I could 
point to it as an additional benefit under security considerations.

Thanks.

Bob

On 18/02/2019 12:36, lloyd.wood=[email-masked] wrote:
> there's the support for the MUST NOT, at least. I's not the lack of 
> scale that is the major issue there, it's being open to abuse. ack 
> counting didn't work...
>
>     https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05
>     <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05 >)
>     regarding

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 22 Feb 2019 10:15:38 -0800

On Thu, Feb 21, 2019 at 2:16 AM Bob Briscoe <[email-masked]> wrote:
> 1/ Let me explain why I would still argue for MUST, even tho it strictly rules out existing DCTCP (cos of its initial use of 3DupACK).
>
> Existing DCTCP does not comply with other requirements in the draft either. The requirements started life as a list of things to change in DCTCP. DCTCP has always been put forward as an interim solution that is sufficiently close to the requirements that it can be used for initial testing / demonstration.
>
> The draft is experimental track. It states how the code of all L4S senders has to behave if you want to get meaningful results from an experiment. For an experimental draft, I would argue against saying "You SHOULD do X", if actually you MUST do X before you can expect to get the results claimed by the draft.
>
> Nonetheless, this discussion has helped me see a deeper problem with this list of L4S congestion control requirements. I think this is the root cause of David's concern.
>
> The time-units requirement is about inter-operability with any links that do not maintain strict ordering. It is not about safe co-existence with other traffic. That's true for some of the other requirements too. So I propose that we split up the list into:
>
> those requirements that really are about safe coexistence with other traffic
> those requirements that are about interoperability with other parts of the L4S experiment (where safety is too strong a word)
>
> Then code that does not meet any safety requirement cannot be used safely on the public Internet. Whereas code that doesn't interoperate with the rest of the L4S experiment can still be used on the Internet, it just won't necessarily give meaningful results in an L4S experiment.
>
> For instance "MUST react to packet loss in a way that will coexist safely with a TCP Reno" is a safety requirement. So a DCTCP that does not satisfy this requirement cannot safely be used on the Internet For instance, Windows DCTCP is safe on the public Internet, but current mainline Linux DCTCP is not because there's bug where it intends to fall back to Reno - here's Koen's patch from 2yrs ago.
how so? I believe Linux DCTCP correctly implements
https://tools.ietf.org/html/rfc8257#section-3.5

AFAIK Koen's patch makes DCTCP reduces cwnd by 75% (instead of 50%)
when a loss happens w/o any prior CE marks (in a round trip).

Could you identify which case you're referring to
1) DCTCP RFC has a design issue
2) Linux implementation of DCTCP RFC has a bug

> should also adjust to the reordering extent - the current wording of L4S seems to preclude any enhancement along those lines to qualify for use of L4S, even thoug a dynamic measurement of the reordering extent would implicitly have some time component as measured, correct?
>
> [BB] If an approach implicitly uses time, surely that satisfies a requirement to use units of time?
>
>
>
> Also, for the proposed RTT/8, what would be a default starting value for RACK? 1 sec / 8 = 125 ms?
>
> [BB] No, I suggest:
>
> Depending on the implementation, the RTT smoothing algo sometimes bootstraps using prior info as well, such as per-destination cache or TFO cookie.
>
> Cheers
>
>
> Bob
>
>
>
> Best regards,
>   Richard
>
> We (TSVWG chairs) are looking for "running code" ... as we're having
>
> difficulty
>
> figuring out how the L4S experiment (the L4S drafts are intended to
>
> become
>
>   experimental RFCs) would be carried out without any transport protocol
>
> code.
> From: tsvwg [mailto:[email-masked]] On Behalf Of Yuchung
>
> Cheng
>
> turned into a "SHOULD" guidance?  Given that we expect reordering to
>
> be
>

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 22 Feb 2019 19:03:36 -0500

> scale that is the major issue there, it's being open to abuse. ack 
I think it's only sort-of related, but not really.

The tricks described there are ways to get a sender to be overly 
aggressive if it doesn't implement ABC (or other mitigations) so is 
growing a congestion window based on units of packets rather than 
bytes.  The L4S requirement here is about retransmission / loss 
detection though (reducing the window) happening based on units of time 
vs packets (rather than bytes vs packets).

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 Feb 2019 17:58:16 +0000

Yuchung, [also adding Olivier who was about to try to repost Koen's 
patch to Linux netdev]

I'm saying the Linux implementation of DCTCP RFC has a bug (not a design 
issue).
[So this fork of the thread probably ought to move to a more appropriate 
list, given it's Linux-specific.]

I wasn't aware of any problem with Koen's patch reducing by 75% instead 
of 50%.
https://www.spinics.net/lists/netdev/msg420916.html
Are you saying you've tried the patch and it reduces by 75%, or you've 
looked at the code and you think it will reduce by 75%?

Bob