Subject: Re: [tsvwg] plan for L4S issue #29

From: Sebastian Moeller <[email-masked]>
Date: Thu, 1 Oct 2020 09:05:20 +0200

Hi Wes,

thank you, more below in-line.

> On Oct 1, 2020, at 05:03, Wesley Eddy <[email-masked]> wrote:
> 
> Replying to your question to the chairs:
> 
> On 9/30/2020 4:00 AM, Sebastian Moeller wrote:
>> 	[SM] I believe that we are conflating two "experiments" here:
>> 
>> 1) the absolutely required experiments to what degree L4S will realize its claimed charcteristics (or its promises) under a number of conditions that are relevant/prevalent in the real world.
>> 
>> 2) how to deploy L4S in the real world.
>> 
>> If we look carefully at these two experiments it becomes obvious, that an RFC in the experimental track seems necessary for 2), but 1) can and should proceed long before 2) is being addressed. To be blunt, if experiment 1) is not clearly demonstrating improvements over the state of the art, then 2) becomes moot.
>> 
>> @CHAIRS: Could you please let me know, if you agree?
> 
> 
> We've heard very strongly from the working group that enabling low-latency is very important to a wide number of stakeholders right now.  Several teams have been experimenting with this, and it's been reported that the industry is actively working on L4S implementations.

	[SM] I heard the same claims, but I am dubious about how much more than a wish for low latency this WG feeling conveyed. I asked repeatedly on this list for data demonstrating L4S keeping its promise over rather normal internet conditions and did not even get a "we have internal confidential data, that convinced us, but that we are not permitted to share" response, but instead nothing at all. I wonder what value the fact "industry is worlking on it" has, if the same industry developers can not easily produce data answering the "does it work under the to be expected conditions, is it safe, is it robust & reliable" triple-question. 

> 
> We also have clearly heard a number of people that are so far unconvinced of the utility or capabilities in their analysis.

	[SM] This is a somewhat distorted way of phrasing this issue. I have repeatedly asked for data and since none has been forthcoming I am forced to assume, that all the people you categorize as "convinced" in the last sentence can at best BELIEVE in L4S capabilities, and I am certain that in science and engineering a believe can be a decent starting point, but never can substitute real data/evidence. All I am asking for is that the WG makes sure to look at real data first instead of making a hard to revoke decision based on hope/wishful thinking/believe... 

>  I think this shows that the operational considerations work that Greg has started editing is important.  It can help people experimenting with methodology for collecting data and having appropriate risk awareness and management plans.

	[SM] As I said before, none of this is a prerequisite for doing internet-size experiments first (size, not scale, I am not asking team L4S to simulate the whole internet, but rather to convincingly demonstrate that L4S keeps its promises over more than the currently over-tested short RTT links, and also assess the side-effects it causes over such internet-style paths).

> 
> So, from my viewpoint as a chair, I don't think the working group seems to want to block right now on convincing every single person to love L4S,

	[SM] This would be a decent argument, if I would measure L4S against some hypothetical magic solution, but I am constantly trying to compare L4Ss implementation against the claims in the drafts in the implementation so far always falls way short of the promises. Expecting L4S to be loved by everybody seems over the top, agreed. At least making sure its RFCs describe its reality however seems rather essential, unless I misunderstood the goal behind the internet engineering task force completely...

> but rather to make sure that it's setup to be sufficiently safe for the wider experimentation plans that we've heard of in prior meetings.

	[SM] Before this de-rails any further into abstract concepts, could you please answer a few questions:

1) Why you consider it essential for these safety tests that the L4S drafts reach RFC status first? Put differently, what unsurmountable obstacles exist today that make it impossible for the required safety tests to be performed right now?

2) And on the topic of failure, will the WG commit to a set of criteria up-front to assess whether a potential L4S experiment succeeded or failed?

3) What to do in the case of failure? Do you consider it out of scope for the WG to require the L4S drafrs ot include explicit section how to un-roll the experiment in case of failure and the expected longer-term side-effects? If no, pleas elaborate why?

Best Regards
	Sebastian

---

From: Pete Heist <[email-masked]>
Date: Thu, 01 Oct 2020 14:13:27 +0200

On Tue, 2020-09-29 at 22:48 +0000, Greg White wrote:
> 
> In my opinion, Issue 29 can be closed. RFC3168 detection should
> continue to evolve and should be referenced in the Operational
> Guidance draft, but fallback should not be required in the
> experimental protocol drafts for the reasons outlined in Issue 29,
> and on the mailing list.

In my opinion, the issues with bottleneck detection and fallback should
be fixed, since this was the proposed way to provide safety for the
alternate meaning of CE being introduced.

The question of whether or not to proceed with these issues still open
sounds like a separate one, but it seems like closing issues without
fixing them would make it hard for someone who wanted to review and
understand what issues are still open.

Pete

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 1 Oct 2020 13:29:04 +0100

> understand what issues are still open.
To be clear, is there a "safety" concern with respect to congestion 
collapse from overload, or is it an issue concerning relative fairness 
between flows?

Gorry

Gorry

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 1 Oct 2020 14:42:32 +0200 (CEST)

> collapse from overload, or is it an issue concerning relative fairness 

I don't know where the cut-off is between these, but my worry is that one 
type of flow might only get few single digit percent of available bw which 
at slower connections (few megabit/s range) might mean the application 
using these flows might become unusable.

We've seen this in some tests already, at that time I believe it was 
during BBR development.

https://lists.bufferbloat.net/pipermail/cerowrt-devel/2016-September/010861.html

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 1 Oct 2020 13:54:48 +0100

On 01/10/2020 13:42, Mikael Abrahamsson wrote:
>> collapse from overload, or is it an issue concerning relative 
>> fairness between flows?
>
> I don't know where the cut-off is between these, but my worry is that 
> one type of flow might only get few single digit percent of available 
> bw which at slower connections (few megabit/s range) might mean the 
> application using these flows might become unusable.
>
Right, so I'd agree that stravation of sharing flows is an important 
case - and I appreciate the work done behind the scenes to make sure 
that BBRv2 didn't replicate this sort of problem.

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 1 Oct 2020 16:08:33 +0300

> On 1 Oct, 2020, at 3:42 pm, Mikael Abrahamsson <swmike=[email-masked]> wrote:
> 
>> To be clear, is there a "safety" concern with respect to congestion collapse from overload, or is it an issue concerning relative fairness between flows?
> 
> I don't know where the cut-off is between these, but my worry is that one type of flow might only get few single digit percent of available bw which at slower connections (few megabit/s range) might mean the application using these flows might become unusable.

I think "congestion collapse" would be characterised by the *total* goodput of the network falling off a cliff.  Absent the sort of implementation bug that we uncovered in Spring this year, I don't think we're very likely to see that - though I think we should all be concerned that it took third-party testing to uncover that problem.  IMHO it points to inadequate testing by the L4S team themselves, and I would sincerely hope that they've taken steps to remedy that by now.

The major practical concern is with relative fairness.  A ratio of maybe 2-3x goodput between flows with different congestion controls is probably tolerable, especially since that is already accepted for CUBIC versus Reno in the regimes where Reno is suboptimal.  When that increases to 10x, concerns should be raised.  Meanwhile slides presented repeatedly by the L4S team during IETF sessions show a 16x ratio over some range of conditions, with the implication that this is normal, expected behaviour for L4S versus CUBIC.  Certainly it's about what is expected from a theoretical analysis of the dynamics, and experience with DCTCP which has essentially the same response to CE marks.

By all means, put me in the camp where accepting a 16x goodput penalty to existing traffic when an L4S flow starts up is unacceptable.  The "classic detection heuristic" was presented as a solution to this problem, but I think we can agree that it did not succeed.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 1 Oct 2020 15:16:24 +0200

Hi Jonathan,

let me chime in below in-line

> On Oct 1, 2020, at 15:08, Jonathan Morton <[email-masked]> wrote:
> 
> The major practical concern is with relative fairness.  A ratio of maybe 2-3x goodput between flows with different congestion controls is probably tolerable, especially since that is already accepted for CUBIC versus Reno in the regimes where Reno is suboptimal.  When that increases to 10x, concerns should be raised.  Meanwhile slides presented repeatedly by the L4S team during IETF sessions show a 16x ratio over some range of conditions,

	[SM] This seems driven by the fact that according to the RFC the dual queue coupled AQ actually is set up with a 1:16 or 1:15 worst case split between the L4S and the non-L4S queue. The idea seems to be to have that as a backstop to avoid "starvation", but only for very peculiar definition of starvation. IMHO differences >= 1:8 over the expected equitable share are clear cut cases of starvation and need to be avoided. 
	The 1:16 rationale falters pretty quickly, because on a massively overloaded link this still does not really guarantee any flow forward progress and simply does not work at low absolute link rates... I note that this is at the core of the whofully under-discussed issue #28...

> with the implication that this is normal, expected behaviour for L4S versus CUBIC.  Certainly it's about what is expected from a theoretical analysis of the dynamics, and experience with DCTCP which has essentially the same response to CE marks.
> 

	[SM] +1

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 1 Oct 2020 16:24:57 +0300

> On 1 Oct, 2020, at 4:16 pm, Sebastian Moeller <[email-masked]> wrote:
> 
> This seems driven by the fact that according to the RFC the dual queue coupled AQ actually is set up with a 1:16 or 1:15 worst case split between the L4S and the non-L4S queue. The idea seems to be to have that as a backstop to avoid "starvation", but only for very peculiar definition of starvation. IMHO differences >= 1:8 over the expected equitable share are clear cut cases of starvation and need to be avoided. 

I suppose we also have to distinguish between the behaviour when the bottleneck is L4S enabled, and when it is not L4S enabled but is ECN enabled, and also when neither L4S nor ECN are enabled.  A robust solution would be one that behaves well in all three cases.

Presently, the 1:16 ratio approximately applies in both the first two cases (but for different reasons) when the baseline path length is short, and is only avoided in the last one where both conventional and L4S flows exhibit an AIMD response to losses.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Thu, 01 Oct 2020 15:35:40 +0200

On Thu, 2020-10-01 at 13:29 +0100, Gorry Fairhurst wrote:
> fairness 

False detection (this issue #29) is by itself less serious directly
than missed detection, because the failure mode is safe interaction
with RFC 3168 flows. However, false detection events become an
incentive to disable detection, so this issue is still part of a chain,
unless detection and fallback is required.

There's a plot over in issue #16 of one example of what happens when
RFC 3168 flows and L4S flows wind up in the same Codel queue (however
that happens) either without detection and fallback enabled, or if a
missed detection occurs:

https://trac.ietf.org/trac/tsvwg/ticket/16

Steady state throughput in that case appears to be about 44Mbps for the
L4S flow and 4Mbps for the RFC 3168 flows. Someone could do more
testing or predictions on how things change under different network
conditions, but a ratio of at least 16:1 is probably possible somehow,
since that was shown in some earlier slides.

In my opinion, approximate fairness is ideal, a result of 2-3x under
some conditions may still be tolerable, and anything beyond that isn't
acceptable. I remember one definition of "starvation" as "you know it
when you see it". Personally, a 10:1 throughput ratio already meets
that definition for me, so qualifies as "unsafe".

Pete

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 1 Oct 2020 15:37:01 +0200

> On Oct 1, 2020, at 15:24, Jonathan Morton <[email-masked]> wrote:
> 

[SM] Good point, the last case however seems genuinely irrelevant to assess L4S' safety, unless we want to go through all of this only to keep the novel low latency mode disabled forever. IMHO especially failure mode 1 is a showstopper, this is L4S failing to keep its promises for the one use-case it was designed for short RTT CDN-to-end-user paths.

Best Regards
	Sebastian

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 1 Oct 2020 11:07:37 -0400

> understand what issues are still open.

I agree.  Open issues,  research questions, and future work are often 
explained in a section of the Exp. RFCs.  This is better than just the 
tracker, since it's right inside the published document.  That has been 
something we've done in the past, like in Section 7 of RFC 8290 and 
Section 7 of RFC 8033, as pertinent examples.  I think similar should be 
expected in this case.  I don't think Section 6 of the ecn-l4s-id draft 
is yet as detailed as I'd hope for yet this regard.

Adding description of this issue to an appropriate place in the document 
suite sounds like a good next step.  The issue is understood and people 
doing implementation & deployment seem to be accepting this condition 
for now, but would probably want to improve it.

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 1 Oct 2020 11:33:02 -0400

I am not sure if your questions are for me or others, but responding to 
the specific questions:

> 1) Why you consider it essential for these safety tests that the L4S drafts reach RFC status first? Put differently, what unsurmountable obstacles exist today that make it impossible for the requir…

I haven't heard anyone state this.  Many people have expressed that they 
are eager for wider experimentation.  I don't know where the word 
'required' is coming from in your question, but it sounds like your own 
requirement rather than something these people agree with.

> 2) And on the topic of failure, will the WG commit to a set of criteria up-front to assess whether a potential L4S experiment succeeded or failed?

I think the operational document Greg has started editing will be 
capturing the collected thoughts on what people working with L4S in 
their networks should be looking for.   But I'm doubtful this is what 
you're asking for, since you seem to be after more of a global pass/fail 
flag on all of L4S, if I understand correctly.  I don't recall seeing 
such in any other Exp. RFC, so it seems unusual to expect.

> 3) What to do in the case of failure? Do you consider it out of scope for the WG to require the L4S drafrs ot include explicit section how to un-roll the experiment in case of failure and the expec…

I think this would be good scope for the operational document. There 
have been some discussions in the working group, and getting the ideas 
matured and into a draft seems like it would be productive.

At higher level, the thread has diverged significantly from issue #29 
subject matter and I think lost that focus.

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 1 Oct 2020 21:31:50 +0200

Hi Wes,

I agree that a topic change was in order. More below in-line.

> On Oct 1, 2020, at 17:33, Wesley Eddy <[email-masked]> wrote:
> 
> I am not sure if your questions are for me or others, but responding to the specific questions:

	[SM] I might not have formulated this clearly enough, the questions are for all, the chairs and interested WG members.

> I haven't heard anyone state this.  Many people have expressed that they are eager for wider experimentation.  I don't know where the word 'required' is coming from in your question, but it sounds like your own requirement rather than something these people agree with.

	[SM] Oh, simply from logic. If the WG makes publication of the L4S dependent on L4S being safe (for the wider internet) as I have taken to understand, the WG needs some criteria to assess safety. And some of these will be hard requirements, "avoid total congestion collapse" comes to mind. I really want to know how the WG is supposed to asses safety, and what criteria the chairs consider. Talking about requiring safety, but not committing to to a definition what is considered safe and what is unsafe can hardly be the WGs final position, so I am clear I am misunderstanding something.

> I think the operational document Greg has started editing will be capturing the collected thoughts on what people working with L4S in their networks should be looking for.   But I'm doubtful this is what you're asking for, since you seem to be after more of a global pass/fail flag on all of L4S, if I understand correctly.  I don't recall seeing such in any other Exp. RFC, so it seems unusual to expect.

	[SM] Most experiments reqire the investment of a rare resource like a codepoint in the shared IPv4/6 header either, so maybe a bit more contingency planning might be in order for L4S than for your run of the mill RFCs, no?

> I think this would be good scope for the operational document. There have been some discussions in the working group, and getting the ideas matured and into a draft seems like it would be productive.

	[SM] Great! I might be confused about the hierarchy of RFCs though, I was under the impression that standard > experimental > informational. But if all published ones apply equally, than I do not care too much where the "how to end the experiment" section (for both success and failure) will live. I assume that all the L4S drafts/RFCs will cross-reference each other. ATM, draft-white-tsvwg-l4sops-00 seems to exist completely outside of the other core L4S drafts, but that might change on the next update cycle?

> 
> 
> At higher level, the thread has diverged significantly from issue #29 subject matter and I think lost that focus.

	[SM] Strictly speaking I agree, and changed the topic (not sure whether this will survive due to email headers).

Best Regards
        Sebastian

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Fri, 2 Oct 2020 10:13:35 -0700 (PDT)

Comment at end.

> I agree.? Open issues,? research questions, and future work are often 
> explained in a section of the Exp. RFCs.? This is better than just the 
> tracker, since it's right inside the published document.? That has been 
> Section 7 of RFC 8033, as pertinent examples.? I think similar should be 
> expected in this case.? I don't think Section 6 of the ecn-l4s-id draft 
> suite sounds like a good next step.? The issue is understood and people 

I do not accept that as a given.  If the people doing implementainos &
deployment where even aware of this situation it would not of taken
the testing by Pete Heist to uncover the issue.

With that said, perhaps documenting this clearly in the drafts would
at least raise the awareness of this issue with L4S for its proponents.

-- 
Rod Grimes                                                 [email-masked]