Subject: [tsvwg] L4S operational guidance draft

From: Wesley Eddy <[email-masked]>
Date: Mon, 16 Nov 2020 13:31:31 -0500

In today's meeting, it was mentioned that the L4S operational guidance 
draft is part of the "safety case" for advancing other L4S drafts.  It's 
not as mature as the other docs yet (e.g. there are several"TODO" 
portions), but we want to understand if people think it's going in the 
right direction and will be suitable for this purpose as it's sketched, 
or what else should be added/changed/etc.

This is on the agenda for Wednesday's meeting.

So, if you have bandwidth, please check out the L4S operational guidance 
draft prior to the Wednesday meeting, so that we can get a better sense 
of how to proceed:

https://datatracker.ietf.org/doc/draft-white-tsvwg-l4sops/

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 16 Nov 2020 22:02:24 +0100

Well, since you asked...

I believe that in parts this document still is way to enthusiastic, eg.:

"4.2 Configure Non-Coupled Dual Queue

   Equipment supporting [RFC3168] may be configurable to enable two
   parallel queues for the same traffic class, with classification done
   based on the ECN field.

   Option 1:

   o  Configure 2 queues, both with ECN; 50:50 WRR scheduler

   o  Queue #1: ECT1 & CE packets - Shallow immediate AQM target

   o  Queue #2: ECT0 & NotECT packets - Classic AQM target

   o  Outcome

      *  n L4S flows and m long-running Classic flows

      *  if m & n are non-zero, get 1/2n and 1/2m of the capacity,
         otherwise 1/n or 1/m

      *  never < 1/2 each flow's rate if all had been Classic

   This option would allow L4S flows to achieve low latency, low loss
   and scalable throughput, but would sacrifice the more precise flow
   balance offered by [I-D.ietf-tsvwg-aqm-dualq-coupled]. "

One of the big ticket items for L4S IMHO is exactly that the dualQ does a terrible job at maintaining a "more precise flow balance" between the different queues. I understand that this document will be mostly written by L4S proponents and hence be positive towards L4S, but unless we call a spade a spade, this will not actually help real world operators to make the L4S experiment any safer.

"4.1 Configure AQM to treat ECT1 as NotECT

   If equipment is configurable in such as way as to only supply CE
   marks to ECT0 packets, and treat ECT1 packets identically to NotECT,
   or is upgradable to support this capability, doing so will eliminate
   the risk of unfairness."

This should present at least some research whether existing deployed RFC3168 AQMs actually allow that level of configuration, because otherwise this leads to unrealistic scenarios like expecting the network operator to bleach ECT(1) before AQM action and restoring it after the AQM is done with no real proposal where to store the state in the interim. In short that seems like a rather theoretical proposition, until and unless it has been shown that deployed AQM can be trivially configured as proposed. If is is realistic, by all means add it, but if this is as purely a theoretical option as it appears to me the draft would be better without presenting this option.

I certainly miss the "what to do on catastrophic failure" section, that explains that ISPs making special amends for the L4S experiment can follow to un-do all of this when it becomes clear that the experiment failed (when not if...))

Best Regards
	Sebastian

> On Nov 16, 2020, at 19:31, Wesley Eddy <[email-masked]> wrote:
> 
> In today's meeting, it was mentioned that the L4S operational guidance draft is part of the "safety case" for advancing other L4S drafts.  It's not as mature as the other docs yet (e.g. there are several"TODO" portions), but we want to understand if people think it's going in the right direction and will be suitable for this purpose as it's sketched, or what else should be added/changed/etc.
> 
> So, if you have bandwidth, please check out the L4S operational guidance draft prior to the Wednesday meeting, so that we can get a better sense of how to proceed:
>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 17 Nov 2020 00:11:10 +0200

> On 16 Nov, 2020, at 8:31 pm, Wesley Eddy <[email-masked]> wrote:
> 

My impression is as follows:

1: The draft is clearly incomplete.  Even if it is filled in tonight or tomorrow, we will not have time to evaluate it in a completed state before Wednesday.  That seems to be acknowledged, but I want it to be crystal clear in advance.

2: Much of the draft is about preparing a network specifically to handle L4S traffic.  The need to do this is a well-established weakness of L4S.  Unfortunately, endpoints cannot reliably determine whether the path they wish to communicate across consists entirely of L4S-aware networks.  Even after they begin communicating, the network does not unambiguously inform them of this fact, either positively or negatively.

3: The rest of the technical requirements in the draft talk about what happens when L4S traffic encounters the boundary of the specially-prepared network, and how to get it between two such networks without exhibiting non-standard behaviour while traversing the standard network in between.

I will not comment on the likely sufficiency of these measures, in both #2 and #3 categories above.  Instead I would like to highlight the basic intent of the draft, which appears to be as follows:

*** L4S traffic MUST be confined to networks that have been specifically prepared to handle it. ***

But this is the same situation that DCTCP has been in from the start, and which L4S was supposed to fix.  The only advantage that L4S now brings to the table is that AIAD traffic is identifiable as distinct from AIMD traffic, and can thus be directed into a separate queue with a specially tuned AQM.  That could have been the work of seven hours with mostly existing tools, not seven years.

Also, if L4S traffic is indeed confined to specially prepared networks, then one of those preparations could be to ensure that a DSCP identifying L4S traffic will survive end-to-end.  This would obviate the main reason for choosing ECT(1) as that identifier, and free up ECT(1) for other proposals that do not sacrifice interoperability on the public Internet.

If I am wrong about the above, then I would appreciate a clarifying discussion on the matter.  Otherwise, the above is the position I'm likely to take on Wednesday.

 - Jonathan Morton

---

From: <[email-masked]>
Date: Tue, 17 Nov 2020 14:17:46 +0000

Hi Wes,

Done. The draft requires more text (the authors left marks themselves, so that isn't news). 

  5.  Researchers

  5.1.  Detection of Classic ECN FIFO Bottlenecks

     TODO: Describe active testing methods, in-band or out-of-band, that
     can distinguish FIFO from FQ.

I'd appreciate a requirements definition of that test. What is desired regarding properties not directly linked to what is called "classic ECN" and FQ. Desired test duration, applicability for a network section (I'd guess that access links are more important), properties like whether controlled bottleneck congestion is required (I think, it is), and suggested test functionality in the presence of background traffic (the test shouldn't disturb active application flows).
I personally would also prefer references, so that active measurement experts without ECN know-how have a chance to understand what's required for such a test without having to search and read numbers of volumes. I'm not out for a discussion on the list (more text in the draft, please). 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von Wesley Eddy
Gesendet: Montag, 16. November 2020 19:32
An: [email-masked]
Betreff: [tsvwg] L4S operational guidance draft

In today's meeting, it was mentioned that the L4S operational guidance draft is part of the "safety case" for advancing other L4S drafts.  It's not as mature as the other docs yet (e.g. there are several"TODO" 
portions), but we want to understand if people think it's going in the right direction and will be suitable for this purpose as it's sketched, or what else should be added/changed/etc.

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 17 Nov 2020 16:07:11 +0000

Hi tsvwg,

Thanks Greg for writing this doc.  I think it's a good step
forward and headed in the right direction.

My main interest at this stage is making sure that L4S won't
break the rollout for regular/classic ECN.

To that end, in light of the support I saw from several in
the WG for moving the L4S experiment forward, what I'd like
to see landed before docs start going to last call is a
normative requirement on L4S senders to avoid sending L4S
traffic with ECT(1) set unless they have good evidence that
there are not classic marking bottlenecks on-path.

I think this doc makes a good start at describing the
justification for that requirement and listing the kinds of
systems and deployment models that can meet that bar of
"good evidence".

And I think it's a good choice to write it up in a separate
doc like this, so that it can be separately updated to
document other kinds of reasonable evidence, or evolving
observations that would have an impact on these
considerations.

So my most significant bit of feedback is +1 for this doc, I
think it moves the discussion in the right direction in a
helpful way.

A couple of refinements I'd like to see, to give a general
sense of how I'd like to see this doc evolve from its
promising start:

- I'm not sure FQ classic ECN should get quite as big a pass
  here as this doc gives it.  The fact that L4S builds a
  standing queue to a classic marking bottleneck's threshold
  is a problem in itself, but also there is some likelihood
  of hash collisions when there are multiple flows, and those
  are problematic for the colliding competing flows the same
  as you get with a shared queue, even though the smaller
  scope of collision limits the damage.

  I'd be ok with outlining these problems, noting they're
  less severe than the shared queue problem, and allowing for
  more flexibility than the harder requirement that's needed
  for avoiding shared marking queues (like being more
  permissive with how recent or compelling the evidence
  should be, since the harm is more constrained).  But I don't
  think classic marking FQ queues should be treated as a non-
  problem.

- I'd like to see either references to good representative
  samples of the published tests that demonstrate the described
  unfairness issues, or failing that some somewhat stronger
  wording that doesn't just abstractly mention 'unfairness' and
  sum up the scope of the issue as "could be a cause of
  concern".

  Something like "The L4S response to CE is not compatible with
  classic ECN marking." seems to me a more appropriate summary
  description of this issue.  But I think just giving a good
  several references that quantitatively demonstrate the scale
  of the problem seems better than quibbling over exactly how
  to describe it.

- I also have a slight concern over this doc being
  informational.

  I'm not sure whether informational docs are ever updated, but
  I don't think I've ever seen it done, and it seems to be a
  possible necessity over the next years to update guidance from
  this doc as we see how the L4S experiment unfolds, or as the
  research recommended in this doc moves forward and perhaps
  solves the classic queue detection problem, or as we learn more
  about the deployment of classic ECN (whether it takes off or
  peters out or continues its incremental growth).

  Is it possible and reasonable to incorporate this as a
  normative part of the experiment and make it experimental,
  instead of just informational?

As the doc gets closer to complete, hopefully I'll be able to
come back to it and give a full review at some point, but I think
these points capture my initial high level reaction to the -01
version of the draft.

Regardless, I would say WG adoption of this doc sounds useful to
me, and that this is on the right track for providing the
resolution we've discussed to my outstanding concerns with the L4S
experiment wrt its interactions with classic queues.

- Jake

On 11/16/20, 10:31 AM, "Wesley Eddy" <[email-masked]> wrote:

https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-white-tsvwg-l4sops/__;!!GjvTz_vk!Axw2CdE9R-Px4wskXijeNFQOgQA6FLFUIyrRhvM67g7NWS1bs6Fc23ELKOixjuw$

---

From: Pete Heist <[email-masked]>
Date: Tue, 17 Nov 2020 21:04:42 +0100

Hi Jake/Wesley, just one add below...

>   think classic marking FQ queues should be treated as a non-

One addition to this is the fact that when tunneled traffic traverses
fq_codel, since the 5-tuple for all of the tunnel's inner flows is the
same, that places them in one fq_codel queue, leading to the same
safety problem when L4S and non-L4S flows meet. So, while single queue
RFC3168 AQMs or hash collisions are one way for that to happen,
tunneled traffic may be a more likely way. We recently added a test
scenario using Wireguard through fq_codel to cover that case, which for
some reason I didn't think to try earlier:

https://github.com/heistp/l4s-tests/#unsafety-in-tunnels-through-rfc3168-bottlenecks

That should probably be covered in the guidance in relation to FQ
classic ECN as well...

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 18 Nov 2020 02:20:37 +0000

Hi Pete,

> That should probably be covered in the guidance in relation to FQ

Thanks Pete, I 100% agree.  I knew I was forgetting something.

I meant to include exactly that sentiment and link, and I thought it
was a very helpful point when you posted it a few weeks ago, but several
hundred messages later it slipped my mind while writing this up.  Thanks
for getting it in there.

Best regards,
Jake

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 18 Nov 2020 06:44:04 +0000

Hi Jake and Pete + others

I try to understand this input, thanks for raining this BTW. 

I interpret this as: 
1) Flow queue AQMs don't work properly across tunnels, as flow isolation based om 5-tuple does not work
2) L4S flows can potentially (or will) get an unfair share as a consequence of #1 above because the bootleneck only implement a RFC3168 ECN marking. 

Is this understanding correct ?

/Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Holland, Jake
> Sent: den 18 november 2020 03:21
> To: Pete Heist <[email-masked]>; Wesley Eddy <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [tsvwg] L4S operational guidance draft
> 
> Hi Pete,
> 
> > scenario using Wireguard through fq_codel to cover that case, which
> > for some reason I didn't think to try earlier:
> >
> > https://protect2.fireeye.com/v1/url?k=aaf3845d-f568bd70-aaf3c4c6-
> 8692d
> > c8284cb-99cc9ade2897f933&q=1&e=b25cd448-5648-4fc4-8858-
> a85d6c2df9b0&u=
> > https%3A%2F%2Fgithub.com%2Fheistp%2Fl4s-tests%2F%23unsafety-in-
> tunnels
> > -through-rfc3168-bottlenecks
> >
> I meant to include exactly that sentiment and link, and I thought it was a very
> helpful point when you posted it a few weeks ago, but several hundred
> messages later it slipped my mind while writing this up.  Thanks for getting it in
> there.
> 
> Best regards,
> Jake
>

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 18 Nov 2020 08:55:45 +0200

> On 18 Nov, 2020, at 8:44 am, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

Essentially yes.

The main assumption is that the AQM is not L4S-aware, ie. treats ECT(1) as meaning the same as ECT(0) as per RFC-3168.  Hence it will apply the same marking rate to all packets in the tunnel, which externally appears to be a single flow and so does not receive the benefit of FQ or AF mechanisms.  If some of those packets belong to L4S AIAD flows while others are conventional AIMD flows, the AIAD flows will respond much less and dominate the AIMD flows.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 08:21:26 +0100

Hi Ingemar,

>isolation based om 5-tuple does not work

         Another, less negative, interpretation is that tunnels, that multiplex multiple flows into one externally visible flow are treated and recognized only as that flow. Which for things like encrypted VPNs is exactly the behavior that seems appropriate, no?

Best Regards
          Sebastian

>2) L4S flows can potentially (or will) get an unfair share as a
>consequence of #1 above because the bootleneck only implement a RFC3168
>ECN marking. 
>
>was a very

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 18 Nov 2020 07:24:01 +0000

Hi Ingemar,

> Is this understanding correct ?

Yes, that's the issue.

The point being that FQ has the same troubles as shared queue for traffic
that shares a tunnel, so FQ on its own is not enough of a mitigation for
that use case, but the -01 version of the operational guidance draft
doesn't really cover that point yet.

Best regards,
Jake

---

From: Ingemar Johansson S <[email-masked]>
Date: Wed, 18 Nov 2020 07:26:50 +0000

OK, then I understand. Thanks!
/Ingemar

> From: Holland, Jake <[email-masked]>
> Sent: den 18 november 2020 08:24
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Pete Heist
> <[email-masked]>; Wesley Eddy <[email-masked]>
> Cc: [email-masked]; Ingemar Johansson S
> <[email-masked]>
> On 11/17/20, 10:44 PM, "Ingemar Johansson S"
> <ingemar.s.johansson=[email-masked]> wrote:
> > I interpret this as:
> > 2) L4S flows can potentially (or will) get an unfair share as a consequence of
> #1 above because the bootleneck only implement a RFC3168 ECN marking.
> >
> The point being that FQ has the same troubles as shared queue for traffic that
> shares a tunnel, so FQ on its own is not enough of a mitigation for that use
> case, but the -01 version of the operational guidance draft doesn't really
> cover that point yet.
> 
> Best regards,
> Jake
>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Nov 2020 08:31:18 +0100

> On Nov 18, 2020, at 08:24, Holland, Jake <jholland=[email-masked]> wrote:
> 
> Hi Ingemar,
> 

	[SM] Sidenote: t is quite disappointing that the TODO list contains a note to explain how FQ does not help against sharding instead:

"TODO: also mention that flow sharding is commonplace, so per-flow
   fairness does not imply per-application fairness"

A point (that lacks references for the expected frequency and is) completely irrelevant for the operational considerations for deploying L4S. It is that level of partisanship the permeates many L4S documents that leaves the impression L4S is not about find a technically sufficient solution, but also has an ax to grind...

Regards
	Sebastian