Subject: [tsvwg] L4S/3168 Coexistence

From: Martin Duke <[email-masked]>
Date: Mon, 10 May 2021 09:49:41 -0700

(As an individual)

I have not been able to follow these threads as well as I'd like. Assuming
I'm not the only one, I would like to confirm that the current state of the
coexistence discussion is that L4S will create problems for "classic flows"
under the following conditions:

(1) There are 3168 routers in the path (order of 1%)
(2) These 3168 routers implement shared queues (no known examples, but it
might happen), AND
(3) The heuristics result in a false negative (<<1%?)

On the face of it (1% * 1%? * <<1%?) seems to be a small price to pay in
the aggregate. I hope we'd agree on that! So the argument against, ISTM,
would rest on two things:

(A) These failures are not uniformly distributed, and there is some
non-negligible use case that will suffer grievously. It would be very
helpful to me if someone could *concisely state* what this use case is and
where it exists. Nothing in the congestion control space is provably
optimal across all parameters and generalized fear that bad parameters
might exist somewhere is IMO not constructive.

(B) Because you can't have a vague MUST, we aren't REQUIRING any particular
limit to false negatives. If experiment participants are going to agree to
do something, I wonder if we can't converge on some words here that prevent
really dumb stuff?

Maybe I have this wrong! The debate is quite hard to follow.

Martin

---

From: Greg White <[email-masked]>
Date: Tue, 11 May 2021 03:26:08 +0000

Hi Martin,

I think you’ve mostly got that right.  The clarifications I’d make to your 3 conditions are:

2a) OR, the router implementation is a non-L4S-aware FQ (all known examples of 3168), and it’s carrying a tunnel with mixed L4S/Classic traffic (??%)

4) AND, a long-running (multiple seconds), capacity-seeking L4S flow exists in the queue at the same time as a Classic flow

But, the current heuristics result in more false positives than we’d like, and introduce complexity that we’d like to avoid where possible.
So, L4Sops says: if you are intending to use L4S for long-running flows, first determine whether the network/path you are using has a likelihood of containing a 3168 bottleneck, then decide whether real-time heuristics and/or fallback are needed.

In Akamai’s data, there were 3 small ASNs (out of ~800) where the apparent prevalence of 3168 was over 20%, another 4 small ASNs where the prevalence was over 10%, and somewhere between 6 and 25 ASNs where the prevalence was over 3%.  So: either avoid using L4S on those networks, or run the heuristics with real-time fallback, or reach out to those networks and let them know what remedies they can implement (listed in L4Sops) to enable L4S in their network.

-Greg

From: tsvwg <[email-masked]> on behalf of Martin Duke <[email-masked]>
Date: Monday, May 10, 2021 at 10:50 AM
To: tsvwg <[email-masked]>
Subject: [tsvwg] L4S/3168 Coexistence

(As an individual)

I have not been able to follow these threads as well as I'd like. Assuming I'm not the only one, I would like to confirm that the current state of the coexistence discussion is that L4S will create problems for "classic flows" under the following conditions:

(1) There are 3168 routers in the path (order of 1%)
(2) These 3168 routers implement shared queues (no known examples, but it might happen), AND
(3) The heuristics result in a false negative (<<1%?)

On the face of it (1% * 1%? * <<1%?) seems to be a small price to pay in the aggregate. I hope we'd agree on that! So the argument against, ISTM, would rest on two things:

(A) These failures are not uniformly distributed, and there is some non-negligible use case that will suffer grievously. It would be very helpful to me if someone could concisely state what this use case is and where it exists. Nothing in the congestion control space is provably optimal across all parameters and generalized fear that bad parameters might exist somewhere is IMO not constructive.

(B) Because you can't have a vague MUST, we aren't REQUIRING any particular limit to false negatives. If experiment participants are going to agree to do something, I wonder if we can't converge on some words here that prevent really dumb stuff?

Martin

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 11 May 2021 09:14:51 +0000

Hi Martin and Greg,

I think there’s a few questionable assumptions in those estimates.

One is that regarding point 2 I think there may be reason for concern about FQ paths, as well ass shared queues and vpns.

I tried to expand on those concerns in my last message in the “ECT(1) Flag Day” thread, though I don’t think I met the “concisely” request.  But a specific concern I mentioned was about routinely user-visible video buffering under an ordinary level of daily load.  I’m not certain it would happen, but I’m also not so sure from the available evidence that it won’t.

Also of note, the shared queues and vpns would have severe and persistent problems, whereas the FQ paths would probably be a lot more sporadic.  While that means that retrying a time or 2 would generally be a workaround, it also means (I think?) the problems are less likely to be noticed and acted upon by the monitoring of L4S senders, even if they reach the level of routinely user-visible.  That’s maybe not a real problem if it’s reliably not more often than once/day or 2, but if it’s often as 6-10 times when they watch an hour of video, it would be a big deal if it’s happening to something in the range of 0.01% - 0.1% of all people on most days, IMO.

The other one is regarding point 3 and the estimate of <<1% as the false negative rate.  I guess this comes from the reports at https://l4s.net/ecn-fbk/results_v2.2/ (as referenced by https://arxiv.org/pdf/1911.00710.pdf) and the conclusion that the scenarios they tested did not hit false negatives?

I think the claim on that false negative detection rate was disputed by some of the results from independent testing, so I guess I’d take it with a grain of salt for the moment until I get a better idea of how the tests differ.  I also am concerned that a high false positive rate often translates in practice into turning true positives into false negatives, depending on how the false positives get worked around.

But more importantly I think the detection rate is a different question from the expected rate of persistent unsolved problems affecting end users, which is a better multiplier to use here.

With a robust automatic failover that can avoid user-visible impact, they might be pretty close to the same.  But I find it harder to estimate the likelihood of end user troubles when administrative human intervention is part of the story, and I’d be surprised if it reached “<<1%”, even if the false negative detection rate is that low.  It also troubles me that there might be some possibility of misaligned incentives that could tend to systematically push it toward the high end of what’s possible.

The whole human in the loop aspect of the plan outlined in l4s-id A.1.5 and l4s-ops confuses me a bit on how best to think about it, but taking a face-value low false negative detection rate as the biggest factor in reaching “several orders of magnitude below 1%” is a little hard to swallow at the moment, for me.

HTH.

Best regards,
Jake

From: Greg White <[email-masked]>
Date: Mon,2021-05-10 at 8:26 PM
To: Martin Duke <[email-masked]>, tsvwg <[email-masked]>
Subject: Re: [tsvwg] L4S/3168 Coexistence

Hi Martin,

-Greg

(As an individual)

Martin

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 11 May 2021 13:00:02 +0300

> On 10 May, 2021, at 7:49 pm, Martin Duke <[email-masked]> wrote:
> 
> (As an individual)
> 

Our data suggests it's more like 10% from an Eastern Europe perspective.  If you happen to be Norwegian or French or Mexican, the deployment prevalence in those places is even higher, due to active adoption by major consumer ISPs in those regions.  There is also evidence that deployment is increasing significantly over time, and I think Stuart Cheshire mentioned that he was aware of some impending major deployments.

Some brands of CPE routinely include RFC-3168 AQM as part of their wifi stack and/or "bufferbloat mitigation" features, which are advertised on the packaging.

> (2) These 3168 routers implement shared queues (no known examples, but it might happen), AND

As you note, single-queue RFC-3168 AQMs are RFC-compliant and should therefore be an expected occurrence on the Internet.  There are also various ways that the "flow aware" mechanisms in most of the actually-deployed AQMs can fail to distinguish flows.  I will leave most of that discussion to Jake for now.

> (3) The heuristics result in a false negative (<<1%?)

Said heuristics were disabled by default in the reference implementation after we published data showing they were nowhere near that level of reliability, and have not yet been replaced by a more reliable algorithm.  I refuse to rely on the reliability of something that I can successfully predict and provoke failure modes for.  These algorithms also have not been specified in an IETF document, only in an external white paper - so from a procedural perspective, it could be argued that they are not even part of the specification.

> (B) Because you can't have a vague MUST, we aren't REQUIRING any particular limit to false negatives. If experiment participants are going to agree to do something, I wonder if we can't converge on…

The above combination is also *not* the only scenario in which great harm may occur to conventional traffic.  Within the past week, we were introduced to the Replay Protection Window that is a feature of many VPN tunnels, and quickly realised that this interacted badly with DualQ to the point where an attacker could conduct a full DoS attack on it, even if the target did not use L4S traffic themselves.  See the other thread for details.

In my risk analysis chart, the RFC-3168 interaction is a potential Major category occurrence, while the VPN Replay Window attack is a Serious category occurrence.  Serious is only one step below Catastrophic, and would require correspondingly rigorous measures to ensure that it basically cannot happen.

Such measures are designed into VPNs to guard against a similar attack involving Diffserv (which has existed for a long time by now), but L4S is a new thing and no such protections yet exist.  Additionally, one of the main techniques for dealing with Diffserv (simply not copying the DSCP to the outer header) is not applicable to L4S, which REQUIRES the special ECN codepoint to avoid triggering the other pathology.

> Maybe I have this wrong! The debate is quite hard to follow.

I think some participants in the discussion deliberately try to make it hard to follow, so that misconceptions easily arise in their favour.  Please be on guard against that.

 - Jonathan Morton

---

From: Martin Duke <[email-masked]>
Date: Fri, 14 May 2021 12:10:04 -0700

Thanks all,

This thread has improved my understanding.

On Tue, May 11, 2021 at 3:00 AM Jonathan Morton <[email-masked]>
wrote:

> > I have not been able to follow these threads as well as I'd like.
> Assuming I'm not the only one, I would like to confirm that the current
> state of the coexistence discussion is that L4S will create problems for
> "classic flows" under the following conditions:
> >
> Our data suggests it's more like 10% from an Eastern Europe perspective.
> If you happen to be Norwegian or French or Mexican, the deployment
> prevalence in those places is even higher, due to active adoption by major
> consumer ISPs in those regions.  There is also evidence that deployment is
> increasing significantly over time, and I think Stuart Cheshire mentioned
> that he was aware of some impending major deployments.
>
> Some brands of CPE routinely include RFC-3168 AQM as part of their wifi
> stack and/or "bufferbloat mitigation" features, which are advertised on the
> packaging.
>
> As you note, single-queue RFC-3168 AQMs are RFC-compliant and should
> therefore be an expected occurrence on the Internet.  There are also
> various ways that the "flow aware" mechanisms in most of the
> actually-deployed AQMs can fail to distinguish flows.  I will leave most of
> that discussion to Jake for now.
>
> Said heuristics were disabled by default in the reference implementation
> after we published data showing they were nowhere near that level of
> reliability, and have not yet been replaced by a more reliable algorithm.
> I refuse to rely on the reliability of something that I can successfully
> predict and provoke failure modes for.  These algorithms also have not been
> specified in an IETF document, only in an external white paper - so from a
> procedural perspective, it could be argued that they are not even part of
> the specification.
>
> helpful to me if someone could concisely state what this use case is and
> > (B) Because you can't have a vague MUST, we aren't REQUIRING any
> particular limit to false negatives. If experiment participants are going
> to agree to do something, I wonder if we can't converge on some words here
> that prevent really dumb stuff?
>
> The above combination is also *not* the only scenario in which great harm
> may occur to conventional traffic.  Within the past week, we were
> introduced to the Replay Protection Window that is a feature of many VPN
> tunnels, and quickly realised that this interacted badly with DualQ to the
> point where an attacker could conduct a full DoS attack on it, even if the
> target did not use L4S traffic themselves.  See the other thread for
> details.
>
> In my risk analysis chart, the RFC-3168 interaction is a potential Major
> category occurrence, while the VPN Replay Window attack is a Serious
> category occurrence.  Serious is only one step below Catastrophic, and
> would require correspondingly rigorous measures to ensure that it basically
> cannot happen.
>
> Such measures are designed into VPNs to guard against a similar attack
> involving Diffserv (which has existed for a long time by now), but L4S is a
> new thing and no such protections yet exist.  Additionally, one of the main
> techniques for dealing with Diffserv (simply not copying the DSCP to the
> outer header) is not applicable to L4S, which REQUIRES the special ECN
> codepoint to avoid triggering the other pathology.
>
> I think some participants in the discussion deliberately try to make it
> hard to follow, so that misconceptions easily arise in their favour.
> Please be on guard against that.
>
>  - Jonathan Morton
>
>