Subject: Re: [tsvwg] links to Canary methods for roll-out of new transport features

From: Martin Duke <[email-masked]>
Date: Fri, 13 Aug 2021 10:13:47 -0700

(As an individual)

The "control" part of the canary experiment is how you identify collateral
damage. The L4S hypotheses, informally, are that
(1) L4S flows will experience lower latency, etc
(2) Not-ECT flows, and 3168 flows that traverse most queueing
configurations, will not suffer significant degradation.
(3) 3168 flows will only suffer starvation in queue configurations that are
not deployed at scale in the internet.

So the proper design of a canary deployment, IMO, is in two steps:
(a) Have servers choose between Not-ECT and ECT(0) to establish a baseline.
(b) servers choose between Not-ECT, ECT(0), and ECT(1) and compare the
results.

On Fri, Jul 30, 2021 at 4:24 AM Gorry Fairhurst <[email-masked]>
wrote:

>
> On 30/07/2021 11:21, Jonathan Morton wrote:
> > On Friday, 30 July 2021, Gorry (erg) wrote:
> >> I am not sure though, I think many CC- related topics can have
> potential collateral damage, and we have managed to deploy these gradually
> and improve the transport were necessary. So, what is different here to
> exploring methods such as larger Initial Window, BBR, Hystart, etc.
> > Hystart makes a transport strictly less aggressive, by exiting the
> exponential growth phase early and continuing with much slower linear or
> polynomial growth.  There is no possibility of collateral damage except in
> case of implementation bugs.  This is an excellent use case for canary
> testing, and I would endorse its use with Hystart++.
> >
> > Large IWs do have potential for collateral damage, but the conditions
> that would trigger it (small buffers) result in effects (high loss to IW)
> that are easily noticed by the transport employing it.  This is therefore
> also suitable for canary testing.
> >
> > BBR is not so clear a case.  I am pleased that Google is taking a
> relatively cautious approach to deploying it in an Internet facing context,
> has designed it with standards-track CC coexistence in mind, and seeks to
> improve it when problems are reported and verified.  However, the ECN
> response introduced with BBRv2 is not standards-track compatible, and the
> likely collateral damage when operating in a shared AQM bottleneck is not
> easily noticed by the BBR transport itself, especially when those
> circumstances arise only infrequently.  Canary testing is thus an
> incomplete solution there.
> >
> > The entire problem here is that L4S is likely to cause *externalised*
> collateral damage which is not easily noticed by the L4S transport itself.
> Unless great care is taken to watch for such problems, canary testing will
> therefore fail to find them.
> >
> > What is more, canary testing has as a prerequisite the confidence of
> correct operation and design gained though lab testing.  Lab testing of L4S
> has not given any of that confidence, thus progression to canary testing
> would be inappropriate.
> >
> >   - Jonathan Morton
>
> I was responding to a request to provide references to canary approaches.
>
> Gorry
>
>

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 21 Aug 2021 22:38:25 +0000

Hi Martin,

From: Martin Duke <[email-masked]>
> Date: Fri,2021-08-13 at 10:14 AM
> The "control" part of the canary experiment is how you identify collateral damage. The L4S hypotheses, informally, are that  
> (2) Not-ECT flows, and 3168 flows that traverse most queueing configurations, will not suffer significant degradation.
> (3) 3168 flows will only suffer starvation in queue configurations that are not deployed at scale in the internet.
>
> (b) servers choose between Not-ECT, ECT(0), and ECT(1) and compare the results.

I don't think I understand this suggestion.

I’m not sure if you’re saying when TCP Prague is turned on, the
server would endeavor to actively send simultaneous competing
non-L4S traffic on secondary flows to the same receiver so they
can detect whether there’s impairment in the secondary flows, like
in the “Out of Band Active Detection” case in the Feb 2021
detection paper (https://arxiv.org/pdf/1911.00710.pdf)?  If so,
this is somewhat more complex than most canary systems--from the
sender you don’t generally have a way to open a second connection
correlated with the first (e.g. when a client is making web requests
to a load-balanced service) so it would need a lot of extra cross-
flow coordination features in the server as compared to a usual
canary test.

But I also don’t see how selecting between ECT(0) and Not-ECT when
establishing a baseline is helpful here, since Non-ECT flows will
get starved the same as ECT(0) flows if they’re traversing a shared
marking classic queue in competition with TCP Prague traffic,
whereas the suggestion makes it seem like the selection between
ECT(0) and N-ECT is meant to do something helpful toward noticing
problems.  (Possible source of confusion: by "3168 flows" in 3), did
you mean ECT(0) flows, thinking only those would be impacted?  Or did
you mean classic traffic in general across a 3168 marking queue?)

This leads me to believe that either I don’t understand the intended
suggestion, or that it didn’t incorporate some of the necessary
considerations of the complicating factors discussed earlier in the
thread.

While the “self-harm” case should in principle be possible to notice
when you do manage to get simultaneous flows for your Prague and
classic traffic across the same marking bottleneck, the “being harmed
by others” case would actually be worse than invisible to the canary,
in that it would corrupt the baseline data and tend to make even the
self-harm detection cases look like noise.  If you had some legit
problem paths with shared marking queues, you’d have results that
look like “yes ECT(0) and N-ECT get impaired for this receiver when
competing with our Prague flows, but they also get impaired randomly
even when we’re not sending competing Prague traffic” (for instance,
when someone else is sending Prague traffic or unresponsive traffic).

For these reasons (and also the reasons Jonathan outlined in response
to Gorry's question about the difference between this and other CC-
related work), I don't see how the canary strategy can be used
effectively here, so to me the detection strategies are a better fit
to the problem (though they do have some of their own challenges, as
previously discussed).

With that said, canary testing would still of course be useful to
detect implementation flaws where L4s/Prague flows systematically
suffer unexpected impairment and I expect would be used as a matter
of course, I'm just saying it would not help with the safety concerns
we've been discussing to detect harm to other people's traffic, as I
don't think they'd be able to see the collateral damage they're
causing.

Best regards,
Jake

---

From: Martin Duke <[email-masked]>
Date: Fri, 27 Aug 2021 13:52:59 -0700

Hi Jake,

I don't think I was clear enough.

The assumption in my proposal is that in a large deployment these things
will average out. So there is no intent to create additional connections
solely to detect self-harm. As large deployments will likely have multiple
connections to each endpoint and even larger pools of connections through
important bottlenecks, I assert that if a provider

(1) establishes a Not-ECT and ECT(0) baseline;
(2) Randomly turns on ECT(1) for 1/3 (?) of flows; and
(3) Both Not-ECT and ECT(0) meet some standard of
non-degradation/non-starvation in the aggregate.

then that is strong evidence that the hypothesis that the problematic 3168
queueing is not operating on the internet.

As for including both Not-ECT and ECT(0) in the baseline, I seem to recall
someone making separate claims about the impact of L4S on each; if that's
inaccurate, I have no objection to collapsing the control into one or the
other.

I hope that's clearer.

Meanwhile, I am not crystal-clear as to whether the proponents intend to
deploy L4S endpoints with Prague, something else that meets the Prague
requirements, or nothing at all.

Warmly,
Martin

On Sat, Aug 21, 2021 at 3:39 PM Holland, Jake <[email-masked]> wrote:

> > The "control" part of the canary experiment is how you identify
> collateral damage. The L4S hypotheses, informally, are that
> > (3) 3168 flows will only suffer starvation in queue configurations that
> are not deployed at scale in the internet.
> >

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 28 Aug 2021 00:12:52 +0300

> On 27 Aug, 2021, at 11:52 pm, Martin Duke <[email-masked]> wrote:
> 
> Hi Jake,
> 
> The assumption in my proposal is that in a large deployment these things will average out. So there is no intent to create additional connections solely to detect self-harm. As large deployments will likely have multiple connections to each endpoint and even larger pools of connections through important bottlenecks, I assert that if a provider
> 
> (3) Both Not-ECT and ECT(0) meet some standard of non-degradation/non-starvation in the aggregate.
> 
> then that is strong evidence that the hypothesis that the problematic 3168 queueing is not operating on the internet.

I believe that would be a false assumption, and consequently a false assertion.

That type of testing is only capable of detecting a problem in a conventional flow that just happens to run simultaneously with an L4S flow.  It is not reasonable to assume that will routinely occur.  This is a general problem with using canary testing to detect externalised problems.

And this is a problem that is theoretically *expected* to occur…

> As for including both Not-ECT and ECT(0) in the baseline, I seem to recall someone making separate claims about the impact of L4S on each; if that's inaccurate, I have no objection to collapsing the control into one or the other.

For at least one type of expected bad interaction, Not-ECT and RFC-3168 ECN traffic will probably be affected equally.  I explain why in my detailed comments document.  Note that merely carrying ECT(1) is not relevant for this case, but the alternative congestion response associated with an L4S transport.

> Meanwhile, I am not crystal-clear as to whether the proponents intend to deploy L4S endpoints with Prague, something else that meets the Prague requirements, or nothing at all.

TCP Prague within a modified Linux kernel seems to be the only known L4S endpoint implementation at present.

 - Jonathan Morton