Subject: [tsvwg] links to Canary methods for roll-out of new transport features

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 26 Jul 2021 23:58:54 +0100

So, there's been a change in the way people roll-out new features, that 
maybe we could say more about in the L4S OPS draft. What I write below 
is not specific to L4S, and I'd really welcome other familiar with using 
and evaluating such methods to chime-in and say more, but anyway here is 
starter:

Canarying is a partial and time-limited deployment of a change in a 
service/protocol and its evaluation as a part of the deployment. The 
method is used throughout the roll-out and helps to decide whether or 
not to continue with the rollout. The part of the service that receives 
the change is “the canary,” and the remainder of the service is “the 
control.” The canary deployment is performed on a small subset of the 
networ/users, than the control. Canarying is evaluated as an A/B testing 
process, to check the impact of the (initial) deployment.

See this from google:

https://urldefense.com/v3/__https://sre.google/workbook/canarying-releases/__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TuTKHR8C$ 
[sre[.]google] [sre.google]
<https://urldefense.com/v3/__https:/sre.google/workbook/canarying-releases/__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1loo_i6Op$>

https://urldefense.com/v3/__https://developer.android.com/distribute/best-practices/launch/test-tracks__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TuinRLVe$ 
[developer[.]android[.]com]
[developer.android.com]
<https://urldefense.com/v3/__https:/developer.android.com/distribute/best-practices/launch/test-tracks__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1lkLr2-F-$>

When working with QUIC  people have released an update to only to a 
small subset of the user base, monitor stability or another metric of 
interest, and
decide whether to roll out the update to more users, to wait for more 
data to come in, or to halt the rollout altogether.If one of the metrics 
you’re monitoring is off, or you check the user reviews and  see issues 
or complaints on a specific topic. You don't need to enable a feature
for anyone/network who you might expect to be hurt.

ECN isn't just "automatically" used, the app can decide (or at least the 
app-supplier), this will always be the case for QUIC anyway. The
result of these tests provide the sort of data that has informed QUIC 
(e.g. Chrome Canary), and I expect the basis of what is reported by
google and others in MAPRG. The point is that this allows statistical 
testing without massive impact, and the incremental roll-out.

This says something about akamai's use:

https://urldefense.com/v3/__https://www.akamai.com/uk/en/products/performance/cloudlets/phased-release.jsp__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TvXdVpGA$ 
[akamai[.]com]
[akamai.com]
<https://urldefense.com/v3/__https:/www.akamai.com/uk/en/products/performance/cloudlets/phased-release.jsp__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1ltQQWNXF$>

Cloudflare, etc have used similar approaches:

https://urldefense.com/v3/__https://medium.com/boozt-tech/canary-release-with-cloudflare-workers-84a9b45bac0f__;!!LpKI!znKgxfRLAd0ogD6r-X5u7fAz0eInch7MTeJDVQB5drmteEU3mhnP0no1TozuZjjw$ 
[medium[.]com]
[medium.com]
<https://urldefense.com/v3/__https:/medium.com/boozt-tech/canary-release-with-cloudflare-workers-84a9b45bac0f__;!!LpKI!x3qiMCrsIjOFTyJVBmvIh9lHMG6KKpySWLDUWq71wo9nz1gKk0cSHRe1lu8IFQaL$>

Gorry

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 30 Jul 2021 02:00:36 +0000

Hi Gorry,

A canary approach works very well for features that exhibit problems
on the connections that are using a new feature.

However, it works less well for features that cause problems with
competing traffic, as is the case for TCP Prague flows traversing
a classic queue.

The issue is that if the new feature (such as a TCP Prague
endpoint) is causing a problem for competing flows, possibly flows
from another different operator, there is no indication of the
problem that's visible to the TCP Prague sender except perhaps
service calls when the issue can be traced back to the source by
someone troubleshooting the impacted services.  From the experimental
endpoint's point of view, the performance is working great.  (Higher
throughput than usual!  Not much loss!)

So it's somewhat more challenging to build a canary that can detect
problems with competing impacted traffic.  This is in contrast to the
usage of canaries for most services, which will alert when the
experimental service or upgrade introduces a regression for that
service.

This problem is closely related to the difficulty of detecting a
classic queue, and in most cases would suffer the same pathologies.

This is covered to some extent in the first resource you linked,
under "Requirements of a Canary Process", and again under "Metrics
Should Indicate Problems".

I hope that's helpful.

Best regards,
Jake

On 07-26, 3:59 PM, "Gorry Fairhurst" <[email-masked]> wrote:

See this from google:

https://sre.google/workbook/canarying-releases/

https://developer.android.com/distribute/best-practices/launch/test-tracks 

https://www.akamai.com/uk/en/products/performance/cloudlets/phased-release.jsp 

https://medium.com/boozt-tech/canary-release-with-cloudflare-workers-84a9b45bac0f 

Gorry

---

From: "Gorry (erg)" <[email-masked]>
Date: Fri, 30 Jul 2021 10:12:36 +0100

I am not sure though, I think many CC- related topics can have potential collateral damage, and we have managed to deploy these gradually and improve the transport were necessary. So, what is different here to exploring methods such as larger Initial Window, BBR, Hystart, etc.

Gorry

> On 30 Jul 2021, at 03:02, Holland, Jake <jholland=[email-masked]> wrote:
> 
> Hi Gorry,
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 30 Jul 2021 10:21:05 +0000

> I am not sure though, I think many CC- related topics can have potential collateral damage, and we have managed to deploy these gradually and improve the transport were necessary. So, what is diffe…

Hystart makes a transport strictly less aggressive, by exiting the exponential growth phase early and continuing with much slower linear or polynomial growth.  There is no possibility of collateral damage except in case of implementation bugs.  This is an excellent use case for canary testing, and I would endorse its use with Hystart++.

Large IWs do have potential for collateral damage, but the conditions that would trigger it (small buffers) result in effects (high loss to IW) that are easily noticed by the transport employing it.  This is therefore also suitable for canary testing.

BBR is not so clear a case.  I am pleased that Google is taking a relatively cautious approach to deploying it in an Internet facing context, has designed it with standards-track CC coexistence in mind, and seeks to improve it when problems are reported and verified.  However, the ECN response introduced with BBRv2 is not standards-track compatible, and the likely collateral damage when operating in a shared AQM bottleneck is not easily noticed by the BBR transport itself, especially when those circumstances arise only infrequently.  Canary testing is thus an incomplete solution there.

The entire problem here is that L4S is likely to cause *externalised* collateral damage which is not easily noticed by the L4S transport itself.  Unless great care is taken to watch for such problems, canary testing will therefore fail to find them.

What is more, canary testing has as a prerequisite the confidence of correct operation and design gained though lab testing.  Lab testing of L4S has not given any of that confidence, thus progression to canary testing would be inappropriate.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 30 Jul 2021 12:23:40 +0100

> What is more, canary testing has as a prerequisite the confidence of correct operation and design gained though lab testing.  Lab testing of L4S has not given any of that confidence, thus progressi…

I was responding to a request to provide references to canary approaches.

Gorry