Subject: [tsvwg] ECT(1) Flag Day Plausibility

From: "Holland, Jake" <[email-masked]>
Date: Sun, 9 May 2021 03:28:19 +0000

Hi tsvwg,

I'm writing this because I think "flag day" hasn't had enough
airtime in this discussion. 

For context, I started out replying to this message:
https://mailarchive.ietf.org/arch/msg/tsvwg/P4miWNzBYUF6GDIie-1eFwsU6FU/
On 05-05, 1:34 PM, "Jonathan Morton" <[email-masked]> wrote:
> Requiring all existing ECN AQMs, which at present rely on RFC-3168, to be changed to treat ECT(1) differently, would require obsoleting RFC-3168 so that it is no longer an active Internet specification.  There is a process to do so, but I do not think that process would succeed if initiated.

But this is also relevant as a response to this thread from today:
https://mailarchive.ietf.org/arch/msg/tsvwg/fqat8qi7L0mwo7hwoGiuH63GIU4/
(no quotes, just the link for reference)

It's kinda long, so I'll break it into sections.

1. Plausibility

There are decent reasons to believe that it's not as impossible
to do this kind of thing as people might think.

I'd like to point to 2 recent successful examples of difficult
interop rollouts that might be relevant food for thought in the
present discussion:

A. The DNS flag days:
- https://dnsflagday.net/2019/
- https://dnsflagday.net/2020/
- https://github.com/dns-violations/dnsflagday

They went ahead in spite of measurements ahead of the first one
suggesting that over 5% of domains would move from working with
poor latency to non-working when support for retry without EDNS0
was removed as a resolver workaround, and in their post-2019
retrospective there were no big regrets and only minor fine-tuning
suggestions on how to improve for the next one.  (Though it should
be noted that the 5% of domains at risk represented a much smaller
proportion of traffic than the sum of all traffic over the 0.3% of
paths that have had observed 3168 CE-marking.)

Their situation, especially for 2019, was a little different in
that they were removing workarounds to support non-compliant
implementations that imposed costs on other operators, whereas
an ECT(1) change would (I'd think) need to have an extra step
first to determine consensus on the right new standard behavior.

B. Upgrading TLS:
I'll point here to a great talk by David Benjamin ("TLS ecosystem
woes" at Real World Crypto, 2018).  The whole presentation is worth
watching, but I'll link to the bit where he describes their
"breakage budget" as "several orders of magnitude below 1-3%":
https://www.youtube.com/watch?v=_mE_JmwFi1Y&17m29s

I think "breakage budget" is a useful concept and something we
should be trying to quantify and articulate better, to better
frame how we should be thinking about adopting changes that are
not obviously compatible with current standards that have some
deployment.  (Also interesting was the bound on the value given
for an appropriate scale in a widely scoped rollout.)

2. What it would take:

With those examples as context, I'll start by stating that I don't
see a reason it would be harmful to change the standard behavior
to treat ECT(1) as NECT instead of as ECT(0) in middle boxes, so I
don't know what objections would be raised to a standards track
document doing that?  (Aside from futility as we move to L4S, which
I'll try to address below.)  So I'm assuming "none" for the sake
of this argument.

If that were coupled with a deployment schedule and an organized
flag day with measurements and active tests to check on deployment
progress as we approach it (plus reports on upgrades to existing
implementations and their deployments), it could clear the way for
a safe L4S deployment that doesn't disrupt more than 1 in 50k users
(and hopefully somewhat fewer) to begin in perhaps as little as
4 years.

(Maybe even less, if the outreach and deployment is amazing enough,
but with a substantial chance at slipping plenty if it takes too
long to get started, or if the outreach and deployment does not go
especially well.)

It's a heavy lift and it seems unfortunate to do this in 2 steps
if we all end up landing on L4S in the end, but to me this would
at least be a sane path forward.

I rate this worth considering because it breaks many fewer people
along the way, and it only breaks people who don't upgrade their
equipment to a new standard in a timely manner.

3. Is it necessary?

Here I think there's 2 main questions:

What's the breakage budget in the context of the L4S discussion?
And what constitutes breakage?

We've seen strong evidence of a lower bound in deployment prevalence
at 0.3% of end users on the internet with marking 3168 queues in
their sometimes-bottleneck paths.

(We have not yet seen anything we can call an upper bound on
deployment prevalence or an accurate estimate, since 0.3% counts
only paths that observed actual CE marks in production traffic, and
we've also seen evidence that known-marking paths often can go quite
some time without marking in practice, plus multiple weak bits of
evidence that marking path prevalence is growing over time, all of
which mean it could very well be substantially higher.)

If 2% of those users' flows would go from capable to incapable of
maintaining non-"buffering" video play with common players during
active internet usage from within a home because of L4S interactions,
is that enough to call it broken?

I would argue yes, this results in frequently-buffering video that
significantly degrades user experience.  (I picked a 2% threshold
because that's less than what I'd expect to see under a birthday-
problem analysis of fq hash collisions at 50% adoption of L4S in
an active family network.)

If that's the case, this is already over a "several of orders of
magnitude below 1-3%" breakage budget, even under the unjustified
assumption that all marks came from FQ systems.  So although we
should not ignore the fact that shared queues and VPNs (where
present) increase the prevalence of trouble, that point may not
really need debating in detail if we're already over budget without
it.

(If you doubt this would make video unwatchable, maybe that's worth
testing?  Would it change your mind if it did?  To remember the
sense of scale of the "gross unfairness but not technically
starved under some definitions" problem we're facing, I sometimes
go back and watch when it happened in a live demo.  If you're
unclear, maybe this will help you as it helps me sometimes, 20s
further into this video where she brings the network briefly into
the state under discussion:
https://www.youtube.com/watch?v=g9VfnNTDv-I&t=8m45s )

Compare this with the "option 1" recommendations from RFC 4774,
as well as the emphasis on harm avoidance in Section 4 of RFC 8890
which, though merely informative, gives us a strong and recent
signal on which way we should be leaning.  Combine this with the
points Jonathan raised last meeting about the way bystanders have
no visibility into what went wrong, their experience will just be
that the internet gets worse and worse for an apparently random
set of content, as more L4S endpoints aside from that content get
rolled out.

I just don't see how this gets through last call without a real
fix that will credibly avoid this level of harm.  An unmanaged
"transition period" that just lets people suffer for potentially
years if L4S is successful at a 50% level in a rollout effort
doesn't look like it's going to have a low enough impact.  (I
know I don't think it's the right thing to do, and I will likely
try to make that point at least one more time if the last call
goes forward without addressing the core problems from TCP Prague
as the most widely applicable example use case.)

In answering the "really necessary?" question, I also have to
mention "SCE-or-similar" one more time.  It's an option 3 of 4774
solution that does not have the problems driving this argument,
which is why it looked interesting in the first place.  Though it
has performance issues, there's some decent evidence (e.g. the
codel-af lab tests) that there's ways to mitigate the performance
issues in ways that would still provide real benefits safely,
tunnel decap issues notwithstanding.

I mention it because for those who might still think a flag day
is a non-starter, this seems to me the 2nd easiest path short of
finding a robust fix for TCP Prague to improve latency, because
I just don't see how the demonstrated harm to bystanders can get
a pass.

4. Conclusion

But with that said, I do take the point that there's better
performance benefits in the long run from sticking with ECT(1) as
a solid classifier for a dual queue.  If the WG is still convinced
that's the best path forward, I'm suggesting that starting to work
toward a flag day to deprecate "treat ECT(1) as ECT(0)" into "treat
ECT(1) as NECT" as soon as possible may be the quickest and least
painful way to get there.  As they say, the future is bigger than
the past.

So I would urge that we not discount without due consideration an
approach based on deprecating the 3168 definition for ECT(1), as
long as it uses a realistic timeline for upgrading/decommissioning
old routers after the adoption of a new standard, and it targets a
sufficiently small impact to bystanders (and only those who failed
to upgrade to a modern standard in a timely manner after the new
behavior was defined, which as Jonathan mentioned, could only begin
after a new standard is fully adopted).

Not saying I love the idea, but it's better than doing nothing,
and it's better than breaking existing 3168 deployments wantonly.

(Also to be clear: I don't object to a properly constrained L4S
production experiment to happen earlier, but I do think the
constraints need to be stronger than an informative ops doc plus
hope.  I agree with Jonathan that as it stands it's well-
demonstrated that L4S is currently an option 1 scenario from
RFC 4774 that needs to have a credible plan for how it's kept
away from harming non-participants in order to conform with the
BCP, especially given the evidence of existing live 3168 deployments
that would be universally impacted in their fq hash collisions at
the very least, and possibly more.)

Just my 2c.

Best regards,
Jake

(tl;dr: "flag day" > "unmanaged L4S rollout".  Outside of SCE or
similar, I still haven't seen another alternative yet, and "flag
day" is more credible than it's gotten credit for.)

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 9 May 2021 16:26:20 -0700

> I'll try to address below.)  So I'm assuming "none" for the sake

Jake, the question that arose for me on seeing this proposal was:

Why not instead try to clean up tunnel behavior so that the transitions
ECT(1) -> ECT(0) and ECT(0) -> ECT(1)  within the network make it through
tunnels? If we go for a flag day, wouldn't this be more profitable?

The specification change would be to one case in RFC 6040 Fig. 4. The other
steps would be as you describe.

The issues that I see with treating ECT(1) and not-ECT are

   1. The fundamental problem is the incompatible meaning of CE, not that
   of ECT(1)
   2. Treating ECT(1) as not ECT forecloses many options. Fixing tunnel
   behavior does not.

Allow me to quote from Martin Duke's message of March 26 (see
https://mailarchive.ietf.org/arch/msg/tsvwg/qosIqFlvjo9ZPKYoTzF1V8ZmI6g/):

On Fri, Mar 26, 2021 at 10:15 AM Martin Duke wrote:

> Bob has identified all of the problems with using DSCP for L4S. Among them
> is running L4S through a tunnel.
>
> Another axis of discussion was changing the L4S signal to be
> ECT(1)->ECT(0) and preserving the CE signal (thread: [1]), which
> comprehensively solves the coexistence issues. But this fell down,
> partially because of the (needless) RFC6040 tunnel decap behavior, which
> will revert any outer-header changes. There are some other problems, of
> course, they don't affect bystanders and I don't see them as likely to be
> fatal.
>
> Honestly, if we're going to break L4S in tunnels anyway, I would just as
> soon break it with the semantic change and not take all the other DSCP
> baggage that Bob describes. We could even bis the mistake in 6040 so that
> "eventually" the needless behavior goes away.
>
> This is not a statement in favor of 1->0 marking. It is a statement that
> if the WG is going to insist on a DSCP to ship this design, I would prefer
> 1->0 over that.
>

Mike Heard

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 10 May 2021 01:02:33 +0000

Hi Mike,

From: "C. M. Heard" <[email-masked]>
> Date: Sun,2021-05-09 at 4:27 PM
...
> Why not instead try to clean up tunnel behavior so that the transitions ECT(1) -> ECT(0) and ECT(0) -> ECT(1)  within the network make it through tunnels? If we go for a flag day, wouldn't this be more profitable?
...
>>Bob has identified all of the problems with using DSCP for L4S. Among them is running L4S through a tunnel.
>>
>> Another axis of discussion was changing the L4S signal to be ECT(1)->ECT(0) and preserving the CE signal (thread: [1]), which comprehensively solves the coexistence issues. But this fell down, partially because of the (needless) RFC6040 tunnel decap behavior, which will revert any outer-header changes. There are some other problems, of course, they don't affect bystanders and I don't see them as likely to be fatal.

That also would be a reasonable approach to me.  I was including
this idea under "SCE-like", in that the change of codepoint in a
packet stream is interpreted as a high-fidelity congestion signal
as output from the network.

But I think Bob and others had some reasonable objections that
not only is the classifier unreliable under that proposal, but
also the unreliability increases with higher adoption, and we
can expect some related performance problems down this path that
would be avoided with the current l4s-id approach, assuming a
way can be found to roll it out safely.

I think we've seen a fair amount of discussion on SCE in general,
and I was assuming we can expect a sort of "reversed SCE" to have
roughly similar characteristics (with some unproven possibility
that there are some further improvements possible).

But we haven't seen much explicit discussion on sticking with
the current l4s-id while making the world ready to handle it
safely as a harm-avoidance strategy.

The threads referenced at the top of the prior message (and
maybe a few others earlier?) had a few people who seemed to be
saying something about the concept, but had some conflicting
views.  People seemed to be taking the position that it was
either trivial or essentially impossible and I don't think
either is true.

I think it would require a challenging but doable flag day.  I
think it would result in a slightly improved performance in the
end, relative to using ECT(1)->ECT(0) as a congestion signal.

And although 2 years ago I would have rated it almost certainly
more trouble than it's worth, the lack of motion on this
discussion (and even more so on the implementations) has me
reconsidering that position.

I hope that helps to clarify the intent behind the proposal.

Best regards,
Jake

(To the tunnel point: I agree 6040 would also need fixing with a
standards document that changes behaviors if we did something SCE-
like, but if I understand correctly enough (not assured! grain
of salt here) it could be done safely in parallel without blocking
on a flag day, and traffic over tunnels mostly just wouldn't get the
latency benefits until they get the upgrade, since they'd be stuck
with basically classic behavior.  Not sure that captures all the
nuances, but I thought I'd try to dodge that rathole in this thread
and leave it as a side issue.  For the sake of staying focused on
a flag day safety discussion, I was aiming to keep the effects here
summarized as "worse performance", some of which comes from the
tunnels having a long tail upgrade cycle that keeps their traffic's
behavior classic, and some of which comes from other effects.)

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 10 May 2021 03:44:26 +0100

Jake,

TL;DR of my contribution: Big question mark over probability of 
collisions maths. If I'm right, this is an incremental deployment not a 
flag day.

inline, tagged [BB]...

> to do this kind of thing as people might think.

[BB] To keep the email shorter, I've snipped the summaries of DNS and 
TLS flagdays - taking them as read.

> an active family network.)

[BB] Whoa! Where do you get 2% from?

We need to calculate the chance that the hashes of 2 or more 
/long-running/ flows collide when they have already collided in time. So 
we need to find the probability that 2 or more long-running flows 
overlap (significantly) in time (irrespective of their hashes), then the 
prob of a hash collision is conditioned on that prob. It's still a 
birthday analysis, but with only a few hashes in the set at any one 
time. Eg, prob of collision when n flows collide in time is p_n
* p_2 = 1 - [          1  * (1 - 1 * 2^(-16))] = 0.0015%
* p_3 = 1 - [(1-p_2) * (1 - 2 * 2^(-16))] = 0.0046%
* p_4 = 1 - [(1-p_3) * (1 - 3 * 2^(-16))] = 0.0092%
* p_5 = 1 - [(1-p_4) * (1 - 4 * 2^(-16))] = 0.015%
etc.

When someone mentioned the birthday analysis for hash collisions on the 
jabber during the last tsvwg in March, I pointed this out.

Surely that's far far far far less than 2% of the flows active at any 
one time. Only about 0.05% of flows last long enough to worry about 
fairness in the first place

Also, bear in mind that, today (without L4S), when a Cubic flow pushes 
in to an established flow, for a few dozen RTTs they're nowhere near 
equal rates. Hence the 'overlap significantly' added above.

And... they might be app-limited and not be filling the pipe anyway.

Here we need data for the correlation between flow arrivals, and 
particularly long flow arrivals. And we need the distribution of flow sizes.
I haven't looked for papers on that for a while, but I couldn't find any 
when we last looked (2016), so we gathered data ourselves. See this 
unpublished paper: https://www.bobbriscoe.net/projects/latency/unsctrdyn.pdf
(It was for another purpose, so it's not exactly what we need here, but 
close.)

We looked at flow size distribution in CAIDA studies over a decade (Fig 
2), but then realized that CAIDA hardly sees CDN->customer traffic 'cos 
it's taken at an Internet exchange. So we passively gathered packet 
traces from an ISP access in Sweden, both fixed and mobile (details in 
the paper). See Fig 4, which compares it with the longitudinal CAIDA 
data from Fig 2 (greyed). For instance, 0.3% of flows are >10MB, which 
would each be complete in a few seconds.

Then Fig 5 attempts to visualize flow arrival correlation, by showing 
the distribution of how many flows were already active when each flow 
started (definition of active is in the text). That's all flows, not 
just long flows.

I don't think this data gives all the parameters we need to calculate 
overlap correlation, but we might be able to produce a better finger in 
the air, at least.

Another aspect is that human usage bunches up - it's obviously not even 
over time: measurements of customer access links in BT (this data was 
cleared for publication) showed an average access link is utilized only 
about 2-3% of the time during the busy hour, a figure which has remained 
relatively constant since the start of mass market Internet (1995) - but 
I haven't been there to see the measurements for the last 6 years.

> really need debating in detail if we're already over budget without

[BB] As well as your point about treating ECT1 as NECT, another possible 
low hanging fruit would be to add the last bit{1} of the ECN field into 
the hash of the flow IDs in FQ schedulers. I don't know how quickly this 
would deploy. But it would address the VPN collisions problem.

{1}: Initially, it might be better to classify CE along with ECT0 and 
NECT. But later in the deployment, an upstream AQM could be more likely 
to be L4S than 3168, so CE would have to be classified with ECT(1).

> https://www.youtube.com/watch?v=g9VfnNTDv-I&t=8m45s )

[BB] Do you mean 20 mins not 20s? Even then, I don't see what you're 
seeing. Perhaps @18 mins?
Can you just say what condition you've noticed.
Also bear in mind that demo was done before the RTT-independence code 
was added to Prague.

Also you're watching TCP there, not a real-time CC that is more 
inelastic and adapts more slowly to change.

> doesn't look like it's going to have a low enough impact.

[BB] The possibility of shared queue Classic AQMs is definitely an 
unknown...

But setting that to one side, and taking your estimate of the likelihood 
of harm assuming mostly FQ, it seems seriously incorrectly calculated.
If this has been the cause of people talking past each other, I don't 
know how it's gone so spectacularly unnoticed.

> (I

[BB] Only now do I see that the thing you're describing as a 'flag-day' 
is "treat ECT1 as NECT". Why? It doesn't need to happen on a 'day', 
because there's no usage of ECT(1) at present. It could happen over 
time, starting ASAP.

If the FQ-CoDel AQM classifier is going to be changed, don't classify 
ECT1 as if it's NECT - no need for that ever (and in principle CAKE is 
no different in this respect).

Instead, simply change the classifier for the immediate shallow ECN 
threshold (which is already in the Linux FQ-CoDel code to support DCTCP 
marking in DCs).

Current:
* Default: Codel marks ECT0 & ECT1
* DC option: Immediate shallow threshold marks ECT0 & ECT1

Proposed:
* Default: Codel marks ECT0;
                 Immediate shallow threshold marks ECT1
* DC option: Immediate shallow threshold marks ECT0 & ECT1 (unchanged)

Latency results for this over Internet paths are nearly as good as the 
DualQ, even at the 99.999%-ile.

At the same time, it would make sense to classify ECT1 as part of the 
hash, as I said earlier in this email.
That will need some debate.

And a debate can be had on other ways to update an FQ-CoDel or CAKE AQM 
to support L4S, but not here and now.

> after a new standard is fully adopted).

[BB] That leaves shared queue 3168 AQMs. If they've been languishing for 
some time, I suspect they won't get updated any time soon either. We 
have the techniques in l4sops that might work on some of them (I think 
the ECT1 bypass tunnel would give the most scope to steer round them 
from any distance). But this is all a big unknown.

> the very least, and possibly more.)

[BB] Thx for starting this conversation.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Pete Heist <[email-masked]>
Date: Mon, 10 May 2021 06:42:00 +0200

Hi Bob/all, some responses below...

On Mon, 2021-05-10 at 03:44 +0100, Bob Briscoe wrote:
> > > Requiring all existing ECN AQMs, which at present rely on RFC-3168,
> > > to be changed to treat ECT(1) differently, would require obsoleting
> > > RFC-3168 so that it is no longer an active Internet specification. 
> > > There is a process to do so, but I do not think that process would
> > > succeed if initiated.
> /long-running/ flows collide when they have already collided in time.

It might help to qualify further what the definition of "long-running"
is. I'll point back to some flow completion time results we posted
earlier, which does show that the FCT of short CUBIC flows are
adversely affected by competing Prague flows relative to competing
CUBIC flows, with the  definition of "short" in this test being a
lognormal length distribution with a P5 of 64KB, and a P95 of 2MB:

https://mailarchive.ietf.org/arch/msg/tsvwg/ICN0ZaxSe4NtAfzyZYLXWoVPIxA/

Regardless of any experimental results though, the L4S transports to
date (TCP Prague and SCReAM at least, according to L4S-ID section 4.3)
share the same response to a CE mark as DCTCP. In section 1 of RFC8257
(DCTCP): "DCTCP MUST NOT be deployed over the public Internet without
additional measures" (one of those measures including flow separation
with DSCP). When it comes to the congestion control response or the
interaction with conventional flows, is there a false equivalence
between DCTCP and the existing L4S transports?

Pete

> So 
> unpublished paper: 
> https://www.bobbriscoe.net/projects/latency/unsctrdyn.pdf

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 10 May 2021 12:17:55 +0100

Mike,

On 10/05/2021 00:26, C. M. Heard wrote:
> Why not instead try to clean up tunnel behavior so that the 
> transitions ECT(1) -> ECT(0) and ECT(0) -> ECT(1) within the network 
> make it through tunnels? If we go for a flag day, wouldn't this be 
> more profitable?
>
> The specification change would be to one case in RFC 6040 Fig. 4. The 
> other steps would be as you describe.
>
>  1. The fundamental problem is the incompatible meaning of CE, not
>     that of ECT(1)

[BB]
On 1. the two meanings of CE are both 'highest severity', which is all 
tunnels need to care about. So I don't see what you mean here.
On 2. I agree - on both your sentences.
But you'll see from my response to Martin just now ( 
https://mailarchive.ietf.org/arch/msg/tsvwg/GuqY3UvUE78VNcZVR5S7PRipAes/ )
that I strongly believe that the 'changing-tunnel-behaviour' ship has 
sailed. We have to move on.

We had to choose which number was higher. In hindsight the other would 
have been better.
a) Enabling both 1>0 and 0>1 in tunnels would foreclose much more 
subtle, but possibly much more important, security options.
b) Switching tunnels from 1>0 to 0>1 would effectively be equivalent to 
(a) for 2 or 3 decades.

Thx for pointing Martin's posting out. I was off mail over that period 
so you'll see I've just replied to it. Pls treat that as a partial 
answer for you to.

Cheers

Bob

> https://mailarchive.ietf.org/arch/msg/tsvwg/qosIqFlvjo9ZPKYoTzF1V8ZmI6g/ 
> <https://mailarchive.ietf.org/arch/msg/tsvwg/qosIqFlvjo9ZPKYoTzF1V8ZmI6g/>):
>
>     Bob has identified all of the problems with using DSCP for L4S.
>     Among them is running L4S through a tunnel.
>
>     partially because of the (needless) RFC6040 tunnel decap behavior,
>     which will revert any outer-header changes. There are some other
>     problems, of course, they don't affect bystanders and I don't see
>     them as likely to be fatal.
>
>     Honestly, if we're going to break L4S in tunnels anyway, I would
>     just as soon break it with the semantic change and not take all
>     the other DSCP baggage that Bob describes. We could even bis the
>     mistake in 6040 so that "eventually" the needless behavior goes away.
>
>     This is not a statement in favor of 1->0 marking. It is a
>     statement that if the WG is going to insist on a DSCP to ship this
>     design, I would prefer 1->0 over that.
>

Bob

---

From: Martin Duke <[email-masked]>
Date: Mon, 10 May 2021 10:42:10 -0700

Hi Jake,

IIUC you are suggesting that we could quickly produce a standards-track
document that says "treat ECT(1) as Not-ECT unless you know what you're
doing" to prep the ground for L4S, which seems like a useful thing to do.

However, I suspect we are "several orders of magnitude below 1%" as it
stands because of several things that have to be true for there to be a
problem:
https://mailarchive.ietf.org/arch/msg/tsvwg/PyKGvGnIQ06BFFSw7a63VpTHsWY/

On Sat, May 8, 2021 at 8:29 PM Holland, Jake <jholland=
[email-masked]> wrote:

> > Requiring all existing ECN AQMs, which at present rely on RFC-3168, to
> be changed to treat ECT(1) differently, would require obsoleting RFC-3168
> so that it is no longer an active Internet specification.  There is a
> process to do so, but I do not think that process would succeed if
> initiated.
>

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 11 May 2021 01:38:40 +0000

Hi Bob,

I will also cut some stuff in the interest of brevity, and respond
to just a few points that seemed to me the most useful.

This isn't meant to dismiss anything else you raised as unimportant,
just I'm first focusing on trying to clarify my meaning from the
prior email where your response indicates I probably didn't get
some key points across very well.

> [BB] Whoa! Where do you get 2% from?

Sure, let me flesh out my back of the envelope reasoning for my "this
looks too harmful" position and speak to the 2% hand-wave.

To clarify up front, I'm not really looking at an "all flows" model,
I'm more looking for an "impact to user experience" framing, for which
I'll try to articulate a useful scenario as a straw man.

Main Question:

Does our lower bound on expected harm put us over our breakage budget
already, or do we have to look to how much higher than our lower bound
the harm prevalence would need to be before we can decide?

Background modeling assumptions:
- lower bound from deployment observations:
  - at least 0.3% of all internet paths
  - they might all be fq
- breakage budget:
  - 2% rate of "buffering" result on video for these reaches the level
    of "unacceptable harm"

Suppose family members during their daily free time are individually
watching ABR video on different services, some of which have L4S support
and others of which are classic, and they're at one of the locations with
one of the more common fq-codel marking AQM deployments.

Sub-question 1:
How many simultaneous ABR streams among the family will result in a 2%
collision rate?

Expected hash collisions for n elements in D hash buckets:
  E = n-D + D((D-1)/D)**n
(from http://matt.might.net/articles/counting-hash-collisions/)

Set D = 1024 based on the default fq-codel settings in tc:
https://man7.org/linux/man-pages/man8/tc-fq_codel.8.html (see "flows")

def e(n):
  return (n-1024) + 1024 * ((1023/1024.)**n)

This crosses the 2% collision threshold when stepping from 6 to
7 active flows:

>>> e(6)
0.014629377978053526
>>> e(7)
0.020474466476116504

Here's my other big questionable (but I think plausible and testable)
assumption:
- 30-45s+ of collision between ABR video running over L4S with TCP Prague
  and ABR video running with a classic TCP will likely result in "buffering"
  for the classic connection.

Then I think about some of the chillaxed family evenings I've seen where
maybe someone is in their room on Youtube and someone else is in their
room on Netflix, and Hulu is on the big screen in the family room but the
4 people mostly-watching the big screen are also scrolling thru their
feeds on Reddit, Tiktok, and Facebook and letting those ubiquitous
30-60s video clips play.

When that's the scene, whichever services are still on classic endpoints
are going to be seeing sporadic buffering at a roughly 2% level during
the active consumption of all those streams.

This can be a new short-lived 2% buffering event whenever one of the
scrolling platforms opens a new flow that lasts for a bit too long and
stabilizes at too high a bitrate, which I rate plausible because of the
degree of unfairness on the co-existence problem when there's a
collision.

Maybe this is too pessimistic?  I'm assuming here that an L4S ABR will
basically not notice the classic flow's existence in a fq collision,
and will reach its ABR rate based on the achievable bitrate within its
FQ hash bucket, crowding out the classic flow almost entirely and
stabilizing at its top rate within ~30-45s, especially given a starting
target informed by recent history.

To me that reaches "already borderline unacceptable" for user experience
if it's happening at a 0.3% scale across the internet, and then we remind
ourselves that we started with super-generous assumptions that could make
the scale much worse.

I mean, some people do have vpns, and maybe there's some ISPs that did
turn on shared queue marking after all, plus we have good reason to
believe that 0.3% of paths is undercounted but we're not sure by how much,
plus who knows when some device on the network is going to decide it
should download an update, which would crowd out a colliding flow even more
if it's L4S.  OTOH, maybe not all of the 0.3%+epsilon paths have 6-person
families, but then again there's also a fair few that share across
multiple apartments, etc.  I'm taking all this fine detail as "in the
noise", if we agree that this is scenario is sufficiently realistic
and already problematic.

This kind of thinking is what pushes me toward wanting a solid harm
reduction strategy that doesn't rely on operators' monitoring, especially
when I think about how they'll be tuning the alert thresholds to whatever
is easiest on them.

I mean sure, Facebook *could* tune it so that they'll get lots of false
positives and very few false negatives so that they don't crowd out Hulu
and cause buffering for Dad's Hulu show, but how likely is that?  How are
they going to respond to the high rate of false positives?  How many
operators won't just point a finger at someone else?  How will they even
see the complaints, when it's their competitors that experience the
buffering video?  Will they really catch "long-lived" flows in the 2-
minute range needed to cause video buffering problems for competing
traffic?

Those chillaxed family evenings get a lot less calm when people start
screaming at each other to stop using the internet so they can watch
their movie in peace, in my experience, so I want to be pretty confident
that won't be the outcome here, and that line of reasoning did not put
me at ease on this score.

NB: I'm using recognizable service names here by way of example, but I
think the problem gets worse again when you consider the long tail of
niche content providers who maybe don't have a dedicated player-
optimization team, in place of any one of the more famous services.

Maybe I have some importantly wrong assumptions baked in there, but I
hope it might be possible to at least use it as a basis for some testable
observables that could resolve the concerns one way or the other.

If this estimate for this kind of scenario can be shown to be too
pessimistic by a couple orders of magnitude, I could see changing my mind
if we find a way to get a handle on the scale of the VPN and shared queue
questions.

And similarly, I hope that if these estimates bear out as roughly not-that-
far-out-of-line, that people will get more concerned about avoiding this
level of harm as part of an experimental rollout.

Anyway, that's my strawman model for the envelope calculation on where
I'm seeing a potential user experience problem.  I hope it's helpful to
have it written out.

I've cut a big section adjacent to the "What Use is Top Speed without
Acceleration?" reference here, in hopes that clarifying the model I was
using was the thrust of the response you needed, but if you'd like me to
answer more on that, I'll have to leave it to another thread.

>> If you're

Sorry.  I've attached screenshots highlighting the before/after that I
saw when the queue changed from dualpi2 to pie between 8:53 and 9:08 in
the linked video, in the expectation this emulates roughly what we'd see
from sharing a queue (either from fq collision or otherwise).

This is just one example of the "extreme unfairness" coexistence
behavior problem we've discussed many times, I just wanted to
provide an easy illustration because from some of the comments I
wasn't sure people had a sense of the scale.

To me this looks like "you probably can't watch ABR video with those
classic flows", but from some of the discussion I'm not sure everyone
agrees, so I wanted to point to an example that demonstrated why I'm
concerned on this point.

> know how it's gone so spectacularly unnoticed.

+1, if this is the crux of our misunderstandings, then by all means
let's stop talking past each other and get this sorted out. 

But with that said, I think a lot of people have been talking instead
about the harm in the unknown scope of the shared queue deployments,
and this discussion doesn't really speak to those points.

Here I'm mostly trying to point out that that we might not even need
to debate those points if we have substantial expectation of harm just
from the FQ collisions, which I think might be able to cut a lot of
wheel-spinning debate if it holds up.

If it doesn't hold up, I'm not sure it answers the rest of the concerns,
though it'll be helpful to me to have the point clarified, since I
think this has been mostly ignored on the list since it was first raised
as a concern some time ago (I think by Pete?  But it was a while back
and I'm sure easily lost in the sea of messages.)

> because there's no usage of ECT(1) at present. It could happen over 

Compatible changes that don't cause harm can be rolled out without a
flag day.

However, incompatible changes that do cause harm need to give fair
warning and make reasonable attempts to get upgrades rolled out at
a scale appropriate to avoid a significant amount of harm occurring
in practice.  This is, to me, what the flag day suggestion is about.

The harm comes when L4S traffic (specifically TCP Prague) starts flowing,
but the harm caused by those flows comes as a result of today's network
treatment of ECT(1).

The idea is that if we intend to start rolling out an incompatible
change that will cause harm, we need to give people (including people
who do not intend to deploy experimental specs without a well-defined
rollback option) a safe alternative behavior, and ensure that it's
rolled out at a suitable scale prior to rolling out any new and harmful
usage.

I agree that queues could probably be upgraded with L4S support as one
option and "ECT(1) as NECT" as the alternative option during the same
upgrade cycle without harm, and the flag day can be geared around
deprecating the 3168 behavior of "Treat ECT(1) as ECT(0)", as opposed
to specifying which specific allowed ECT(1) behavior should be adopted
by upgrading queues.  Again, the harm doesn't actually start until the
traffic starts flowing.

However, if the harm concerns are well-founded, before any TCP Prague
endpoints can be safely enabled, all (or at least almost-all) previously
existing queues need to be upgraded with *some* change to how they
interpret ECT(1).  And many (perhaps most?) of these, for their own
policy reasons, need to have a standards-track option deployed.  And
as standards-reviewers, it's our role to make sure that option is
known to be safe in the presence of L4S traffic by the time L4S
traffic starts flowing.

This is why I'm saying this approach could use a "treat ECT(1) as NECT"
document as a prerequisite to a flag day, with L4S traffic to start
no earlier than the date of the flag day under the presumption (and
some degree of confirmation) that almost all deployed 3168 queues have
been upgraded to some kind of safe behavior.

"Treat ECT(1) as NECT" is an option 1 (of RFC 4774) change to RFC 3168
that could be shipped as a standards-track document and would be safely
upgradeable from there to L4S behavior, thus the suggestion that it's a
useful part of making a safely-deployable change.  (Obviously this would
be instead of the also-safe but not upgradeable to L4S alternative of
using an ECT(1)->ECT(0) transition as a congestion signal, since that
option 1 alternative has the intractable performance issues you've
outlined.)

> That will need some debate.

I'm starting skeptical on this as a solution.

I think this doesn't really avoid hash collisions, it just puts them in
different places and adds some funny ordering issues.  But if it's worth
discussing in more depth, that's definitely a topic for another thread.

> to support L4S, but not here and now.

Sure, I'm just saying that a known-safe standards track behavior for
the treatment of ECT(1) in a world with TCP Prague endpoints rolling out
needs to be an option, if the safety concerns on coexistence are
legitimate issues.

Best regards,
Jake

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 11 May 2021 12:42:32 +0200

Hi Jake,

A top post response because I want to address your conceptual point and not (all of the well thought out) details (with which I agree).

I think you hit the problem spot on and I agree with your way of assessing the potential harm and mapping that back into a believable model of end-user behavior.

Your description below seems to describe a realistic scenario where multiple concurrent long running flows will exist that are both capacity* and latency** sensitive, and that have obvious failure modes ("buffering") that even a lay person can unambiguously recognize.

These lay persons are also the target audience for L4S, if these folks recognize that L4S destroys streaming, they will shut L4S off in a heart-beat (like end-users do for IPv6, based on little more than a gut-feeling). L4S needs to improve things without obvious costs/harm, or consumers will not accept it.

Best Regards
	Sebastian

*) Capacity sensitive within reason, typically the is an upper limit to quality/resolution and hence data size, but below that reductions in throughput cause often quite noticeable reductions in video quality, these might not be as problematic as "buffering-"stalls, but still are noticeable deviations from what users expect.

**) DASH style "streaming" video delivery, seems latency sensitive, but not overly so, for one thing video timing has frame granularity putting us into the 8 to 16 ms range, but events like Stuart's searching/skipping ahead will expose the amount of pre-buffering the stream uses (its "pulse" granularity if you will wich will be in the dozens to 100s of ms, with such pulses being required so that on developing congestion the stream can switch to requesting the next "segments" at lower quality/size, before the buffers empty, but if the on-path buffers are bloated, it will take a while for the new date to make it onto the users screen).

On 11 May 2021 03:38:40 CEST, "Holland, Jake" <jholland=[email-masked]> wrote:
Hi Bob,

Main Question:

Sub-question 1:

Expected hash collisions for n elements in D hash buckets:
  E = n-D + D((D-1)/D)**n
(from http://matt.might.net/articles/counting-hash-collisions/) <http://matt.might.net/articles/counting-hash-collisions/)>

Set D = 1024 based on the default fq-codel settings in tc:
https://man7.org/linux/man-pages/man8/tc-fq_codel.8.html <https://man7.org/linux/man-pages/man8/tc-fq_codel.8.html> (see "flows")

def e(n):

e(6)
0.014629377978053526
e(7)
0.020474466476116504

If you're
unclear, maybe this will help you as it helps me sometimes, 20s
further into this video where she brings the network briefly into
the state under discussion:
https://www.youtube.com/watch?v=g9VfnNTDv-I&t=8m45s <https://www.youtube.com/watch?v=g9VfnNTDv-I&t=8m45s> )

Best regards,
Jake

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 13 May 2021 13:39:57 +0100

Jake,

On 11/05/2021 00:25, Holland, Jake wrote:
> This isn't meant to dismiss anything you raised as unimportant,
> I'll try to articulate a useful straw-man scenario.
>
> Main Question:
>

[BB] Perhaps I'm being thick, but it doesn't say 2% of what, e.g. 
whether it's 2% of flows within some time period, or 2% of videos that a 
family watches at some time during a day, or 2% of flows at each instant 
(you still have to know how long an instant is to be able to calculate 
anything, given there are continual flow arrivals and departures), or 
whatever.

And I think it still doesn't explain where the 2% comes from.

Would you mind explaining this longhand. I'm just not getting it.

>
> Suppose family members are individually watching ABR video on different
> services during their daily free time, some of which have L4S support and
> others of which are classic, and they're at one of the locations with one
> of the more common fq-codel marking AQM deployments.
>
> Sub-question 1:
>>>> e(6)0.014629377978053526
>>>> e(7)0.020474466476116504
>>>> Here's my other big questionable (but plausible and testable) assumption:
>>>> - ~30-45s+ of collision between ABR video running over L4S with TCP Prague
>>>> are going to be seeing sporadic buffering at a 2% level based on the
>>>> activity of others that are on L4S.

[BB] If I could understand it, this might start to get to the bottom of 
why some people think hash collisions will be a massive problem and 
others don't. Thank you for continuing to try.

Let me try to read out to you all the factors I can see here, some of 
which are silently included in your model:

First, assume we confine ourselves to residential broadband for this 
email - OK willing to do that (but note that mobile doesn't have 
multiple users per access link).

Then:
a) 100% of homes in the population under study have FQ on their access 
link (by definition of the study);
b) 100% of their FQ algos use the worst-case hash algo (without 
set-associative hash);
c) 100% non-upgradeable FQ classifiers
d) Everyone has worst-case 50:50 L4S:Classic traffic mix;
e) Every home has 100% utilization of the access link continuously

All these assumptions lead to over-estimates of the harm. I believe b) 
and e) contribute most greatly to the overestimate, but I'll go through 
them all.

a) A harm metric should be across all users; but I'll be generous and 
allow you to confine the population of the study to those who have 
chosen to use FQ (given, if there is harm, they can never not use FQ to 
avoid it)

b) I thought all CAKE is deployed with the set-associative hash.
Why wouldn't there be deployments of FQ-CoDel with the set-associative 
hash in future / soon / already?
What is the % deployment of the set-associative hash likely to be by the 
time L4S is deployed?
With the set-associative hash:
* What is the probability of at least 2 flows colliding out of 6 flows 
(say)?
* What number of flows leads to 2% probability of at least 2 flows 
colliding?
(I could work that out, but not right now...!)

c) For remotely upgradeable and new FQ deployments, even without the 
set-associative hash, the measures outlined in the last email would 
solely involve classifier changes, perhaps implying they're more trivial 
to implement without resource implications than improving the hash algo, 
i.e.:
i) enabling the shallow threshold for ECT1 by default (it's stateless, 
and it's currently a config option for all ECT).
ii) and classifying on ECN + flow-ID

Bullet (i) means that, if you get a hash collision between Classic and 
L4S, the Classic flow at least dominates the L4S flow, rather than the 
other way round. {Note 1}
Bullet (ii) separates out Classic and L4S for VPNs (in normal 
circumstances, but not if there are also collisions of course). So at 
least Classic doesn't dominate L4S when they are both in the same VPN.

I suspect most ISP-supplied home gateways are remote-upgradeable, and 
would therefore pick up any such changes within a few months (depending 
how long the operator take to validate the change). On a quick search I 
haven't been able to find the percentage of gateways bought from third 
parties in each of the world's main markets, but I'm sure such stats are 
available somewhere.

{Note 1}: Note this will be true starvation of L4S by Classic, not just 
an unfair balance, because the Classic flow is blind to the shallow 
ECT(1) threshold and drives the queue over it. So it might be necessary 
to deploy this change in two stages - first slightly shallower than 
'target' for ECT(0), before going fully shallow.

d) If we assume L4S is successfully deployed, one would certainly expect 
everyone to go through a 50:50 point at some time, but not everyone at 
the same time.

e) A broadband service would be unusable if regular 'busy hour' usage 
was keeping it at 100% utilization, as you describe.
This needs a little more explanation that the previous four items, but 
it's also the most important point....

* As I said, on average /during the busy hour/ a broadband link is about 
2-3% utilized (that was the planning rule I was used to - other 
operators might work differently but not that differently). Operators 
continually grow link capacities as expectations of what people can do 
with the capacity grow. Think back to when 4Mb/s capacity was the norm. 
You wouldn't have expected to be able to run 7 video streaming sessions 
at the same time.

* Given your chillax scenario seems realistic, why isn't 100% 
utilization realistic? It's because there's very little similarity 
between how many /sessions/ seem to be proceeding, and what is actually 
happening on the link when you watch flows and packets. Typically, 
either these sessions are app-limited and not filling the link, or one 
is briefly filling the link to refill the playout buffer then it goes 
idle, so they very rarely all overlap at once.

* If the scenario you describe with these 7 foreground sessions 
continuously consuming 100% of the capacity were ever the norm for more 
than a tiny  percentile of customers, then 1/7 or capacity per flow 
would drop to 1/8 (or say 1/15) whenever another application transferred 
1 more flow (or say 8 more flows) for a few seconds. Then the videos for 
those homes would already be buffering a lot of the time today, due to 
FQ and insufficient capacity.

Note, there are two meanings of 3% utilization depending on the flow 
behaviour:
* Capacity-seeking:
     100% utilization 3% of the time, then flows run in fast but short 
trains, so it's "much less likely" that 7 flows (or whatever the limit 
is) will all coincide, relative to what you were imagining with 100% 
utilization
* App-Limited:
     3% utilization, 100% of the time, then there will never be any 
congestion and no harm
* a mix of the two behaviours:
      which would mean the chance of 7 simultaneous flows was somewhere 
between zero and "much less likely".

___________________________
So, lets go through a) to e) again, putting in variable letters where 
you had assumed 100%:
a) 100% of customers have FQ on their access link [by definition of the 
scenario];
b) b:(100-b)% of their FQ algos use the set-associative : simple hash;
c) c:(100-c)% upgradeable : non-upgradeable FQ classifiers
d) A distribution of L%:C% for the L4S:Classic traffic mix at any one time;
e) Average u% utilization of the access link, with a distribution around 
that. I say u = 2-3%, but let's hear what other operators aim for these 
days.

I suggest you say what reasonable numbers would be for the variables.
Then (assuming no correlation) a weighted average will give a 
back-of-the-envelope harm metric.

It will be orders of magnitude lower than 2%, even taking b) and e) alone.

But this still depends on the questions:
* "2% of flows,... over what period?" and
* where did 2% come from?

> stabilizes at too high a bitrate, which is plausible because of the

[BB] An aside: The large-screen TV, with its higher definition, will be 
wanting to take much higher bit-rate than any of the phones scrolling 
past 'thumbnail' videos. If your 100% utilization model were really 
true, the FQ would be forcing them all to have the same bit-rate and the 
large-screen would be sacrificing its quality, while the FQ would be 
encouraging the small screens to rise to their highest quality. This is 
another reason why 100% utilization is always avoided in practice (and 
why FQ would not be a sensible solution if 100% utilization was common).

>
> Maybe this is too pessimistic?

[BB] I believe it is ultra-pessimistic.

But allegedly "ultra" is a hype word that has been removed from my 
permitted vocabulary. So you're not allowed to understand what I mean, 
even though most English speakers would understand it to mean "extremely 
pessimistic" which is all I was trying to say ;)

> I'm assuming here that an L4S ABR will

[BB] Why assume that? We have figures from large numbers of experiments 
for the rate ratio when the capacity per flow is reduced.
Here's results for L4S vs Cubic over CoDel with different numbers of 
flows in the same queue (without the 3168 fall-back active):
https://l4s.net/ecn-fbk/results_v2.2/baseline/codel_release/#whisker_plots

>
> To me that reaches "already borderline unacceptable" for user experience,
> at a 0.3% scale, and then we remind ourselves that we started with super-
> generous assumptions that could make the scale much worse.
>
> I mean, some people do have vpns,

[BB] That is certainly true for existing FQ systems that are unlikely to 
be upgraded.
For those with upgradeable gateways, using ECN as part of the classifier 
is intended to address that problem.

> and maybe there's some ISPs that did
> turn on shared queue marking after all,

[BB] Yes, that's always a possibility. That's where l4sops comes in.

To be clear, I do not believe the techniques in l4sops to work round 
3168 will be necessary for FQ systems. And I think you will agree once 
we get this collision model closer to reality.

> plus we have good reason to
> should download an update, which would *really* crowd out a colliding flow
> if it's L4S.

[BB] Yes. E.g. some people do loads of torrents, etc.

> OTOH, maybe not all of the 0.3%+epsilon paths have 6-person
> families,

[BB] https://en.wikipedia.org/wiki/List_of_countries_by_number_of_households
Sort by number in household, or the % of households with 6+ members. I 
think you'll find you've succumbed to some exaggeration here. 
Particularly if you weight the figures by Internet penetration in each 
country.

> but then again there's also a fair few that share across

[BB] Certainly, IMO, the 100% utilization aspect is far more important. 
But even if we set utilization aside (and we really shouldn't) the no. 
of members per household is not just fine detail. Because birthday 
analysis is sensitive to small changes in the number of simultaneous 
sessions.

For instance, a weighted avg of the above wikipedia table is 3.9 members 
per household worldwide.{Note 2} And, even with your continuously 100% 
utilization scenario and with a simple hash, 4 sessions instead of 6-7 
brings the collision probability down from 2% to below 0.6%. With a 
sensible utilization level, the number of simultaneous sessions drops 
much lower, so I'm sure the collision probability will fall an order of 
magnitude lower, if not two orders.

{Note 2} I suspect if members per household were also weighted by 
Internet penetration, it would tip towards the no. of members per 
household in more affluent countries. For instance, in the top third of 
countries by affluence, 20-40% of households have 1 member. But let's 
not get hung up on these arguments, because it's the utilization figure 
that's critical.

> when I think about how they'll be tuning the alert thresholds to whatever

[BB] The idea is that operators analyse their monitoring data to 
/exclude/ all the cases where the CE-marking was due to FQ, because 
operators only need to act on the non-FQ cases.

...because, once we get the utilization model closer to reality, I'm 
convinced the hash collision problem will be tiny (without even needing 
code changes to be deployed, except probably for the VPN case).

> buffering video?  Those chillaxed family evenings get a lot less calm
> when people start screaming at each other to stop using the internet so
> they can watch their movie in peace, in my experience.
>
> NB: I'm using recognizable names here by way of example, but I think the
> problem gets worse again when you consider the long tail of niche content
> providers who maybe don't have a dedicated player-optimization team, in
> place of any one of the more famous services.

[BB] I think there's a huge misunderstanding here. I working on the 
basis of no expectation that CDN operators will need to do anything 
about the FQ cases. The monitoring is to find the non-FQ cases.

> If this kind of scenario can be shown to be too pessimistic by a couple
> orders of magnitude, I could see changing my mind if we find a way to get
> a handle on the scale of the VPN and shared queue questions.
>

[BB] Yes, understanding whether we're looking for
     A) solely non-FQ, or
     B) (non-FQ) AND (cases of FQ where collisions might lead to harm)
is why it's important to get the mental model right. Once it's right, 
I'm sure FQ collisions will be a tiny factor that operators don't have 
to act on (case A).

> using was the thrust of the needed response, but if you'd like me to
> saw when the queue changed from dualpi2 to pie, in the expectation this
> emulates roughly what we'd see from sharing a queue (either from fq
> collision or otherwise).
>
> This is just one an example of the "extreme unfairness" coexistence
> starting there.

[BB] OK, I can see the event in the video you're referring to now: at 
9:04 from the start. Yes - and obviously we have much more data on the 
distribution of that effect for all different link rates and RTTs, 
rather than relying on just one brief instant in a video. In fact the 
same link as before:
https://l4s.net/ecn-fbk/results_v2.2/baseline/codel_release/#whisker_plots

> to debate those points if we have substantial harm scenarios just from
> the FQ collisions, which I think might be able to cut a lot of wheel-
> spinning debate if it holds up.
>

[BB] Yes, I have been interpreting concerns about unfairness /within/ FQ 
(resulting from hash collisions) as contrived for the sake of debate. 
Whereas I see now that such concerns can be due to how people understand 
how flow arrivals, departures and congested periods relate to the 
sessions we see at the human level.

> I agree that queues could be upgraded with L4S support as part of the
> change without harm, and the flag day can be geared around deprecating
> the 3168 behavior of "Treat ECT(1) as ECT(0)", as opposed to specifying
> which specific ECT(1) behavior should be adopted.  Again, the harm
> doesn't actually start until the traffic starts flowing.
>
> using an ECT(1)->ECT(0) transition as a congestion signal, since it
> has the intractable performance issues you've outlined.)

[BB] I can understand how you're thinking now, and I hope you can see 
that, if a more realistic utilization model leads to a vanishingly small 
likelihood of collisions in FQ, the question of a flag-day is moot, 
unless there are large numbers of shared queue RFC3168 AQMs out there.

> /and/

[BB] Sorry, I didn't explain well. Including ECN in the flow classifier 
is primarily to address the VPN case.
On the assumption that a shallow ECT(1) threshold is also being rolled 
out (which inverts the unfairness problem in Classic's favour).

> needs to be an option, if the safety concerns on coexistence are

[BB] We're inexorably converging towards improved mutual understanding.

Bob

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Fri, 14 May 2021 14:35:03 +0200

Hi,

I find it somewhat odd to deprecate/revise a Proposed Standard in order 
to rollout an Experiment.

Moreover, where can I find statistics about ECN usage along Internet 
paths? I'm aware of 
https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-02.html, 
but are there others?
I'd expect many home routers to be the bottleneck, so their potential
CE marks wouldn't be visible in the Internet, but merely the
corresponding ECE flags in TCP packets.

Regards,
  Roland

On 09.05.21 at 05:28 Holland, Jake wrote:
> But this is also relevant as a response to this thread from tknowoday:

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 14 May 2021 22:24:27 +0300

> On 14 May, 2021, at 3:35 pm, Bless, Roland (TM) <[email-masked]> wrote:
> 

Additionally, if the traffic passing through them is Not-ECT, neither CE marks nor ECE flags would be visible.  Since the majority of endpoints are still not configured to initiate ECN negotiation, this is problematic for finding these deployments with either a passive monitor or an external active probe.

However, they would still have the same interaction between L4S and non-L4S traffic, even if the latter is Not-ECT, so they can't be dismissed just because they are difficult to see.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 14 May 2021 22:05:08 +0000

Hi Roland,

> to rollout an Experiment.

RFC 8311 also did this, I don't see this idea as especially different
on this point.  I mean sure, maybe it's odd, but sometimes the right
thing to do can be odd.

From the poll at the interim, there seems to be significant support in
the WG for moving this ahead regardless of the conflicts with existing
BCPs, which is about what I expected based on past discussions.  The
flag day proposal is just trying to look into one of the possibilities
for making that align better.

> https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-02.html, 

I did a presentation last year sharing a few observations I made:
https://www.youtube.com/watch?v=1326B7YYwLM&t=4338s
https://datatracker.ietf.org/meeting/interim-2020-maprg-01/materials/slides-interim-2020-maprg-01-sessa-latency-aqm-observations-on-the-internet-01.pdf

Those slides reference another presentation with some other numbers
collected differently:
https://www.youtube.com/watch?v=wKDgVSMUvis&t=32m23s
https://www.ietf.org/proceedings/98/slides/slides-98-maprg-tcp-ecn-experience-with-enabling-ecn-on-the-internet-padma-bhooma-00#page=12

Outside Pete's writeup that's all I've seen published yet.

> corresponding ECE flags in TCP packets.

That's what I did, yes.

Apple's measurements were from agents on the receive side IIUC and
I think they said included upload traffic, which I took to mean they
took both CE and ECE that didn't trigger their heuristics as evidence
of a CE-marking path.

HTH.

Best regards,
Jake

---

From: Greg White <[email-masked]>
Date: Mon, 17 May 2021 21:29:49 +0000

On 5/14/21, 6:35 AM, "tsvwg on behalf of Bless, Roland (TM)" <[email-masked] on behalf of [email-masked]> wrote:

I'm attempting to collect all known observations/statistics and summarize them in the L4Sops draft:
https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4sops-00.html#name-recent-studies

Also, Sebastian pointed to: https://ieeexplore.ieee.org/document/8909187 
But this paper seems less useful than the three listed in L4Sops currently.

-Greg

    Regards,
      Roland

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 18 May 2021 09:04:33 +0200

Hi Jake,

> on this point.  I mean sure, maybe it's odd, but sometimes the right
I agree.
> for making that align better.
Unfortunately, I missed the interim meeting.
> https://datatracker.ietf.org/meeting/interim-2020-maprg-01/materials/slides-interim-2020-maprg-01-sessa-latency-aqm-observations-on-the-internet-01.pdf
I missed that.
> Outside Pete's writeup that's all I've seen published yet.

Thanks for the pointers!

> took both CE and ECE that didn't trigger their heuristics as evidence

Thanks a lot!

Regards,

  Roland