Subject: [tsvwg] Fwd: New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-00.txt

From: Pete Heist <[email-masked]>
Date: Thu, 18 Feb 2021 17:38:27 +0100

> A new version of I-D, draft-heist-tsvwg-ecn-deployment-observations-00.txt
> has been successfully submitted by Peter G. Heist and posted to the
> IETF repository.
> 
> Name:		draft-heist-tsvwg-ecn-deployment-observations
> Revision:	00
> Title:		Explicit Congestion Notification (ECN) Deployment Observations
> Document date:	2021-02-18
> Group:		Individual Submission
> Pages:		27
> URL:            https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-heist-tsvwg-ecn-deployment-observations/
> Html:           https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.html
> Htmlized:       https://tools.ietf.org/html/draft-heist-tsvwg-ecn-deployment-observations-00
> 
> Abstract:
>   This note presents data gathered at an Internet Service Provider's
>   gateway on the observed deployment and usage of ECN.  Relevant IP
>   counter and flow tracking data was collected and analyzed for TCP and
>   other protocols.

This draft adds some data on the current usage of ECN. It was gathered over several weeks at a cooperative ISP with around 660 members, and looks at ECN endpoint activity, AQM deployment and ECN usage on non-TCP protocols. While this study is still relatively small, it’s hopefully at least a little more useful than the stateless counter data I posted late last year, which should set the bar suitably low… :)

Pete

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 19 Feb 2021 11:44:12 +0000

Hi Pete,

thanks for putting this together and sharing!

I have one question on the data. Maybe I'm not readying this correctly but if I look at the big table at the end, then I see a lot of cases where there are much more CE marks than ECT(0) marks. That's a bit unexpected as usually an AQM should only mark a small portion of the packets. Or do I interpret the data there incorrectly?

Mirja

    Pete

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 19 Feb 2021 13:48:41 +0200

> On 19 Feb, 2021, at 1:44 pm, Mirja Kuehlewind <mirja.kuehlewind=[email-masked]> wrote:
> 

We believe that's evidence of the ECN field being used incorrectly by some UDP-based applications.  This is discussed in the draft and could be due to people putting values intended for the DSCP field (6 bits, left justified) or the obsolete Precedence-TOS fields (3+4 bits, left justified) directly into the TOS byte.

For that reason, we only count TCP and what are probably tunnelled TCP traffic as evidence of actual ECN and AQM deployment.

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Fri, 19 Feb 2021 03:54:54 -0800

What stood out for me was the turn data.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Feb 2021 12:50:41 +0000

Pete,

Thank you for this work.

Given you have close contact with this ISP, have you asked, or could you 
ask them whether they have ever deployed any FIFO ECN-enabled AQMs 
themselves?

Cheers

Bob

On 18/02/2021 16:38, Pete Heist wrote:
>> URL:https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.txt
>> Status:https://datatracker.ietf.org/doc/draft-heist-tsvwg-ecn-deployment-observations/
>> Html:https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.html
>> Htmlized:https://tools.ietf.org/html/draft-heist-tsvwg-ecn-deployment-observations-00
>>
>> Abstract:
> This draft adds some data on the current usage of ECN. It was gathered over several weeks at a cooperative ISP with around 660 members, and looks at ECN endpoint activity, AQM deployment and ECN usage on non-TCP protocols. While this study is still relatively small, it’s hopefully at least a little more useful than the stateless counter data I posted late last year, which should set the bar suitably low…:)
>
> Pete
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Pete Heist <[email-masked]>
Date: Fri, 19 Feb 2021 14:06:11 +0100

Yes, that was somewhat unexpected to us at first too, but misuse of the
ECN field esp. on non-TCP packets seems to explain it in cases where
there ratios don't look like AQM signaling. The data for TCP is overall
easier to attribute to ECN when you can see feedback via ECE.

I didn't editorialize much on the results, but I'll use this as a
chance to add what struck me:

* While ECN negotiations were relatively small at around 1.44%, they
were spread across 45% of the IPs, so the proportion of paths using it
seems significant.

* The data seem to show significant 3168 marking AQM deployment, when
24% of LAN IPs that negotiated ECN saw CE or ECE. The draft mentions
how some of it is from known AQM instances and some not. Congestion
overall doesn't seem excessive, but that could be quantified better.

* A wider survey might help on the non-TCP data.

Anyway thanks for taking a look...

Pete

On Fri, 2021-02-19 at 11:44 +0000, Mirja Kuehlewind wrote:
> I have one question on the data. Maybe I'm not readying this
> correctly but if I look at the big table at the end, then I see a lot
> of cases where there are much more CE marks than ECT(0) marks. That's
> a bit unexpected as usually an AQM should only mark a small portion
> of the packets. Or do I interpret the data there incorrectly?
> 
> Mirja
> 
> 
> 
> On 18.02.21, 17:38, "tsvwg on behalf of Pete Heist"
> <[email-masked] on behalf of [email-masked]> wrote:
> 
> 
>     > A new version of I-D, draft-heist-tsvwg-ecn-deployment-
> observations-00.txt
>     > has been successfully submitted by Peter G. Heist and posted to
> the
>     > IETF repository.
>     > 
>     > Name:             draft-heist-tsvwg-ecn-deployment-observations
>     > Revision: 00
>     > Title:            Explicit Congestion Notification (ECN)
> Deployment Observations
>     > Document date:    2021-02-18
>     > Group:            Individual Submission
>     > Pages:            27
>     > URL:           
> https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.txt
>     > Status:        
> https://datatracker.ietf.org/doc/draft-heist-tsvwg-ecn-deployment-observations/
>     > Html:          
> https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-00.html
>     > Htmlized:      
> https://tools.ietf.org/html/draft-heist-tsvwg-ecn-deployment-observations-00
>     > 
>     > Abstract:
>     >   This note presents data gathered at an Internet Service
> Provider's
>     >   gateway on the observed deployment and usage of ECN. 
> Relevant IP
>     >   counter and flow tracking data was collected and analyzed for
> TCP and
>     >   other protocols.
> 
>     This draft adds some data on the current usage of ECN. It was
> gathered over several weeks at a cooperative ISP with around 660
> members, and looks at ECN endpoint activity, AQM deployment and ECN
> usage on non-TCP protocols. While this study is still relatively
> small, it’s hopefully at least a little more useful than the
> stateless counter data I posted late last year, which should set the
> bar suitably low… :)
> 
>     Pete
> 
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 19 Feb 2021 15:34:07 +0200

> On 19 Feb, 2021, at 2:50 pm, Bob Briscoe <[email-masked]> wrote:
> 
> Given you have close contact with this ISP, have you asked, or could you ask them whether they have ever deployed any FIFO ECN-enabled AQMs themselves? 

As far as I'm aware, they have only used fq_codel for ECN.  Before that, they were using SFQ in some places, without any AQM.

However, I would remind you that neither SFQ nor fq_codel can distinguish between flows carried inside an encrypted tunnel, so this cannot be relied upon alone to make L4S safe.  Pete's data shows significant tunnelled traffic, much of which is probably due to people working from home and using VPNs to access a corporate network.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Fri, 19 Feb 2021 21:08:13 +0100

Hi Bob, I got a reply back from one admin that to his knowledge they
never used single queue AQMs in this environment. Jon summed up the
rest...

Regards,
Pete

On Fri, 2021-02-19 at 12:50 +0000, Bob Briscoe wrote:
> Pete,
> 
> Thank you for this work.
> 
> Given you have close contact with this ISP, have you asked, or could
> you ask them whether they have ever deployed any FIFO ECN-enabled
> AQMs themselves? 
> 
> Cheers
> 
> 
> Bob
> 
> > URL:
> > Status:
> > Html:
> > Htmlized:

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Feb 2021 20:43:02 +0000

Thanks

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Feb 2021 21:36:36 +0000

Jonathan,

> As far as I'm aware, they have only used fq_codel for ECN.  Before that, they were using SFQ in some places, without any AQM.

[BB] So do you suspect that any of the CE-marking in traffic not 
traversing their backhaul FQ-CoDel boxes is from a FIFO ECN-enabled AQM?

Or is this ISP particularly attractive to users into free and 
open-source software, in which case I would have thought the CE marking 
is most likely to be coming from FQ-CoDel? Or can we just not really tell?

As the rest of your email changed subject to tunnels, I've split that 
into a separate thread.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Feb 2021 21:54:52 +0000

Jonathan,

> However, I would remind you that neither SFQ nor fq_codel can distinguish between flows carried inside an encrypted tunnel, so this cannot be relied upon alone to make L4S safe.  Pete's data shows…

[BB] So would you not advise this ISP to remove the FQ on their 
backhaul, given they are serving large amounts of tunnelled VPN traffic?

The degree of flow rate inequality that FQ causes in this scenario is 
unbounded. For instance, if someone was running a torrent of N flows 
within their VPN, each of their flows would be squeezed to 1/N of the 
capacity of each of all the other non-tunnelled long-running flows. N is 
pretty much  unbounded.

There's no L4S there of course. The FQ is quite capable of wreaking this 
havoc all on its own. This isn't the first time you've taken a 
well-known failing of FQ, then dropped L4S into the FQ in an attempt to 
deflect the pre-existing failing of FQ onto L4S. You don't need to do 
this. Everyone knows L4S has a potential problem in Classic ECN AQMs. We 
laid that out from the very first day we brought L4S to the IETF. 
Debating games like this just turn people off.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 19 Feb 2021 23:56:09 +0200

> On 19 Feb, 2021, at 11:36 pm, Bob Briscoe <[email-masked]> wrote:
> 

> [BB] So do you suspect that any of the CE-marking in traffic not traversing their backhaul FQ-CoDel boxes is from a FIFO ECN-enabled AQM?
> 
> Or is this ISP particularly attractive to users into free and open-source software, in which case I would have thought the CE marking is most likely to be coming from FQ-CoDel? Or can we just not really tell?

They're a co-operative ISP, which means they have a smallish hardware budget and want to make the most of it in terms of utility to the customer.  That is why they've deployed fq_codel in the places where that's feasible so far.

We have no insight at present into the nature of observed ECN activity from elsewhere, except for an observation that some ECN traffic is carried through tunnels.  We do consider it *likely*, however, that most of it is flow-aware AQM of some description, simply because practical AQM deployment has been driven by the ease of fq_codel configuration.

> As the rest of your email changed subject to tunnels, I've split that into a separate thread.

I mentioned tunnels because I infer your aim is to discredit the argument, made more than a year ago, that legacy AQMs may exist that don't also perform flow isolation, and that this is problematic for the L4S safety case.  Encrypted tunnels that intentionally hide the distinction between flows carried within them exhibit similar behaviour, and it's easier to show that they exist in significant numbers.

There is no need to make a new thread about that unless you have something genuinely new to say about tunnels.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 20 Feb 2021 00:11:22 +0200

> On 19 Feb, 2021, at 11:54 pm, Bob Briscoe <[email-masked]> wrote:
> 
> [BB] So would you not advise this ISP to remove the FQ on their backhaul, given they are serving large amounts of tunnelled VPN traffic?

No, of course not.  Max-min fairness is the gold standard, after all, and that is what FQ is designed to provide at a saturated bottleneck.  I honestly don't understand why you keep arguing, in effect, for RTT-fairness while promoting a specification which mandates working to eliminate it.

At a bottleneck which is *not* persistently saturated, FQ serves to insulate latency-sensitive flows from bursty ones, and sparse flows from spurious AQM activity sparked by transient saturating loads.  Are these not good things, even absent any rigorous notion of "fairness"?

It is possible, of course, that improved *perceived* max-min fairness would be achieved by moving to an algorithm that is fair between subscriber IP addresses as well as between flows to each subscriber.  Cake is capable of doing that, and I have plans to upgrade some of my other qdiscs to match.  Would that solve your objection to flow isolation?

> Debating games like this just turn people off.

Yes, they do - and in recognition of that, perhaps we could avoid going over old ground like this, yet again.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 19 Feb 2021 23:26:08 +0000

Jonathan,

> At a bottleneck which is *not* persistently saturated, FQ serves to insulate latency-sensitive flows from bursty ones, and sparse flows from spurious AQM activity sparked by transient saturating lo…

[BB] On a highly aggregated backhaul link that is only seeing transient 
queuing (probably of the order of 10 or 100 us), there's little need for 
an AQM at all.

But if this backhaul link is capable of being saturated by the sum of 
the access links (i.e. it's theoretically overbooked but relying on 
average usage patterns to remain under-utilized), it would be preferable 
to deploy a FIFO AQM there. With an FQ AQM, if a large proportion of the 
traffic is in VPNs, just when the AQM is most needed, it would makes 
matters worse (i.e. all the flows within the VPNs would get a focused 
hit on their throughput, so they would collectively cause massive queues 
within each per-flow-queue being used by each VPN).

> It is possible, of course, that improved *perceived* max-min fairness would be achieved by moving to an algorithm that is fair between subscriber IP addresses as well as between flows to each subsc…

[BB] You are talking about instantaneous rate fairness, with no concept 
of occupancy over time. There's been so much more water under this 
bridge since the 1980s/90s. Try this: 
https://bobbriscoe.net/presents/1210isoc/1210isoc-briscoe.pdf

Clearly you never read/absorbed what I wrote for you (specifically for 
you more than anyone else) about this:
Per-Flow Scheduling and the End-to-End Argument: 
https://bobbriscoe.net/pubs.html#FQ_v_E2E

I can tell 'cos:
a) you somehow think I am arguing for RTT-fairness (when have I ever 
done that?) and
b) the assertion that max-min fairness is the gold standard. If you had 
read anything from me, you would at least make a nod to my explanations 
about why max-min only makes any sense during a persistent famine (I 
mean famine for hours/days/weeks), and in all other scenarios max-min is 
the opposite of gold - it's the most rubbish monkey-metal crazy standard 
in a non-famine world.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 20 Feb 2021 01:59:41 +0200

> On 20 Feb, 2021, at 1:26 am, Bob Briscoe <[email-masked]> wrote:
> 
> You are talking about instantaneous rate fairness, with no concept of occupancy over time. There's been so much more water under this bridge since the 1980s/90s. Try this: https://bobbriscoe.net/presents/1210isoc/1210isoc-briscoe.pdf
> 
> Clearly you never read/absorbed what I wrote for you (specifically for you more than anyone else) about this:
> Per-Flow Scheduling and the End-to-End Argument: https://bobbriscoe.net/pubs.html#FQ_v_E2E

I've read your papers already.  I happen to disagree vigorously with their conclusions, and I believe others with far more standing than myself also do so.

> I can tell 'cos:
> a) you somehow think I am arguing for RTT-fairness (when have I ever done that?) and

I'll quote you:  "…it would be preferable to deploy a FIFO AQM there."

If you pass AIMD traffic through a single FIFO with a single AQM instance, what you get is a (slow) convergence towards equal cwnd values and thus RTT-fairness.  That is not a controversial statement, but the truth.  And that is what you explicitly recommended just a few sentences before asking this question.

> b) the assertion that max-min fairness is the gold standard. If you had read anything from me, you would at least make a nod to my explanations about why max-min only makes any sense during a persistent famine (I mean famine for hours/days/weeks), and in all other scenarios max-min is the opposite of gold - it's the most rubbish monkey-metal crazy standard in a non-famine world.

That is precisely where I'm aware of your point of view, but diametrically opposed to it.

At this stage I can only recommend that you install Cake, turn on the host-and-flow fairness options, and just try it out for a while.  You might be pleasantly surprised by how well it works in practice.  In fact, until you have done so, I will simply not respond to any further posts by you on this subject.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 20 Feb 2021 00:44:46 +0000

Jonathan,

Please engage in reasoned discussion. Particularly about occupancy over 
time.
If you've read what I've written and you disagree, please talk about it 
in a calm, measured manner, to argue against the actual points made, 
instead of asserting that you disagree, then threatening to run away 
with the ball unless I run some software you've written.

I'm going to take the list off future emails, 'cos this is not edifying 
for anyone else.

one inline...

> If you pass AIMD traffic through a single FIFO with a single AQM instance, what you get is a (slow) convergence towards equal cwnd values and thus RTT-fairness.  That is not a controversial stateme…

[BB] There is no truth there, because the RTT-Fairness doesn't come from 
the FIFO. It comes from the assumption you've just added - that an 
ACK-clocked AIMD is running in all the senders. Anything can be run on 
the end-systems. You know I've argued that that is a "good thing".

It might help if you tried to summarize what you understand from the 
above paper about "flow sched and the e2e arg".
But pls, remove the list. The worst way to find why two people are 
disagreeing is to play to a crowd in the process.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 20 Feb 2021 00:57:51 +0000

Jonathan,

On 19/02/2021 23:26, Bob Briscoe wrote:
>>>> However, I would remind you that neither SFQ nor fq_codel can 
>>>> distinguish between flows carried inside an encrypted tunnel, so 
>>>> this cannot be relied upon alone to make L4S safe.  Pete's data 
>>>> shows significant tunnelled traffic, much of which is probably due 
>>>> to people working from home and using VPNs to access a corporate 
>>>> network.
>> No, of course not.  Max-min fairness is the gold standard, after all, 
>> and that is what FQ is designed to provide at a saturated 
>> bottleneck.  I honestly don't understand why you keep arguing, in 
>> effect, for RTT-fairness while promoting a specification which 
>> mandates working to eliminate it.
>>
>> At a bottleneck which is *not* persistently saturated, FQ serves to 
>> insulate latency-sensitive flows from bursty ones, and sparse flows 
>> from spurious AQM activity sparked by transient saturating loads.  
>> Are these not good things, even absent any rigorous notion of 
>> "fairness"?
>
> [BB] On a highly aggregated backhaul link that is only seeing 
> transient queuing (probably of the order of 10 or 100 us), there's 
> little need for an AQM at all.
>
> average usage patterns to remain under-utilized), it would be 
> preferable to deploy a FIFO AQM there. With an FQ AQM, if a large 
> proportion of the traffic is in VPNs, just when the AQM is most 
> needed, it would makes matters worse (i.e. all the flows within the 
> VPNs would get a focused hit on their throughput, so they would 
> collectively cause massive queues within each per-flow-queue being 
> used by each VPN). 

Here is an algorithm to find the truth. Take one of your emails, and do 
a diff with the previous email in the thread. Then the other person's 
text that you silently deleted (as above) will invariably be the truth.

Bob

---

From: Dave Taht <[email-masked]>
Date: Fri, 19 Feb 2021 17:21:45 -0800

It takes an awful lot here for me to bother to reply to a thread.

> Here is an algorithm to find the truth. Take one of your emails, and do a diff with the previous email in the thread. Then the other person's text that you silently deleted (as above) will invariably be the truth.

Here is a better algorithm to find the truth. Purchase any of dozens
of commercial routers today running cake or fq_codel, or the thousands
available via reflash to openwrt/dd-wrt/tomato construct a repeatable
experiment, and publish the code and results. The results from
running, repeatable, code trumps theoretical objections every time.

The evenroute v3 and edgerouter X series are pretty good bases for
experimentation.

Wilful ignorance, and the lack of a willingness to construct
repeatable experiments is not science. If you have a point to make,
make it with a repeatable experiment against running code, please.

Tuning out again.

-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

[email-masked] <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 20 Feb 2021 09:36:11 +0000

Pete,

If you plan to post a revision, in the TCP table (Sec.5.3) would you pls 
consider including the counts of ECT packets, as in the non-TCP table in 
Sec.5.3? Otherwise we don't know what percentage of the total ECN 
traffic each count represent.

I'm trying to separate out the IPs known to be behind the FQ-CoDel nodes 
in the backhaul. I.e. separating known from unknown causes.

Separately, a count of total packets, or equivalently a count of 
not-ECT, would be v useful too.

If you can do this without losing the ECE counts, even better. Having CE 
and ECE together enables a rough estimate of the average window.{Note 1}

Specifically,
     avgWindow = ECE * delAckRatio / CE

Thank you (again)

Bob

{Note 1} And lots of other assumptions, such as:
- delayed Ack Ratio including stretch ACKs has to be guessed (e.g. 2???)
- minimal additional ack coalescing in the network
- minimal reordering
- traffic volume dominated by elephants
- the TCP protocol is working as it should
- and so on.

---

From: Pete Heist <[email-masked]>
Date: Sat, 20 Feb 2021 14:39:37 +0100

New revision posted and some comments below...

> traffic each count represent.

That's a useful addition. An ECT(0) column is now added, and the TCP
table's column order is now similar to the non-TCP table. ECE for the
opposite direction is still next to CE as that seems most readable. I
left out ECT(1) for TCP as there's so little of that, and then it gets
harder with the line length limit.

> in the backhaul. I.e. separating known from unknown causes.

That's mentioned in Section 3.2.

> not-ECT, would be v useful too.

I didn't capture Not-ECT by IP address, mainly because some testing
made me question how at least one ipset lookup per-packet would affect
performance on their production gateway. I added that to the
Limitations section as it could be useful if available.

Pete

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 20 Feb 2021 18:42:57 +0000

Dave, Jonathan

Getting back to the subject. 

Quoting Jonathan " However, I would remind you that neither SFQ nor fq_codel can distinguish between flows carried inside an encrypted tunnel, so this cannot be relied upon alone to make L4S safe.  Pete's data shows significant tunnelled traffic, much of which is probably due to people working from home and using VPNs to access a corporate network."

This line of reasoning makes me confused.
Section 6.2 in RFC8290 ( https://tools.ietf.org/html/rfc8290#section-6.2) clearly says that fq-codel will not deliver its intended benefits for encrypted tunnels. If ISP's still rely on fq-codel to do it´s job well even though a substantial share of the traffic is VPN, then that is of course a problem. But I see that only as an fq-codel problem.

So what I cannot at all understand.. Why should a known problem (or read design feature) in fq-codel (or SFQ) be interpreted as something that makes L4S unsafe ?. 

/Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Dave Taht
> Sent: den 20 februari 2021 02:22
> To: Bob Briscoe <[email-masked]>
> Cc: TSVWG <[email-masked]>
> Subject: Re: [tsvwg] FQ & VPNs
> 
> > Here is an algorithm to find the truth. Take one of your emails, and do a diff
> with the previous email in the thread. Then the other person's text that you
> silently deleted (as above) will invariably be the truth.
> 
> Here is a better algorithm to find the truth. Purchase any of dozens of
> commercial routers today running cake or fq_codel, or the thousands
> experiment, and publish the code and results. The results from running,
> repeatable, code trumps theoretical objections every time.
> 
> Wilful ignorance, and the lack of a willingness to construct repeatable
> experiments is not science. If you have a point to make, make it with a
> repeatable experiment against running code, please.
> 
> Tuning out again.
> 
> >
> >
> > Bob
> >
> > --
> >
> > Bob Briscoe
> https://protect2.fireeye.com/v1/url?k=e8b458ab-b72f6049-e8b41830-
> 86073b36ea28-0086813e29ae2dfb&q=1&e=e1b27d62-c59c-43ba-b512-
> e8cf13c35f2c&u=http%3A%2F%2Fbobbriscoe.net%2F
> 
> 
> 
> --

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 20 Feb 2021 23:28:45 +0200

> On 20 Feb, 2021, at 8:42 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

I will just note here that although treating a VPN containing multiple flows as a single flow results in a reduction in throughput, when it has to compete against other traffic at a saturated bottleneck, this is a relatively minor and infrequent event at the ISP in question.  

The degree of impairment experienced by the VPN is also strictly bounded by the number of saturating flows carried within it.  The VPN user is thus in a position to predict, understand, and possibly mitigate the effect by splitting the traffic across multiple VPN flows, if performance turns out to be more important than hiding that piece of information.

> So what I cannot at all understand.. Why should a known problem (or read design feature) in fq-codel (or SFQ) be interpreted as something that makes L4S unsafe ?. 

The problem is entirely due to L4S flows' harm to conventional AIMD flows when passing through the same queue and AQM instance, unless the AQM is of a type that specifically understands L4S.  This is a predictable and inevitable consequence of L4S flows responding qualitatively less (by what is effectively an Additive Decrease, despite claims to the contrary) to each CE mark than defined in either RFC-3168 or RFC-8511 (which both mandate a Multiplicative Decrease to a single CE mark).

Consequently, if L4S and AIMD flows share a single VPN tunnel, which passes through a conventional ECN AQM at the bottleneck, the AIMD flows will collapse to a very low throughput.  It is crucial to understand here that all existing ECN-marking AQMs deployed in the Internet today - most of which are fq_codel or similar - fall into this category.

I attach an example of the likely consequences.  The red trace at the top is L4S throughput; the light blue trace at the very bottom is the AIMD flow starting ten seconds later.  You may refer to previous data sets published by Pete Heist and myself for further examples.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 21 Feb 2021 02:20:30 +0100

Ingemar, List,

let me note that a tunneled connection as you seem to describe, will either yield and behave like a single flow at any AQM/bottleneck or will risk to look like a non- or  under-responsive flow that violates the internet's assumption of rough equitable sharing of capacity (TCP-friendlyness).

	Sure, the tunnel end-points might know that the tunnel carries X flows and hence might consider it desirable and "fair" to get roughly X/N throughput on a congested node with N true flows, but how is the intermittent node expected to figure this out and still assure rough sharing?

	Are you really proposing that all bottleneck nodes should give the least responsive flows most capacity? That seems like an ideal strategy to be 
exploited/gamed by all transport protocols, leading to a race to congestion collapse. 

	As before, I am happy to concede that approximate flow fair queueing is almost never optimal, but also rarely pessimal, and without additional information probably the most deterministic/predictable behavior an intermittent node can choose. 

	In theory we could use the IPv6 flow label to pass information about flow identities to the outside of tunneled packets, in which case FQ will work reasonably well (this is harder to game, unless the traffic is completely re-ordering tolerant, as an FQ system only maintains order inside each "flow", but is free to re-order between flows).

	None of that discussion seems novel or particularly pertinent at this point in L4S' evolution though; how did we end up in this side-discussion again? Ah, yes, L4S' lack of safety by design; the point is that the argument has been made multiple times that L4S' under responsiveness to CE marks could be ignored assuming most all rfc3168 AQM employ FQ and hence the fall out of L4S's wrong interpretation of CE marks would be restricted to the (stochastic) queue used mostly be the L4S flow itself and hence would not cause harm to other traffic. Which has two issues, one: due to the birthday paradox hash collisions are not that rare, that L4S will cause harm to non-L4S bystander flows that happen to hash into the same queue, and the whole thing goes pear shaped with tunneled traffic containing a mix of L4S and non L4S packets. 

Best Regards
	Sebastian

> On Feb 20, 2021, at 19:42, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Dave, Jonathan
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 21 Feb 2021 07:04:26 +0000

Hi

Trying to summarize. Your concern is that the L4S flows starve out the non-L4S flows when they share the same VPN tunnel. 

You proposed the solution yourself below:  

“The VPN user is thus in a position to predict, understand, and possibly mitigate the effect by splitting the traffic across multiple VPN flows, if performance turns out to be more important than hiding that piece of information”. 

The same rationale can be used to put the L4S flows in a separate VPN tunnel, problem solved.

And then, there is always a possibility to upgrade the AQMs to support L4S as well ?. It always appear from your argumentation that fq-codel is some kind of untouchable monolith that cannot at any cost be modified, unless of course you see a possibility that VPN tunnels can somehow leak inner header information (see separate discussion around draft-ietf-tsvwg-transport-encrypt)

With that said.. I believe that Bob is right. It benefits the TSVWG group better that we continue this discussion off list. It is just the same old arguments over and over again, albeit packaged in different wording. 

/Ingemar

From: Jonathan Morton <[email-masked]> 
Sent: den 20 februari 2021 22:29
To: Ingemar Johansson S <[email-masked]>
Cc: Dave Taht <[email-masked]>; Bob Briscoe <[email-masked]>; TSVWG <[email-masked]>
Subject: Re: [tsvwg] FQ & VPNs

On 20 Feb, 2021, at 8:42 pm, Ingemar Johansson S <[email-masked] <mailto:[email-masked]> > wrote:

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Sun, 21 Feb 2021 04:00:08 -0800

Participating reluctantly, as usual.

On Sat, Feb 20, 2021 at 11:04 PM Ingemar Johansson S <
[email-masked]> wrote:

> Hi
>
>
>
> Trying to summarize. Your concern is that the L4S flows starve out the
> non-L4S flows when they share the same VPN tunnel.
>
> “The VPN user is thus in a position to predict, understand, and possibly
> mitigate the effect by splitting the traffic across multiple VPN flows, if
> performance turns out to be more important than hiding that piece of
> information”.
>

In terms of VPNs there are multiple other problems. Site-site, bare metal,
running fq_codel over ipsec

> And then, there is always a possibility to upgrade the AQMs to support L4S
> as well ?. It always appear from your argumentation that fq-codel is some
> kind of untouchable monolith that cannot at any cost be modified, unless of
> course you see a possibility that VPN tunnels can somehow leak inner header
> information (see separate discussion around
> draft-ietf-tsvwg-transport-encrypt)
>
>
>

It's not an untouchable monolith. It is merely the default qdisc in nearly
every linux distribution at this point, ios, and osx. It is used by 10s,
maybe 100s of millions of devices at this point. It is part of billions of
containers. With nearly 9 years of deployment into an ever increasing
number of embedded devices that typically lag 5 or more years behind the
mainline OSes. Making changes to how ecn is interpreted would take 8+ years
to flush out of the field.

Total (documented) L4S market penetration of its mis-interpretation of the
CE field - 0. I might be kind and say "alternate", rather than mis, but
slamming L4S through a rfc3168 fq_codel'd system does have the documented
effects so often discussed here.

The ce_threshold parameter has been part of fq_codel for many years also.
Tweaking that to instead trigger off of ECT(1) is straightforward. Without
clear data - which we didn't have at the time of this debate hitting public
mailing lists - doing that seemed dicy - and there was indeed a
encapsulation bug regarding that transition that was found and fixed a
while back which will take years to propagate to the field, also.

The SCE alternative is a backward compatible interpretation of the ECN
related bits - and fails safe -  and could have gone into wider
distribution 2 years ago with near zero impact on end-user OSes, and as
gradual upgrade, and been (well, actually, has, as fq_codel_fast and the
alternate cake have been tested by a few parties at this point) tested as
to other impacts. Coupled also with a DSCP field a la "nqb" (and preferably
a few other dscp bits) it could be effective, sooner.

I have been tempted a lot to at least make SCE available as an option in
the upcoming openwrt release, but neither side has made a convincing
argument that doing this to ECN at all makes sense, and in fact the flood
of negative data on it (rtt unfairness in particular), makes me more
inclined to just recommend ecn be disabled entirely.

> With that said.. I believe that Bob is right. It benefits the TSVWG group
> better that we continue this discussion off list. It is just the same old
> arguments over and over again, albeit packaged in different wording.
>
>
>
> /Ingemar
>
>
>
> *From:* Jonathan Morton <[email-masked]>
> *Sent:* den 20 februari 2021 22:29
> *To:* Ingemar Johansson S <[email-masked]>
> *Cc:* Dave Taht <[email-masked]>; Bob Briscoe <[email-masked]>;
> TSVWG <[email-masked]>
> *Subject:* Re: [tsvwg] FQ & VPNs
>
>
>
> Quoting Jonathan " However, I would remind you that neither SFQ nor
> fq_codel can distinguish between flows carried inside an encrypted tunnel,
> so this cannot be relied upon alone to make L4S safe.  Pete's data shows
> significant tunnelled traffic, much of which is probably due to people
> working from home and using VPNs to access a corporate network."
>
> Section 6.2 in RFC8290 ( https://tools.ietf.org/html/rfc8290#section-6.2)
> clearly says that fq-codel will not deliver its intended benefits for
> encrypted tunnels. If ISP's still rely on fq-codel to do it´s job well even
> though a substantial share of the traffic is VPN, then that is of course a
> problem. But I see that only as an fq-codel problem.
>
>
>
> I will just note here that although treating a VPN containing multiple
> flows as a single flow results in a reduction in throughput, when it has to
> compete against other traffic at a saturated bottleneck, this is a
> relatively minor and infrequent event at the ISP in question.
>
>
>
> The degree of impairment experienced by the VPN is also strictly bounded
> by the number of saturating flows carried within it.  The VPN user is thus
> in a position to predict, understand, and possibly mitigate the effect by
> splitting the traffic across multiple VPN flows, if performance turns out
> to be more important than hiding that piece of information.
>
>
>
> So what I cannot at all understand.. Why should a known problem (or read
> design feature) in fq-codel (or SFQ) be interpreted as something that makes
> L4S unsafe ?.
>
>
>
> The problem is entirely due to L4S flows' harm to conventional AIMD flows
> when passing through the same queue and AQM instance, unless the AQM is of
> a type that specifically understands L4S.  This is a predictable and
> inevitable consequence of L4S flows responding qualitatively less (by what
> is effectively an Additive Decrease, despite claims to the contrary) to
> each CE mark than defined in either RFC-3168 or RFC-8511 (which both
> mandate a Multiplicative Decrease to a single CE mark).
>
>
>
> Consequently, if L4S and AIMD flows share a single VPN tunnel, which
> passes through a conventional ECN AQM at the bottleneck, the AIMD flows
> will collapse to a very low throughput.  It is crucial to understand here
> that all existing ECN-marking AQMs deployed in the Internet today - most of
> which are fq_codel or similar - fall into this category.
>
>
>
> I attach an example of the likely consequences.  The red trace at the top
> is L4S throughput; the light blue trace at the very bottom is the AIMD flow
> starting ten seconds later.  You may refer to previous data sets published
> by Pete Heist and myself for further examples.
>
>
>
>  - Jonathan Morton
>
>
>
>

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 21 Feb 2021 19:49:57 +0200

> On 21 Feb, 2021, at 9:04 am, Ingemar Johansson S <[email-masked]> wrote:
> 

In the current situation, where the flows sharing the tunnel do so on a roughly equal basis, the choice is an *optional* performance enhancement, to be undertaken only when actual experience shows it to be necessary.  Usually it is not enough of a problem to bother with the hassle - and it *is* a hassle, because by default VPNs do not work that way.  I mentioned it only to illustrate that the VPN user has some agency and is reasonably able to predict what will happen, and know that it won't be too bad either way.

Adding L4S to the mix means that the flows no longer share the tunnel on anything even approximating an equal basis.  A single L4S flow will bully all others out of its way, as soon as the bottleneck is at a conventional ECN AQM (whether with FQ, AF, or not).  It does not even matter whether the conventional flows support ECN themselves; if not, the conventional AQM will simply drop their packets at the same rate that it would have marked them, producing the same effect on congestion control and throughput.

Which means that separating the L4S flows out to a separate tunnel becomes *necessary*, not optional, to retain anything resembling normal performance.  And this would *not* be helped by deleting FQ from the AQM in question, as Bob suggested earlier; that would only permit the L4S flow to bully flows outwith the tunnel that happen to traverse the same bottleneck, rather than its harm being contained to the tunnel it occupies.

While the VPN user might reasonably be able to estimate the number of flows running through his tunnel, he might not be aware of the particularly drastic effect that L4S traffic would have.  I think it likely that his troubleshooting will lead to a conclusion that the L4S traffic is "unresponsive" to congestion control signals, immediate shutdown of the L4S traffic source, and a strongly worded letter of complaint to the vendor(s) perceived to be responsible for it.

> And then, there is always a possibility to upgrade the AQMs to support L4S as well ?

To be deployable outside of carefully prepared networks, L4S *MUST* be capable of coexisting, at default settings, with conventional traffic in existing networks.  That is mandatory, not optional.  Pete's new data is prima facie evidence that fq_codel (and other ECN AQMs) is a feature of existing networks.  I'm sure you can work out the logical conclusion to that equation.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 21 Feb 2021 18:48:53 +0000

Hi

The question.. fq-codel took 8 years to deploy, does this mean that it takes 8 more years (or 5 or whatever) to roll out fq-codel that supports L4S ?. Note that we have by now spent almost two years arguing on the TSVWG list.  These two years could have been spent more wisely, for instance on studies on how to integrate L4S in fq-codel (or the like). 

And also.. even though I am enthusiastic about L4S, I don’t expect it to happen overnight. This would give you time to update fq-codel for L4S and have it deployed. Meanwhile algorithms that fallback to RFC3168 ECN and/or separate VPN tunnels for L4S traffic can do some damage control. 

/Ingemar 

From: Dave Taht <[email-masked]> 
Sent: den 21 februari 2021 13:00
To: Ingemar Johansson S <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; Bob Briscoe <[email-masked]>; TSVWG <[email-masked]>
Subject: Re: [tsvwg] FQ & VPNs

On Sat, Feb 20, 2021 at 11:04 PM Ingemar Johansson S <[email-masked] <mailto:[email-masked]> > wrote:

Hi

 

In terms of VPNs there are multiple other problems. Site-site, bare metal, running fq_codel over ipsec

 

It's not an untouchable monolith. It is merely the default qdisc in nearly every linux distribution at this point, ios, and osx. It is used by 10s, maybe 100s of millions of devices at this point. It is part of billions of containers. With nearly 9 years of deployment into an ever increasing number of embedded devices that typically lag 5 or more years behind the mainline OSes. Making changes to how ecn is interpreted would take 8+ years to flush out of the field. 

Total (documented) L4S market penetration of its mis-interpretation of the CE field - 0. I might be kind and say "alternate", rather than mis, but slamming L4S through a rfc3168 fq_codel'd system does have the documented effects so often discussed here.

The ce_threshold parameter has been part of fq_codel for many years also. Tweaking that to instead trigger off of ECT(1) is straightforward. Without clear data - which we didn't have at the time of this debate hitting public mailing lists - doing that seemed dicy - and there was indeed a encapsulation bug regarding that transition that was found and fixed a while back which will take years to propagate to the field, also. 

The SCE alternative is a backward compatible interpretation of the ECN related bits - and fails safe -  and could have gone into wider distribution 2 years ago with near zero impact on end-user OSes, and as gradual upgrade, and been (well, actually, has, as fq_codel_fast and the alternate cake have been tested by a few parties at this point) tested as to other impacts. Coupled also with a DSCP field a la "nqb" (and preferably a few other dscp bits) it could be effective, sooner.

I have been tempted a lot to at least make SCE available as an option in the upcoming openwrt release, but neither side has made a convincing argument that doing this to ECN at all makes sense, and in fact the flood of negative data on it (rtt unfairness in particular), makes me more inclined to just recommend ecn be disabled entirely.

/Ingemar

From: Jonathan Morton <[email-masked] <mailto:[email-masked]> > 
Sent: den 20 februari 2021 22:29
To: Ingemar Johansson S <[email-masked] <mailto:[email-masked]> >
Cc: Dave Taht <[email-masked] <mailto:[email-masked]> >; Bob Briscoe <[email-masked] <mailto:[email-masked]> >; TSVWG <[email-masked] <mailto:[email-masked]> >
Subject: Re: [tsvwg] FQ & VPNs

 - Jonathan Morton

-- 

"For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman

[email-masked] <mailto:[email-masked]>  <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

---

From: Tom Henderson <[email-masked]>
Date: Sun, 21 Feb 2021 10:56:51 -0800

On 2/21/21 4:00 AM, Dave Taht wrote:
> On Sat, Feb 20, 2021 at 11:04 PM Ingemar Johansson S 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi____
> 
>     __ __
> 
>     Trying to summarize. Your concern is that the L4S flows starve out
>     the non-L4S flows when they share the same VPN tunnel. ____
> 
>     You proposed the solution yourself below: ____
> 
>     “The VPN user is thus in a position to predict, understand, and
>     possibly mitigate the effect by splitting the traffic across
>     multiple VPN flows, if performance turns out to be more important
>     than hiding that piece of information”. ____
> 
>     The same rationale can be used to put the L4S flows in a separate
>     VPN tunnel, problem solved.____
> 
>     __
> 
> 
>     __
>     And then, there is always a possibility to upgrade the AQMs to
>     support L4S as well ?. It always appear from your argumentation that
>     fq-codel is some kind of untouchable monolith that cannot at any
>     cost be modified, unless of course you see a possibility that VPN
>     tunnels can somehow leak inner header information (see separate
>     discussion around draft-ietf-tsvwg-transport-encrypt)____
> 
>     __
> 
> 
> It's not an untouchable monolith. It is merely the default qdisc in 
> nearly every linux distribution at this point, ios, and osx. It is used 
> by 10s, maybe 100s of millions of devices at this point. It is part of 
> billions of containers. With nearly 9 years of deployment into an ever 
> increasing number of embedded devices that typically lag 5 or more years 
> behind the mainline OSes. Making changes to how ecn is interpreted would 
> take 8+ years to flush out of the field.

Dave, it seems to me that this could be taken in two steps, in much less 
time than 8+ years for the majority of devices, given that most are (or 
should be) patched for other security reasons.

1) upstream a patch to the Linux kernel (and ask other OS vendors to do 
similarly) to treat ECT(1) as not-ECT
2) later add options to enable alternative semantics of ECT(1)

I don't see any reason to postpone the first step.

- Tom

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 21 Feb 2021 19:07:05 +0000

Jonathan

So.. on the one hand you can set up multiple VPN's to overcome the fq-codel problem. But then when I mention that you can use that very same method to put L4S traffic it becomes overly complex. I am not going to analyze what makes it so hard. 
I've seen similar arguments in the past in the form of edge devices that are impossible to upgrade.  

I believe that we (the WG), has to make a decision. The current discussion has been ongoing for almost 2 years. Are we going to put arguments back and forth for two more years.. or work together to bridge the gaps ?. After all fq-codel with L4S support would have been ripe and ready in one year if there had only been an incentive to have it done. Why not spend the time and effort on that instead?  

/Ingemar

> Sent: den 21 februari 2021 18:50
> > On 21 Feb, 2021, at 9:04 am, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > Trying to summarize. Your concern is that the L4S flows starve out the non-
> L4S flows when they share the same VPN tunnel.
> In the current situation, where the flows sharing the tunnel do so on a
> roughly equal basis, the choice is an *optional* performance enhancement,
> to be undertaken only when actual experience shows it to be necessary.
> Usually it is not enough of a problem to bother with the hassle - and it *is* a
> hassle, because by default VPNs do not work that way.  I mentioned it only
> to illustrate that the VPN user has some agency and is reasonably able to
> predict what will happen, and know that it won't be too bad either way.
> 
> Adding L4S to the mix means that the flows no longer share the tunnel on
> anything even approximating an equal basis.  A single L4S flow will bully all
> others out of its way, as soon as the bottleneck is at a conventional ECN AQM
> (whether with FQ, AF, or not).  It does not even matter whether the
> conventional flows support ECN themselves; if not, the conventional AQM
> will simply drop their packets at the same rate that it would have marked
> them, producing the same effect on congestion control and throughput.
> 
> Which means that separating the L4S flows out to a separate tunnel becomes
> *necessary*, not optional, to retain anything resembling normal
> performance.  And this would *not* be helped by deleting FQ from the AQM
> in question, as Bob suggested earlier; that would only permit the L4S flow to
> bully flows outwith the tunnel that happen to traverse the same bottleneck,
> rather than its harm being contained to the tunnel it occupies.
> 
> While the VPN user might reasonably be able to estimate the number of
> flows running through his tunnel, he might not be aware of the particularly
> drastic effect that L4S traffic would have.  I think it likely that his
> troubleshooting will lead to a conclusion that the L4S traffic is "unresponsive"
> to congestion control signals, immediate shutdown of the L4S traffic source,
> and a strongly worded letter of complaint to the vendor(s) perceived to be
> responsible for it.
> 
> as well ?
> To be deployable outside of carefully prepared networks, L4S *MUST* be
> capable of coexisting, at default settings, with conventional traffic in existing
> networks.  That is mandatory, not optional.  Pete's new data is prima facie
> evidence that fq_codel (and other ECN AQMs) is a feature of existing
> networks.  I'm sure you can work out the logical conclusion to that equation.
> 
>  - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 21 Feb 2021 21:10:09 +0200

> On 21 Feb, 2021, at 8:56 pm, Tom Henderson <[email-masked]> wrote:
> 
> 1) upstream a patch to the Linux kernel (and ask other OS vendors to do similarly) to treat ECT(1) as not-ECT

I do.

RFC-3168 specifies that routers "treat ECT(0) and ECT(1) as equivalent".  So implementing this first step would cause these Linux AQMs to violate RFC-3168.  I'm sure the kernel maintainers will be justified in rejecting patches on that basis.  Certainly I would raise such objections were such a patch to come to my attention.

To make that objection go away, you must first update and obsolete RFC-3168 so that it no longer contains that requirement.  That requires convincing the IETF that it's a good idea.  Again, I will maintain that it is a spectacularly bad idea.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 21 Feb 2021 19:24:42 +0000

Jonathan.
Please read
https://tools.ietf.org/html/rfc8311
/Ingemar

Hämta Outlook för iOS<https://aka.ms/o0ukef>
________________________________
Från: Jonathan Morton <[email-masked]>
Skickat: söndag, februari 21, 2021 8:10 em
Till: Tom Henderson
Kopia: Dave Taht; Ingemar Johansson S; TSVWG
Ämne: Re: [tsvwg] FQ & VPNs

I do.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 21 Feb 2021 21:31:35 +0200

> On 21 Feb, 2021, at 9:24 pm, Ingemar Johansson S <[email-masked]> wrote:
> 
> Please read

Yes, I did.  And what it actually says is:

> This memo updates RFC 3168 to allow routers to treat the ECT(0) and ECT(1) codepoints differently, provided that the changes from RFC 3168 are documented in an Experimental RFC in the IETF document stream.  The specific change to 
> RFC 3168 is to insert the words "unless otherwise specified by an Experimental RFC in the IETF document stream" at the end of the above sentence.

Which is pretty much equivalent to what I just said to Tom.  RFC-8311 does not *in itself* allow routers to treat the two ECT codepoints differently; it merely allows the IETF to approve an experiment which does so.  It has not yet done so.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 21 Feb 2021 21:23:24 +0000

Hi Jonathan

Help me to understand this right. 

You argue that Tom's suggestion is not possible because it requires that the
group agrees to start an L4S experiment that stipulates an alternate
interpretation for the ECT(1) codepoint (as per RFC8311). 
And further you point at the problem that RFC3168 AQMs cannot distinguish
between ECT(0) and ECT(1) as evidence that the L4S experiment cannot be
(safely) run, and consequently the L4S experiment should not run.

Do I understand right?, I am really confused. It all sounds like a catch-22
situation to me ?

/Ingemar 

> Sent: den 21 februari 2021 20:32
> Cc: Tom Henderson <[email-masked]>; Dave Taht <[email-masked]>;
> TSVWG <[email-masked]>
> > On 21 Feb, 2021, at 9:24 pm, Ingemar Johansson S
> > This memo updates RFC 3168 to allow routers to treat the ECT(0) and
> > ECT(1) codepoints differently, provided that the changes from RFC 3168
are
> documented in an Experimental RFC in the IETF document stream.  The
> specific change to RFC 3168 is to insert the words "unless otherwise
specified
> by an Experimental RFC in the IETF document stream" at the end of the
> above sentence.
> 
> Which is pretty much equivalent to what I just said to Tom.  RFC-8311 does
> not *in itself* allow routers to treat the two ECT codepoints differently;
it
> merely allows the IETF to approve an experiment which does so.  It has not
> yet done so.
> 
>  - Jonathan Morton

---

From: Tom Henderson <[email-masked]>
Date: Sun, 21 Feb 2021 14:33:38 -0800

> To make that objection go away, you must first update and obsolete RFC-3168 so that it no longer contains that requirement.  That requires convincing the IETF that it's a good idea.  Again, I will…

Jonathan,
I do not see any practical downside to disabling classic ECN response to 
ECT(1).  I have thought about writing a draft on this point or 
suggesting an errata to RFC 8311 on it, but decided to raise it here now.

There is a point in RFC 8311 (section 2.2 point 1) that says that 
treating ECT(0) and ECT(1) as equivalent is the preferred behavior.

However, point 3 says that hosts MUST NOT originate ECT(1) traffic 
according to old semantics.  Furthermore, the IANA protocol registry 
says that ECT(1) is for experimental use only.  I have been considering 
whether the above two points are sufficient in themselves to justify 
such a Linux patch.

The data that Pete Heist just published also would seem to support 
taking the step I suggested.  It seems to me that if the WG wants to 
support L4S experiments of any flavor, it would support first disabling 
the classic response to ECT(1).

- Tom

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 21 Feb 2021 23:50:06 +0100

Hi Tom,

> On Feb 21, 2021, at 23:33, Tom Henderson <[email-masked]> wrote:
> 
> I do not see any practical downside to disabling classic ECN response to ECT(1).  I have thought about writing a draft on this point or suggesting an errata to RFC 8311 on it, but decided to raise it here now.
> 
> There is a point in RFC 8311 (section 2.2 point 1) that says that treating ECT(0) and ECT(1) as equivalent is the preferred behavior.
> 
> However, point 3 says that hosts MUST NOT originate ECT(1) traffic according to old semantics.  Furthermore, the IANA protocol registry says that ECT(1) is for experimental use only.  I have been considering whether the above two points are sufficient in themselves to justify such a Linux patch.
> 
> The data that Pete Heist just published also would seem to support taking the step I suggested.  It seems to me that if the WG wants to support L4S experiments of any flavor, it would support first disabling the classic response to ECT(1).

	[SM] Why? Honest question. Sure L4S requires exclusive access to ECT(1) but in SCE promoting a weak ECT(1) to a stronger CE at a later node seems like exactly the right behavior. 

Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 22 Feb 2021 01:24:46 +0200

> On 21 Feb, 2021, at 11:23 pm, Ingemar Johansson S <[email-masked]> wrote:
> 

It's not a Catch 22.  It's just the heart of the problem with L4S.

L4S cannot safely be deployed on existing networks - that is axiomatic in its current form.  Therefore, you have to prepare the network as fully L4S-aware before you can safely deploy your first endpoints - on Internet scale, that is not really feasible within a decade.  It would be easier to start deployment if you had a good way of containing L4S traffic to a small portion of the network (eg. a CDN and an associated ISP) in which dealing with the network preparation is actually feasible in the short term.

Using a DSCP as the L4S identifier would be such a method, permitting fencing off the L4S traffic at the network boundaries, and because the network needs to be prepared anyway, there should be no problem with DSCP bleaching.  Therefore there is actually no reason to consume ECT(1) as the L4S ID.

Until you figure out the above, you will not get a published Experimental RFC, which is what RFC-8311 requires to override that requirement in RFC-3168.  But there *is* a path forward to success, and therefore it is not a Catch 22.  It's just going to be a very difficult and drawn-out process to deploy L4S, unless some relatively significant design change occurs (which we haven't seen in the past two years) to cure some of the deficiencies.

Conversely, let's look at SCE from the same perspective.  Remember, SCE is capable of achieving the same performance as L4S, given similar marking and response algorithms.  The difference is in backwards compatibility.

SCE is specifically designed to be compatible with existing networks, and to behave identically to conventional traffic in that case.  Instead of identifying itself to the network and trusting that the network understands that message, the network distinguishes the SCE signal from the existing ECN signal unambiguously, and trusts that existing receivers will merely ignore a signal that is currently specified to be ignored.  This means SCE endpoints and middleboxes can be deployed safely on independent schedules, and it is not necessary to eliminate old equipment, nor to fence off the edges of networks to contain SCE traffic.

It is of course possible to label SCE traffic with a DSCP at source, and thus use the simplest possible two-queue AQM structure that can run at a bajillion packets per second.  It doesn't cause any safety problems if the DSCP is bleached en route, because the receiver can still distinguish the intent of the marks.  In practice, most people are likely to use flow-aware AQM or none at all, just as they do today, and this does not require a DSCP.

SCE also requires an Experimental RFC to be published before its new use of ECT(1) is permitted in public networks.  This is no different from the situation with L4S.  But because there are few if any safety concerns of the type that L4S raises (in particular, SCE's failure modes tend to *reduce* its load on the network, not increase it), this should not be a big obstacle - if only the WG would adopt it.  Currently L4S stands in the way of that.

But that is why we proposed SCE as an alternative to L4S in the same problem space.  It works just as well in the ideal case, and is much easier to deploy.

 - Jonathan Morton

---

From: Tom Henderson <[email-masked]>
Date: Sun, 21 Feb 2021 15:36:20 -0800

> 	[SM] Why? Honest question. Sure L4S requires exclusive access to ECT(1) but in SCE promoting a weak ECT(1) to a stronger CE at a later node seems like exactly the right behavior.

Sebastien,
OK, I see your point now; keeping legacy behavior would still allow 
legacy code to generate CE for an SCE flow.  However, it would have to 
be a multiple bottleneck scenario with the second bottleneck being the 
classic one for this to matter, I suppose.

I was hoping to propose a way forward for the backward compatibility 
concerns by decoupling these steps, but if it is not agreeable, I guess 
the deadlock remains.

- Tom

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 22 Feb 2021 01:41:44 +0200

> On 22 Feb, 2021, at 1:36 am, Tom Henderson <[email-masked]> wrote:
> 
> I was hoping to propose a way forward for the backward compatibility concerns by decoupling these steps, but if it is not agreeable, I guess the deadlock remains.

Hypothetically, if the L4S RFC were indeed published, and included a requirement that any non-L4S AQMs treat ECT(1) as Not-ECT, then that would begin the process you described.  But it does not change the timeline wherein the network must be thoroughly prepared before L4S endpoints can be deployed.

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Sun, 21 Feb 2021 23:49:41 +0000

On 2/21/21, 4:25 PM, "tsvwg on behalf of Jonathan Morton" <[email-masked] on behalf of [email-masked]> wrote:

[GW] This is blatantly false. In the vast majority of paths, L4S is considered safe for deployment today.  In a minority of paths there exists the rare possibility of L4S flows achieving more than their "fair" share of the bottleneck link bandwidth, and an extremely rare possibility that this unfairness results in a significant temporary degradation of classic traffic performance.  In these paths there is almost certainly only one network element that will need to be updated in order to prevent any possibility of such degradation, and as far as anyone can discern at the moment those network elements are largely prosumer grade home routers that are increasingly being actively patched already. 

[snip]

[GW] The WG considered SCE and has decided against it.  You seem to be repeating the same arguments (with the same transparent hyperbole) that led to that conclusion in hopes that the outcome somehow changes. 

[snip]

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sun, 21 Feb 2021 16:17:09 -0800 (PST)

> [GW] This is blatantly false. In the vast majority of paths, L4S is considered safe for deployment today.  In a minority of paths there exists the rare possibility of L4S flows achieving more than…

I strongly disagree that it should be considered safe.

"Only one network element" in a path with an internet of millions of paths == millions of nodes to be updated.  One must also realize every path has a bottle neck someplace, and that TCP is a capacity seeking protocol so the probablity of an issue should be rather high, not "extremly rare."

You know its really simple to solve most of these arguments, deploy L4S in an AS behind a DSCP and get some real working data.  All the handwaving and guessing is not going to get anyone there on these issues.
> [GW] The WG considered SCE and has decided against it.  You seem to be repeating the same arguments (with the same transparent hyperbole) that led to that conclusion in hopes that the outcome someh…

The consideration was weither to use ECT(1) as an input or output or something else.  There was never a question of "SCE" on the table.  IMHO, that whole process was seriously flawed as what was initial presented as "not an SCE vs L4S" decision got baked by the chair(s) into almost exactly that by how they presented and guided that decision process.

-- 
Rod Grimes                                                 [email-masked]

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 22 Feb 2021 00:34:29 +0000

Pete,

Thx for doing this. I'm afraid I won't be able to now analyse the data 
you've provided for a couple of days, 'cos I have to focus on a couple 
of deadlines tomorrow, one being the IETF draft deadline.

But whatever, thx v much.

Bob

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 22 Feb 2021 06:07:45 +0000

Hi Tom, Sebastian
Just to clarify.. there is no SCE alternative on the table in TSVWG. SCE is only an individual contribution, it is not a WG item. L4S got a wide support, while SCE was only supported by a few when this discussion was brought up 9-10 months ago.
Only Sebastian and others can explain why they repeatedly bring up SCE over and over again..

Tom: I think the two step approach proposed by you is definitely worth a closer look.

/Ingemar

Hämta Outlook för iOS<https://aka.ms/o0ukef>
________________________________
Från: tsvwg <[email-masked]> på uppdrag av Tom Henderson <[email-masked]>
Skickat: måndag, februari 22, 2021 12:37 fm
Till: Sebastian Moeller
Kopia: TSVWG
Ämne: Re: [tsvwg] FQ & VPNs

Sebastien,

- Tom

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 22 Feb 2021 08:53:31 +0100

Hi Ingemar,

> On Feb 22, 2021, at 07:07, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Tom, Sebastian 

	[SM] Since you seem to ask... I have come to the conclusion that L4S does not achieve its promises and for several years there have been essential zero effective efforts to remedy this (a few attempts have been started, like the ill-fated and misnamed RTT-independence push, that basically never achieved their goals and then just petered out; but these left me with an impression of being tailored at silencing objections to L4S instead of actually improving the L4S design and implementation). So I assume that L4S is going to fail, sooner (hopefully) or later (after first being deployed and hogging the ECT(1) codepoint for years to come), but failure is IMHO certain. So I bring up SCE only as an example as a post-L4S High Frequency/Low Amplitude congestion signaling contender. In the context of Tom's mail I simply pointed out that stopping ECN treatment of ECT(1) packets is not a clear cut improvement over the status quo. That is all.

Best Regards
	Sebastian

> Hämta Outlook för iOS

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 22 Feb 2021 10:04:38 +0000

Sebastian

You are not required to like or believe in L4S, but there is support for it in TSVWG. 
There are however a few roadblocks along the way. One particular solution is to patch OS kernels to treat ECT(1) as not-ECT as a first step as suggested by Tom. You don't want that do happen for whatever reasons you list below. 

But maybe this discussion is pointless?, after all you are not the only one to decide what goes into Linux or other OS kernels. Updates there happen with or without the blessing of the IETF.

/Ingemar

> From: Sebastian Moeller <[email-masked]>
> Sent: den 22 februari 2021 08:54
> Cc: Tom Henderson <[email-masked]>; TSVWG <[email-masked]>
> > On Feb 22, 2021, at 07:07, Ingemar Johansson S
> > Just to clarify.. there is no SCE alternative on the table in TSVWG. SCE is only
> an individual contribution, it is not a WG item. L4S got a wide support, while
> SCE was only supported by a few when this discussion was brought up 9-10
> months ago.
> 	[SM] Since you seem to ask... I have come to the conclusion that L4S
> does not achieve its promises and for several years there have been
> essential zero effective efforts to remedy this (a few attempts have been
> started, like the ill-fated and misnamed RTT-independence push, that
> basically never achieved their goals and then just petered out; but these left
> me with an impression of being tailored at silencing objections to L4S instead
> of actually improving the L4S design and implementation). So I assume that
> L4S is going to fail, sooner (hopefully) or later (after first being deployed and
> hogging the ECT(1) codepoint for years to come), but failure is IMHO certain.
> So I bring up SCE only as an example as a post-L4S High Frequency/Low
> Amplitude congestion signaling contender. In the context of Tom's mail I
> simply pointed out that stopping ECN treatment of ECT(1) packets is not a
> clear cut improvement over the status quo. That is all.
> 
> Best Regards
> 	Sebastian
> 
> 
> > >        [SM] Why? Honest question. Sure L4S requires exclusive access to
> ECT(1) but in SCE promoting a weak ECT(1) to a stronger CE at a later node
> seems like exactly the right behavior.
> >
> > Sebastien,
> > concerns by decoupling these steps, but if it is not agreeable, I
> > guess the deadlock remains.
> >
> > - Tom

---

From: Pete Heist <[email-masked]>
Date: Mon, 22 Feb 2021 11:27:46 +0100

Hi Greg, some comments below [PH].

On Sun, 2021-02-21 at 23:49 +0000, Greg White wrote:
> On 2/21/21, 4:25 PM, "tsvwg on behalf of Jonathan Morton" <
> [email-masked] on behalf of [email-masked]> wrote:
> 
>     L4S cannot safely be deployed on existing networks - that is
> axiomatic in its current form.  Therefore, you have to prepare the
> network as fully L4S-aware before you can safely deploy your first
> endpoints - on Internet scale, that is not really feasible within a
> decade.  It would be easier to start deployment if you had a good way
> of containing L4S traffic to a small portion of the network (eg. a
> CDN and an associated ISP) in which dealing with the network
> preparation is actually feasible in the short term.
> 
> [GW] This is blatantly false. In the vast majority of paths, L4S is
> considered safe for deployment today.  In a minority of paths there
> exists the rare possibility of L4S flows achieving more than their
> "fair" share of the bottleneck link bandwidth, and an extremely rare
> possibility that this unfairness results in a significant temporary
> degradation of classic traffic performance.  In these paths there is
> almost certainly only one network element that will need to be
> updated in order to prevent any possibility of such degradation, and
> as far as anyone can discern at the moment those network elements are
> largely prosumer grade home routers that are increasingly being
> actively patched already. 

[PH] In regards to the proportion of paths with RFC3168 AQMs deployed,
referring to section 3.2 in our recent draft, and cautioning that this
is not authoritative:
https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-01.html#name-rfc3168-aqm-activity

If we remove the IP addresses that I had an influence on, leaving only
the unknown sources of AQM marking, then the proportion of IPs that
successfully negotiated ECN and saw CE or ECE, i.e. apparently had an
AQM on the path, is (90-38)/382 = 0.14.

I also noticed during a few script runs, as it was refined, that as we
let it run longer, more IP addresses appeared that saw CE or ECE. AQMs
are only visible to us when flows both negotiate ECN and experience
congestion at those AQMs. It takes time for detection when 1.44% of
SYNs had ECN capability, so we may have missed some.

The 1.44% ECN SYN proportion may also make it more difficult to detect
ECN traffic within tunnels, because now we have the product of ECN
capable traffic with traffic that's traversing a tunnel. However, that
product does not apply to the potential for induced harm, since that
occurs to both ECN and non-ECN capable flows.

[PH] As to the level of harm induced when traffic winds up in the same
queue, it appears that in the presence of a "long-running" L4S flow,
short flows as well as long flows, ECN capable or not, may be affected.
As flows start up, there can be "good" cases:

http://sce.dnsmgr.net/results/l4s-2021-02-19T194323-s8/l4s-s8-rfc3168-second/l4s-s8-rfc3168-second-ns-prague-vs-cubic-ecn-fq_codel_1q_-50Mbit-20ms_tcp_delivery_with_rtt.svg

and "not as good" cases:

http://sce.dnsmgr.net/results/l4s-2021-02-19T193357-s8/l4s-s8-rfc3168-second/l4s-s8-rfc3168-second-ns-prague-vs-cubic-ecn-fq_codel_1q_-50Mbit-80ms_tcp_delivery_with_rtt.svg

It might be useful to have a histogram of FCT vs flow length, to
project what the user experience might be like for web browsing or
other applications in the presence of long-running L4S flows.

Regards,
Pete

> [snip]
> 
>     Conversely, let's look at SCE from the same perspective. 
> Remember, SCE is capable of achieving the same performance as L4S,
> given similar marking and response algorithms.  The difference is in
> backwards compatibility.
> 
> [GW] The WG considered SCE and has decided against it.  You seem to
> be repeating the same arguments (with the same transparent hyperbole)
> that led to that conclusion in hopes that the outcome somehow
> changes. 
> 
> [snip]
> 
>

---

From: Martin Duke <[email-masked]>
Date: Fri, 26 Feb 2021 15:31:12 -0800

Thanks Pete!

I wonder if the non-TCP ECN traffic is QUIC? I don't think the main
implementations are doing it, but there are a few ECN-capable
implementations in production.

On Sun, Feb 21, 2021 at 4:34 PM Bob Briscoe <[email-masked]> wrote:

> >>>>       > has been successfully submitted by Peter G. Heist and
> >>>> posted to
> >>>> the
> >>>>       > IETF repository.
> >>>>       >
> >>>>       >   counter and flow tracking data was collected and
> >>>> analyzed for
> >>>> TCP and
> >>>>       >   other protocols.
> >>>>

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 27 Feb 2021 01:57:47 +0200

> On 27 Feb, 2021, at 1:31 am, Martin Duke <[email-masked]> wrote:
> 
> I wonder if the non-TCP ECN traffic is QUIC? I don't think the main implementations are doing it, but there are a few ECN-capable implementations in production.

This would be a reasonable interpretation of traffic on ports typically used with HTTP or similar L5 protocols.  We have, similarly, interpreted non-TCP traffic using ports commonly associated with tunnel protocols as actually being (at least partly) TCP through a tunnel.

However, most of what we saw was on random high ports, and some that are most often used by BitTorrent clients.  So we think it's probably due to BT clients trying to do "clever" things with TOS/DSCP but misunderstanding it.  I believe the default in libtorrent is to set CS1, correctly, so I don't know which particular clients might be so affected, or whether it is due to end users setting manual firewall mangle rules incorrectly.

Since the data collection tool we used doesn't have a way to unambiguously discriminate between QUIC and any other UDP-supported protocol, it's difficult for us to say more about how much QUIC-ECN was seen.  If you have a criterion we could use for that, we might be able to build it into a future run.

 - Jonathan Morton