Subject: [tsvwg] I-D Action: draft-ietf-tsvwg-l4sops-00.txt

From: [email-masked]
Date: Wed, 05 May 2021 17:33:48 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Transport Area Working Group WG of the IETF.

        Title           : Operational Guidance for Deployment of L4S in the Internet
        Author          : Greg White
	Filename        : draft-ietf-tsvwg-l4sops-00.txt
	Pages           : 18
	Date            : 2021-05-05

Abstract:
   This document is intended to provide guidance in order to ensure
   successful deployment of Low Latency Low Loss Scalable throughput
   (L4S) in the Internet.  Other L4S documents provide guidance for
   running an L4S experiment, but this document is focused solely on
   potential interactions between L4S flows and flows using the original
   ('Classic') ECN over a Classic ECN bottleneck link.  The document
   discusses the potential outcomes of these interactions, describes
   mechanisms to detect the presence of Classic ECN bottlenecks, and
   identifies opportunities to prevent and/or detect and resolve
   fairness problems in such networks.  This guidance is aimed at
   operators of end-systems, operators of networks, and researchers.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4sops/

There is also an HTML version available at:
https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4sops-00.html

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Greg White <[email-masked]>
Date: Thu, 6 May 2021 03:54:01 +0000

All,

I had hoped to upload this further in advance of the interim meeting next week so that participants would have more chance to review it beforehand. 

There are quite a few changes from the last individual draft, as can be seen in the diff: https://www.ietf.org/rfcdiff?url1=draft-white-tsvwg-l4sops-02&url2=draft-ietf-tsvwg-l4sops-00 
And, there remain a few TODO items as well.  These changes/TODOs were the mainly result of review comments on the mailing list, though I received some offline comments and suggested text as well.

I'll do a relatively quick summary of the changes during the timeslot on Monday, but for now, the most notable changes are:

New Section 3.1 summarizing recent studies of RFC3168 deployment (Jake, Pete, please review and see if the text is an accurate summary).
New Section 6 discussing actions that can be taken by an operator of FQ bottlenecks.
New Section 7 discussing conclusion of the L4S experiment (Sebastian, for now I went with the text proposed on the mailing list, we can revise it in the next draft if needed).

-Greg

On 5/5/21, 6:34 PM, "tsvwg on behalf of [email-masked]" <[email-masked] on behalf of [email-masked]> wrote:

    Abstract:

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 6 May 2021 11:53:31 +0200

Hi List, hi Greg,

sorry for being late, but...

...speaking of evidence for ECN marking in the wild; here an analysis of CAIDA internet "back-bone link " traffic from 2018 https://ieeexplore.ieee.org/document/8909187 that has not beed discussed as far as I can tell. The data showed between ~5% of IPv4 packets carrying something other than NotECT in the ECN bitfield, a more detailed look at port resolution showed 0.26% (port 80) or 0.3% (port 443) in IPv4. I do not think this was broken down by network, but indicates non-zero ECN usage in the wild in 2018. IMHO the percentage of traffic might be a better proxy for ECN prevalence than number of networks. That said for figuring out how many/which operators should be pro-actively informed of the coming changes identifying the networks seems helpful.

> On May 6, 2021, at 05:54, Greg White <[email-masked]> wrote:
> 
> All,
> 

[SM] See above for an additional relevant study with 2018 data that might be worth adding in the next revision.

> New Section 6 discussing actions that can be taken by an operator of FQ bottlenecks.

[SM] There I find:
"This unfairness is compounded by the fact that the FQ	
 	   scheduler will already be causing unfairness to flows within the	
 	   tunnel relative to flows that are not tunneled."

This argues that the AQM "should" divine the number of true capacity seeking flows in the compound tunnel aggregate and assign the proper share. I argue that this is impossible to robustly solve in general, and, in the case of encrypted tunnels also arguably the wrong thing to do, as doing so would introduce additional, potential severe re-ordering in the encrypted flow. But a partial solutions already exist, on IPv6 a tunnel could make use of the flow-label field to give additional flow identifier to AQMs. Arguably that is what tunnels should do, as that keeps the policy decision how much reordering one will accept in the hand of the tunnel operators.

> New Section 7 discussing conclusion of the L4S experiment (Sebastian, for now I went with the text proposed on the mailing list, we can revise it in the next draft if needed).

[SM] Much appreciated, while I have my quibbles, getting this section in feels quite helpful (and I do not pretend my initial draft was in any way complete and/or perfect). But for completeness, I still think it a good idea to lay out the gamut of potentially required/recommended actions at network termination. The argument will be better the more objective it is, and it already carries a noticeable pro-L4S bias.

Regards
	Sebastian

---

From: Pete Heist <[email-masked]>
Date: Fri, 07 May 2021 10:40:25 +0200

Hi Greg, response below...

On Thu, 2021-05-06 at 03:54 +0000, Greg White wrote:
> All,
> 
> I had hoped to upload this further in advance of the interim meeting
> next week so that participants would have more chance to review it
> beforehand. 
> 
> There are quite a few changes from the last individual draft, as can be
> seen in the diff:  
> https://www.ietf.org/rfcdiff?url1=draft-white-tsvwg-l4sops-02&url2=draft-ietf-tsvwg-l4sops-00
>  
> And, there remain a few TODO items as well.  These changes/TODOs were
> the mainly result of review comments on the mailing list, though I
> received some offline comments and suggested text as well.
> 
> I'll do a relatively quick summary of the changes during the timeslot
> on Monday, but for now, the most notable changes are:
> 
> New Section 3.1 summarizing recent studies of RFC3168 deployment (Jake,
> Pete, please review and see if the text is an accurate summary).

-> "A small number of recent studies have attempted to gauge the level
of RFC3168 deployment in the internet."

[PH] Suggestion: "Several recent studies..." would be more neutral.

-> "but in other subnets there were 33 IPs where CE-marking was
possibly observed"

[PH] Suggestion: "...where possible AQM activity was observed via CE
marks and/or ECE flags"

-> "It was agreed
(https://mailarchive.ietf.org/arch/msg/tsvwg/Rj7GylByZuFa3_LTCMvEfb-CYp
w/) that these were likely to be due to fq_codel implementations in
home routers deployed by members of the cooperative."

[PH] I'd like to see WiFi access points mentioned, as they were also
discussed at some point. Suggestion: "...likely to be due to either
fq_codel implementations in home routers, or the drivers of WiFi access
points."

Pete

> New Section 7 discussing conclusion of the L4S experiment (Sebastian,
> for now I went with the text proposed on the mailing list, we can
> revise it in the next draft if needed).
> 
> -Greg
> 
> 
> On 5/5/21, 6:34 PM, "tsvwg on behalf of [email-masked]" <
> [email-masked] on behalf of [email-masked]> wrote:
> 
>

---

From: Greg White <[email-masked]>
Date: Fri, 7 May 2021 15:26:55 +0000

Hi Sebastian,

Thanks for pointing this paper out.  I had not seen it previously, and will include a mention of it in the draft.  The results seem a bit puzzling though.  They report that, of the ~5% of IPv4 packets with a non-zero ECN field, 94% of them were CE!  It is hard to believe that this is actual evidence of ECN marking in the wild.  It seems more like evidence of some kind of ToS munging happening for traffic on this ISP's link between NYC and Sao Paulo.  

While in terms of total packets that CAIDA dataset is huge, in terms of visibility to traffic in the wild (roughly 80-90% of which is directly peered locally as I understand it) it might not be that representative.  Does anyone have further insight into this paper, or the peculiarities of the CAIDA equinix-nyc trace?

So far, I think the data from Akamai and Apple have been the most useful in understanding the situation.

-Greg

On 5/6/21, 3:53 AM, "Sebastian Moeller" <[email-masked]> wrote:

    Hi List, hi Greg,

    [SM] There I find:

    Regards
    	Sebastian

---

From: Greg White <[email-masked]>
Date: Fri, 7 May 2021 18:42:19 +0000

Hi Pete, 

Thanks for the review and the comments.  See below.

On 5/7/21, 2:40 AM, "Pete Heist" <[email-masked]> wrote:

[GW] I suppose it depends on whether we're content with the studies that have been done so far, or whether we want to encourage more.  "Several recent studies" to me sounds like we feel as if the topic has already been reasonably well covered, and there wouldn't be a lot of interest in new data, which I don't think is true.  Also, we've only got 2 studies that cover a large number of paths (one of which shared very little data on this subject) and one study that covered a single small ISP (and none of them were crafted to try to detect whether the CE-marking was a shared queue or FQ).  In my view this is a small number of studies.   

[GW] Thanks, I'll make that change.

    -> "It was agreed

[GW] Ok, but we still believe that, between these two, home routers are likely (by far) to be the dominant source of the CE-marks for these 33 IPs, correct?  I would assume that home routers are A) much more common than APs, and B) much more likely to be configured as the bottleneck (via SQM).   If it is more appropriate, I could add a mention of Access Points in another place (e.g. Section 6).

    Pete

---

From: Pete Heist <[email-masked]>
Date: Sat, 08 May 2021 11:00:57 +0200

Hi Greg, responses below...

On Fri, 2021-05-07 at 18:42 +0000, Greg White wrote:
>     > All,
>     > 
>     > I had hoped to upload this further in advance of the interim
> meeting
>     > next week so that participants would have more chance to review
> it
>     > beforehand. 
>     > 
>     > There are quite a few changes from the last individual draft,
> as can be
>     > seen in the diff:  
>     >    
>     >  
>     > And, there remain a few TODO items as well.  These
> changes/TODOs were
>     > the mainly result of review comments on the mailing list,
> though I
>     > received some offline comments and suggested text as well.
>     > 
>     > I'll do a relatively quick summary of the changes during the
> timeslot
>     > on Monday, but for now, the most notable changes are:
>     > 
>     > New Section 3.1 summarizing recent studies of RFC3168
> deployment (Jake,
>     > Pete, please review and see if the text is an accurate
> summary).
> 
> [GW] I suppose it depends on whether we're content with the studies
> that have been done so far, or whether we want to encourage more. 
> "Several recent studies" to me sounds like we feel as if the topic
> has already been reasonably well covered, and there wouldn't be a lot
> of interest in new data, which I don't think is true.  Also, we've
> only got 2 studies that cover a large number of paths (one of which
> shared very little data on this subject) and one study that covered a
> single small ISP (and none of them were crafted to try to detect
> whether the CE-marking was a shared queue or FQ).  In my view this is
> a small number of studies.   
> 

[PH] I agree we'd always like more data, but each of the studies
confirmed RFC3168 AQM activity on the Internet. Ours showed a known AQM
detection rate of 60%, suggesting that AQMs are also likely to be
under-counted. Ideally, the language used would not minimize what *was*
learned, as that is relevant to safety. However, I do think the safety
issue itself is more important to resolve than this.

>     (  
> https://mailarchive.ietf.org/arch/msg/tsvwg/Rj7GylByZuFa3_LTCMvEfb-CYp
> [GW] Ok, but we still believe that, between these two, home routers
> are likely (by far) to be the dominant source of the CE-marks for
> these 33 IPs, correct? 

[PH] I'm not sure, to be honest. The Atheros, MediaTek and Intel
chipsets supported by the drivers are fairly widely used. It seems to
me that over time, users may be just as likely, or more, to get
fq_codel/ECN by buying a WiFi AP or router that has it built in and
enabled by default, than by configuring a shaper with fq_codel or Cake.
Since WiFi rates can vary by orders of magnitude and have low rates
when there's poor coverage, they should become the bottleneck sometimes
as well. It would be good if this is mentioned somewhere.

Thanks for responding...

Pete

>  I would assume that home routers are A) much more common than APs,
> and B) much more likely to be configured as the bottleneck (via
> SQM).   If it is more appropriate, I could add a mention of Access
> Points in another place (e.g. Section 6).
> 
> 
> 
>     Pete
> 
>     > New Section 6 discussing actions that can be taken by an
> operator of FQ
>     > bottlenecks.
>     > New Section 7 discussing conclusion of the L4S experiment
> (Sebastian,
>     > for now I went with the text proposed on the mailing list, we
> can
>     > revise it in the next draft if needed).
>     > 
>     > -Greg
>     > 
>     > 
>     > On 5/5/21, 6:34 PM, "tsvwg on behalf of  
> [email-masked]" <
>     > [email-masked] on behalf of [email-masked]>
> wrote:
>     > 
>     > 
>     >     A New Internet-Draft is available from the on-line
> Internet-Drafts
>     > directories.
>     >     This draft is a work item of the Transport Area Working
> Group WG of
>     > the IETF.
>     > 
>     >             Title           : Operational Guidance for
> Deployment of
>     > L4S in the Internet
>     >             Author          : Greg White
>     >         Filename        : draft-ietf-tsvwg-l4sops-00.txt
>     >         Pages           : 18
>     >         Date            : 2021-05-05
>     > 
>     >     Abstract:
>     >        This document is intended to provide guidance in order
> to ensure
>     >        successful deployment of Low Latency Low Loss Scalable
>     > throughput
>     >        (L4S) in the Internet.  Other L4S documents provide
> guidance for
>     >        running an L4S experiment, but this document is focused
> solely
>     > on
>     >        potential interactions between L4S flows and flows using
> the
>     > original
>     >        ('Classic') ECN over a Classic ECN bottleneck link.  The
>     > document
>     >        discusses the potential outcomes of these interactions,
>     > describes
>     >        mechanisms to detect the presence of Classic ECN
> bottlenecks,
>     > and
>     >        identifies opportunities to prevent and/or detect and
> resolve
>     >        fairness problems in such networks.  This guidance is
> aimed at
>     >        operators of end-systems, operators of networks, and
>     > researchers.
>     > 
>     > 
>     >     The IETF datatracker status page for this draft is:
>     >     https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4sops/
>     > 
>     >     There is also an HTML version available at:
>     >      
>     > 
>     > 
>     >     Please note that it may take a couple of minutes from the
> time of
>     > submission
>     >     until the htmlized version and diff are available at
>     > tools.ietf.org.
>     > 
>     >     Internet-Drafts are also available by anonymous FTP at:
>     >     ftp://ftp.ietf.org/internet-drafts/
>     > 
>     > 
>     > 
> 
> 
>

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 8 May 2021 11:33:12 +0200

Hi Pete, Greg,

see below prefixed [SM]

> On May 8, 2021, at 11:00, Pete Heist <[email-masked]> wrote:
> 

	[SM] Actually, with 1Gbps and even 10 Gbps home-links coming online in Europe in increasing number (e.g. in 10 Gbps symmetric XGS-PON in Switzerland and France, asymmetric 1 Gbps Docsis3.1 in Germany), WiFi is going to become the relevant bottleneck more and more (getting 1 Gbps thoughput on WiFi is hard, getting 10 Gbps seems unrealistic for consumer grade gear). This is having two consequences:
a) an increased likelihood of fq_codel/ECN action (which is a good thing, Linux airtime fairness enabled WiFi stack works wonders if WiFi is the bottleneck, so these patches arrived just in time)
b) an increase in burstiness in the range of WiFi's 4ms aggregation window (resulting in reduced throughput for ECT(1) flows)

both of these are already known to be challenging for L4S...

>>     > New Section 6 discussing actions that can be taken by an
>> operator of FQ
>>     > bottlenecks.
>>     >     A New Internet-Draft is available from the on-line
>> Internet-Drafts
>>     > directories.
>>     >     This draft is a work item of the Transport Area Working
>> Group WG of
>>     > the IETF.
>>     > 
>>     >             Title           : Operational Guidance for
>> Deployment of
>>     > L4S in the Internet
>>     >             Author          : Greg White
>>     >        running an L4S experiment, but this document is focused
>> solely
>>     > on
>>     >        potential interactions between L4S flows and flows using
>> the
>>     > original
>>     >        mechanisms to detect the presence of Classic ECN
>> bottlenecks,
>>     > and
>>     >     Please note that it may take a couple of minutes from the
>> time of
>>     > submission

---

From: Greg White <[email-masked]>
Date: Tue, 11 May 2021 23:21:39 +0000

Sebastian,

On the L4S Interim call yesterday, you mentioned that the data on ECN usage in this Roddav paper for ports 80 and 443 were more realistic.  I'm not sure that I agree!

The Port 80 traffic shows 0.26% having a non-zero ECN field.  Ok, fine.   But, then it shows the breakdown of that 0.26% as:  0.22% ECT(1), 88.22% ECT(0) and 11.55% CE.  Disregarding the small amount of ECT1 traffic, this is 11.6% CE-marking!  While not as obviously broken as the overall statistic of 94% CE marking, it does still seem awfully high to be actual CE-marking for Classic ECN.  

For Port 443, the picture is a little bit more believable, but I'm not sure what we learn from it:  0.3% prevalence of non-zero ECN field, and, of that, 1.15% CE.   
We don't know what the end-to-end RTT is for these connections, but I would estimate the RTT for the NYC-SaoPaulo link itself to be about 130ms, so maybe end-to-end it's 150ms RTT typically? 
If 100% of the ECT connections were traversing an RFC3168 bottleneck (unlikely), I think a 1.15% CE marking rate would mean on the order of a few hundred kbps per flow.  Fairly low, but believable.  If it is more like 10% of them (aligning with Pete's data) that were experiencing a 3168 AQM, then that means 11.5% CE marking for those flows (and a throughput more like ~100 kbps - pretty low, but still believable).  So, is this "evidence" that the prevalence of 3168 AQMs is between 10% and 100%?  I doubt it.

Is it possible that this link was carrying traffic from Buenos Aires?

Currently, I plan to write:

Other studies (e.g. [Trammel], [Bauer], [Mandalari]) have examined ECN traversal, but have not reported data on prevalence of CE-marking by networks. Another [Roddav] examined traces from a Tier 1 ISP link in 2018 and observed that 94% of the non-zero ECN marked packets were CE, which appears to reflect a misconfiguration of equipment using that link, as opposed to providing evidence of RFC3168 AQM deployment.

Let me know if you think there is something more insightful we can conclude from that paper.

-Greg

On 5/7/21, 9:28 AM, "tsvwg on behalf of Greg White" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Sebastian,

    -Greg

        Hi List, hi Greg,

        [SM] There I find:

        Regards
        	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 12 May 2021 16:08:11 +0200

Hi Greg,

thanks for getting back on this topic.

> On May 12, 2021, at 01:21, Greg White <[email-masked]> wrote:
> 
> Sebastian,
> 

	[SM] Not wanting to go into semantics here, but to me the ~0.3% seem in line with Jake's report and considerably less problematic than the 5% overall number. Calling that "more realistic" seems defensible to me ;). But please note that if we multiply the 5.23% of IP4 packets with ECN bis != 00, with the relative 4.63% of ECT(0) we get 5.23*0.0463 = 0.24% which again fits the numbers from Jake and the port resolved data as well.

> The Port 80 traffic shows 0.26% having a non-zero ECN field.  Ok, fine.   But, then it shows the breakdown of that 0.26% as:  0.22% ECT(1), 88.22% ECT(0) and 11.55% CE.  Disregarding the small amount of ECT1 traffic,

	[SM] Which indicates potentially a small mis-marked ECT(1) background, that might be unavoidable, at first glance I would accept that as real data, but not necessarily as indication of rfc3168 compliant ECN using ECT(1) (could be, but doesn't have to be).

> this is 11.6% CE-marking!  While not as obviously broken as the overall statistic of 94% CE marking, it does still seem awfully high to be actual CE-marking for Classic ECN.  

	[SM] Yes might indicate some issues (that potentially show up for CE as that is part of a potential 0xFF pattern for the TOS byte), but it might also be real AQM action. Hard to differentiate from the outside. If I had to assess risk, I would assume all of these to be genuine ECN packets though, as that is the more conservative interpretation.

> For Port 443, the picture is a little bit more believable, but I'm not sure what we learn from it:  0.3% prevalence of non-zero ECN field, and, of that, 1.15% CE.   

	[SM] What percentage of CE marks would you expect on a reasonably busy backbone link? I have no good intuition or data what marking percentage to expect.

> We don't know what the end-to-end RTT is for these connections, but I would estimate the RTT for the NYC-SaoPaulo link itself to be about 130ms, so maybe end-to-end it's 150ms RTT typically? 

	[SM] No idea. The geographical distance is 7681Km (but the network path sure is longer), so by rule of thumb the RTT > 76ms, so your 150ms seems in the right range.

> If 100% of the ECT connections were traversing an RFC3168 bottleneck (unlikely), I think a 1.15% CE marking rate would mean on the order of a few hundred kbps per flow.  

	[SM] According to https://www.caida.org/catalog/datasets/trace_stats/ the mean rate (per flow I assume) was between roughly 400 and 600 Kbps in that data file; which does not rule out anything...

> Fairly low, but believable.  If it is more like 10% of them (aligning with Pete's data) that were experiencing a 3168 AQM, then that means 11.5% CE marking for those flows (and a throughput more like ~100 kbps - pretty low, but still believable).  So, is this "evidence" that the prevalence of 3168 AQMs is between 10% and 100%?  I doubt it.

	I would take it as non-zero evidence of rfc3168 network activity.

> Is it possible that this link was carrying traffic from Buenos Aires?

	[SM] Well possible, although not documented one way or another. Apple's Argentina number was reported 1 year earlier, at what time had the Argentinean ISP you must be thinking of here, been contacted? I thought someone, maybe Bob, reported a follow-up and that the issue was fixed. Would be interesting to learn whether that was before or after March-May 2018?

> Let me know if you think there is something more insightful we can conclude from that paper.

	[SM] I would agree that the 5% number seems too vague, but the ~0.3% number I would take and report as data. And I would not try to interpret the data much more than stating that in passive traces ECN functionality can not be confirmed one way or the other.

Best Regards
	Sebastian