Subject: Re: [tsvwg] FW: New Version Notification for draft-white-tsvwg-l4sops-00.txt

From: Sebastian Moeller <[email-masked]>
Date: Sat, 1 Aug 2020 21:49:19 +0200

Hi Greg,

section 4&5:

"4.  Operator of a Network

   While it is, of course, preferred for networks to deploy L4S-capable
   high fidelity congestion signaling, a network operator who has
   deployed equipment in a likely bottleneck link location (i.e. a link
   that is expected to be fully saturated) that is configured with an
   RFC3168 FIFO AQM can take certain steps in order to improve rate
   fairness between classic traffic and L4S traffic."

[SM] I beg to differ, it is far from clear whether it is generally preferred for networks to deploy  L4S-capable high fidelity congestion signaling unless the L4S team finally coughs up the long requested tests over something else than the one thing we know this would work, the low hop-count short RTT path from close CDNs to end-users. 

"4.1.  Configure AQM to treat ECT1 as NotECT

   If equipment is configurable in such as way as to only supply CE
   marks to ECT0 packets, and treat ECT1 packets identically to NotECT,
   or is upgradable to support this capability, doing so will eliminate
   the risk of unfairness."

[SM] Quick poll, does anybody know of equipment configurable to that degree? So is this a REALISTIC option or are we enumerating theoretical possibilities here? I fear it is the latter but would love to be wrong.

"4.2.  Configure Non-Coupled Dual Queue

   Equipment supporting RFC3168 may be configurable to enable two
   parallel queues for the same traffic class, with classification done
   based on the ECN field.

   Option 1:

   o  Configure 2 queues, both with ECN; 50:50 WRR scheduler

   o  Queue #1: ECT1 & CE packets - Shallow immediate AQM target

   o  Queue #2: ECT0 & NotECT packets - Classic AQM target

   o  Outcome

      *  n L4S flows and m long-running Classic flows

      *  if m & n are non-zero, get 1/2n and 1/2m of the capacity,
         otherwise 1/n or 1/m

      *  never < 1/2 each flow's rate if all had been Classic"

[SM] Please correct me if I am wrong, but that sems likely to introduce gratuitous re-ordering of packets in true rfc3168 flows, just like the main L4S design?

   "Option 2:

   o  Configure 2 queues, both with AQM; 50:50 WRR scheduler

   o  Queue #1: ECT1 & NotECT packets - ECN disabled

   o  Queue #2: ECT0 & CE packets - ECN enabled

   o  Outcome

      *  ECT1 treated as NotECT

      *  Flow balance for the 2 queues the same as in option 1"

[SM] This seems the safer option regarding potential re-ordering, and that fact shoud be explicitly mentioned, no?

"4.4.  ECT1 Tunnel Bypass

   Using an RFC6040 compatibility mode tunnel, tunnel ECT1 traffic
   through the RFC3168 bottleneck with the outer header indicating Not-
   ECT.

   Two variants

   1.  per-domain: tunnel ECT1 pkts to domain edge towards dst

   2.  per-dst: tunnel ECT1 pkts to dst"

[SM] Hrm, IMHO there are only two rationally defensible tunnel designs with regards to ECN psiible, either always copy to out on en- and copy to inner on de-capsulation, or never do so (basically ECN-opaque and ECN-transparent tunneling) all the other clever recommendations of the past lead us into the mess that we are in today. This proposal would be going in the opaque tunneling direction, except it seems to advertize doing this differentially for ECT(1) and ECT(0) which I consider being part of the problem, not of the solution.

"4.5.  Disable RFC3168 ECN Marking

   While not a recommended alternative, disabling RFC3168 ECN marking
   eliminates the fairness issue.  Clearly a downside to this approach
   is that classic senders will no longer get the benefits of Explict
   Congestion Notification.

4.6.  Re-mark ECT1 to NotECT Prior to AQM

   While not a recommended alternative, remarking ECT1 packets as NotECT
   ensures that they are treated identically to classic NotECT senders.
   However, this also eliminates the possibility of downstream L4S
   bottlenecks providing high fidelity congestion signals."

[SM] I appreciate that this list both of the ultima-ratio solutions with their immediate consequences, but without any bias which of the two is less despicable.

"5.  Researchers

5.1.  Detection of Classic ECN FIFO Bottlenecks

   TODO: Describe active testing methods, in-band or out-of-band, that
   can distinguish FIFO from FQ."

[SM] IMHO this will need to take care to explicitly elaborate which heuristics come with which failure modes. Also this needs to acknowledge that FQ can ameliorate the issue, but at least in ist stochastic version is not a full solution.

"5.2.  End-to-end measurement of L4S vs. Classic performance

   TBD"

[SM] Funny, that, as far as I can tell there is quite a lot of that still on the TODO list for team L4S ;)

Many thanks for starting this

Best Regards
	Sebastian

> On Jul 30, 2020, at 13:05, Greg White <[email-masked]> wrote:
> 
> TSVWG members-
> 
> I've posted a rough draft of Operational Guidance for L4S deployment.  This is not much more than an outline at this point, and is almost certainly incomplete even at that, so please read it with that in mind. 
> 
> -Greg
> 
> 
> From: "[email-masked]" <[email-masked]>
> Date: Thursday, July 30, 2020 at 4:53 AM
> To: Greg White <[email-masked]>
> Subject: New Version Notification for draft-white-tsvwg-l4sops-00.txt
> 
> A new version of I-D, draft-white-tsvwg-l4sops-00.txt
> has been successfully submitted by Greg White and posted to the
> IETF repository.
> 
> Name:		draft-white-tsvwg-l4sops
> Revision:	00
> Title:		Operational Guidance for Deployment of L4S in the Internet
> Document date:	2020-07-30
> Group:		Individual Submission
> Pages:		7
> URL:            https://www.ietf.org/internet-drafts/draft-white-tsvwg-l4sops-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-white-tsvwg-l4sops/
> Htmlized:       https://tools.ietf.org/html/draft-white-tsvwg-l4sops-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-white-tsvwg-l4sops
> 
> 
> Abstract:
>   This is an early, work-in-progress draft - a start at getting some of
>   the ideas from the mailing list and email exchanges on paper.
> 
>   This draft is intended to provide guidance to operators of end-
>   systems, operators of networks, and researchers in order to ensure
>   reasonable fairness between L4S and Classic flows sharing a single-
>   queue RFC3168 bottleneck link.  This draft identifies opportunites to
>   prevent and/or detect and resolve fairness problems in such networks.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 
> 
>

---

From: Greg White <[email-masked]>
Date: Mon, 3 Aug 2020 23:35:24 +0000

Apologies in advance for the mangled URLs in the quoted text below.  My IT department has implemented a system that mangles URLs on inbound email.

From: Sebastian Moeller <[email-masked]>
Date: Thursday, July 30, 2020 at 3:43 PM
To: Greg White <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] FW: New Version Notification for draft-white-tsvwg-l4sops-00.txt

Hi Greg,

my comments for sections 1 & 2 below (more will follow):

"1.  Introduction

   In the majority of network paths, including paths where the

   bottleneck link utilizes packet drops (either due to buffer overrun

   or active queue management) in response to congestion, as well as

   paths that implement a 'flow-queuing' scheduler such as fq_codel or

   Cobalt, and those that implement dual-Q-coupled AQM, L4S traffic

   coexists well with classic congestion controlled traffic."

[SM] This description ignores the birthday paradox issue that Pete Heist demonstrated. A misbehaving L4S will cause less havok there but it is not without side-effects on stochastic flow queueing systems, we might as well acknowledge that, no?

[GW] IIRC that issue can occur in fq_codel, but is much less likely in Cobalt (due to set-associative hashing).  Is that correct?

"   On network paths where the bottleneck link implements a shared-queue

   (FIFO) with an Active Queue Management algorithm that provides

   Explicit Congestion Notification signaling according to RFC3168, it

   has been demonstrated that when a set of long-running flows

   comprising both "Classic" congestion controlled flows and L4S-

   compliant congestion controlled flows compete for bandwidth, the

   classic congestion controlled flows may achieve lower throughput when

   compared to the L4S congestion controlled flows.  This 'unfairness'

   between the two classes appears to be more pronounced on longer RTT

   paths (e.g. 50ms and above) and/or at higher link rates (e.g. 50 Mbps

   and above)."

This is rather cautious I would use more drastic terms, the observed unfairness approaches starvation of the non-L4S flows and this text makes it sound like it is a minor almost theoretical concern.

[GW] We can definitely wordsmith the text, but we need to be sure that we are characterizing the situation accurately and not trying to portray things in drastic terms (or glossy ones) unless it is warranted.  My belief is that it isn’t warranted to use drastic terms here.  I think it is a fairly widely held view that single queue RFC3168 bottlenecks are uncommon at best and the majority of the ones that do exist are likely to be RED AQMs.  The testing with RED AQMs showed approximate fairness in most cases (though more testing would be welcome).  We’ll need to get a sense from the WG as to how to characterize the situation, given what we know (and what we don’t know).

"  The root cause of this unfairness is that RFC3168 does not

   differentiate between packets marked ECT0 (used by classic senders)

   and those marked ECT1 (used by L4S senders), and provides an

   identical congestion signal (CE marks) to both classes, whereas the

   two classes respond differently to that congestion signal."

How about keeping causality intact and frame this as a consequence of L4S redefining what CE means. There are reasons for doing that, but this text leaves it unclear who caused this problem (or rather the text implicates rfc3168, which IMHO violates temporal causality).

[GW] Point taken.  I’ll try to rearrange that.

"The result is that the

   classic senders respond to the CE marks provided by the bottleneck by

   yielding capacity to the L4S flows.  While this has not been

   demonstrated to cause starvation of the classic flows, the resulting

   rate imbalance can be a cause of concern."

Mmmh, https://protection.greathorn.com/services/v2/lookupUrl/43da5eeb-7a88-4655-a9ba-bec785235113/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=sce.dnsmgr.net&path=/results/ect1-2020-04-23-final/l4s-s2-twoflow/l4s-s2-twoflow-ns-cubic-vs-prague-codel1q_20ms_-160ms_tcp_delivery_with_rtt.svg

pretty much demonstrated starvation and that is with TCP-Prague with rfc3168 detection.

[GW] Ok, I will update the text to be more clear.   It seems that it can be demonstrated in contrived setups that it is possible to cause starvation.  That condition was a single queue CoDel (do these exist in the wild?) with heavily modified settings. CoDel with default settings did not show unfairness.

"2.  Per-Flow Fairness

   There are a number of factors that influence the relative rates

   achieved by a set of congestion controlled flows sharing a queue in a

   bottleneck link.

   TODO: discuss startup & convergence times, short flows, RTT-

   unfairness, differences in deployed CC algorithms, etc.

   TODO: also mention that flow sharding is commonplace, so per-flow

   fairness does not imply per-application fairness"

This obviously needs flashing out before one can meaningfully comment, but regarding the last section, you realize that application not the relevant classification for intermediate nodes? I would assume that for an ISP a (potentially weighted) per-end-host fairness would be the exact tool to avoid sharding as work-around... This is obviously not a new idea or comment, so if you opt for elaborating on fairness, please also include known remedies. Or better avoid that discussion here altogether.

[GW] Thanks. Yes per-end-host fairness could be useful in some situations.  AFAIK most residential ISPs implement some form of per-customer fairness, which could be argued is more appropriate in that context. I guess the point is that per-flow fairness isn’t always (or even usually) the most important metric for most users/systems.  It seems important to discuss this, seeing as per-flow unfairness is the main concern that has been raised, but I realize this section could be challenging to write in way that all will agree with.

Best Regards

  Sebastian

> On Jul 30, 2020, at 13:05, Greg White  wrote:

> From: "[email-masked]"

> To: Greg White

> URL:            https://protection.greathorn.com/services/v2/lookupUrl/f7c346c8-8443-44e5-b71e-435217709fda/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=www.ietf.org&path=/internet-drafts/draft-white-tsvwg-l4sops-00.txt

> Status:         https://protection.greathorn.com/services/v2/lookupUrl/e5e7af28-9419-4d0e-a559-650861096bf9/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=datatracker.ietf.org&path=/doc/draft-white-tsvwg-l4sops/

> Htmlized:       https://protection.greathorn.com/services/v2/lookupUrl/3b91a498-7692-49a5-aeb2-601ef0ba245a/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=tools.ietf.org&path=/html/draft-white-tsvwg-l4sops-00

> Htmlized:       https://protection.greathorn.com/services/v2/lookupUrl/813cf103-4e26-44a1-8c2f-a2648d5dacc0/327/acff24c9401e4e78e9981cf8beb4d98870d4f8bf?domain=datatracker.ietf.org&path=/doc/html/draft-white-tsvwg-l4sops

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 4 Aug 2020 04:05:08 +0000

> [GW] IIRC that issue can occur in fq_codel, but is much less likely in Cobalt (due to set-associative hashing).  Is that correct?

The set-associative hash (presently in Cake, not actually part of COBALT) does significantly push back the onset of flows colliding in the same queue.  It does not 100% eliminate the possibility, so you still have to manage the risk (product of likelihood and consequence) appropriately.  Two flows with mutually compatible congestion responses colliding is a mild enough consequence, but that is not nearly so clear for when the responses are incompatible, especially when it is the newcomer that is dominant.

What no fancy hashing algorithm can compensate for is when the distinction between flows is hidden by a tunnel encapsulation.  Since the L4S team has been adamant in other conversations that tunnels are a common feature on the internet, and I believe we should expect a mixture of conventional and high-fidelity flows within these tunnels, that should definitely factor into your analysis.

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 4 Aug 2020 12:30:24 +0200

Hi Greg,

more below in-line.

> On Aug 4, 2020, at 01:35, Greg White <[email-masked]> wrote:
> 

	[SM2] As Jonathan wrote less likely in cake, but still possible. Also, my hypothesis is that fq_codel is more widely in use than cake.

> [GW] We can definitely wordsmith the text, but we need to be sure that we are characterizing the situation accurately and not trying to portray things in drastic terms (or glossy ones) unless it is warranted.  

	[SM2] Well, the thing is Pete Heist's testing (see https://github.com/heistp/sce-l4s-ect1#typical-internet-jitter) demonstrated that L4S with rfc3168 detection, which as far as I can tell is still required for a L4S-compatible transport, is quite bad at achieving fairness, even with RED. 
Sidenote: calling 50ms "longer" RTT paths supports my claim, that L4S is really ONLY ever designed and more cricially TESTED for low-hop count low RTT paths.

> My belief is that it isn’t warranted to use drastic terms here.  

	[SM2] I accept your believe, but think that we should base operator recommendations on hard data if possible, that way our respective beliefs do not muddy the waters.

> I think it is a fairly widely held view that single queue RFC3168 bottlenecks are uncommon at best and the majority of the ones that do exist are likely to be RED AQMs.  

	[SM2] Data please, or we are back at Bob's line or reasoning, that because he is bad at measuring something that something might not even exist.

> The testing with RED AQMs showed approximate fairness in most cases (though more testing would be welcome).

	[SM2] What is your definition of "approximate fairness"? Can we agree on calling imbalances inside of 2 orders of base2-magnitude (aka faktor of 4) as approximately fair and everything beyond 3 orders of bas2-magnitude (factor 8) starvation? If you disagree with my numbers, please propose your own set, but let's get away from the idea that approximate fairness and starvation would be concepts that can not be defined numerically.

>  We’ll need to get a sense from the WG as to how to characterize the situation, given what we know (and what we don’t know). 

	[SM2] With all due respect this WG has not been great at giving feed-back on simple questions like "what is starvation" in the past; as much as I would like to be wrong, I do not see this changing now.

> [GW] Point taken.  I’ll try to rearrange that.

	[SM2] Thanks!

> [GW] Ok, I will update the text to be more clear.   It seems that it can be demonstrated in contrived setups that it is possible to cause starvation.  That condition was a single queue CoDel (do th…

	[SM2] Okay, no please proof that these conditions do not actually exist in the real world and you are scot free... And no, changing the target value is not a contrived set-up, for example have a look at the PIE RFCs where there it is mentioned that for short RTT paths adjusting the latency target is recommended, and for codel/fq_codel/cake the same logic applies. This gets ironic, given the fact that L4S itself is really only taylored for similarly short RTT paths...

> [GW] Thanks. Yes per-end-host fairness could be useful in some situations.  AFAIK most residential ISPs implement some form of per-customer fairness, which could be argued is more appropriate in th…

	[SM2] Okay, I see why you want to discuss that here. Please keep in mind, that while per-flow-fairness is not the ultimate solution it clearly is further along that path than L4S's "anything goes". Specifically everything that works around the strikter fq-sheduling will also wreck havoc on "anything goes"-scheduling, so IMHO the argument that FQ is not perfect is beyond the point, as it seems in all relevant criteria to be better or equal to "anything goes" (mostly better) with the sole exemption of computational complexity. I really believe that this document could do without opening that can of worms...

Best Regards
	Sebastian