Subject: [tsvwg] Gorry Fairhurst Individual thoughts on choosing whether/how to advance ECN work.

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 15 May 2020 15:54:04 +0100

Some people have asked me about various aspects of this debate, and 
towards the end of this process, I’d like to share (as an individual) 
some of my thoughts. Whatever the outcome, I think we have developed a 
lot of material to help us move forward as a group.

We do have a shortage of bits (not uncommon in the IETF), and there are 
many parts to this story:

1. A signal to the network to steer traffic towards or away from the new 
marking: Flow queues avoid needing the endpoint reaction to be known by 
the AQM, and allow each flow have a separate treatment, but have to 
consider the awkward  question of aggregates of flows, and applications 
that choose to shard over multiple flows.

2.  Signal to the network to support network operations, especially for 
wide-scale experiment: It would be good for an operator to know a flow 
is expected to be treated different in their network segment or those 
network segments that follow, and whether they for operational reasons 
wish to avoid that different treatment within their network (e.g., 
following a discovered configuration issue). L4S makes the two transport 
behaviours explicit. I expect some thought would be needed about how 
ECT(1). I think SCE makes it hard to explicitly exclude a flow from 
being a part of an experiment, so we need to be sure that any remarking 
is always benign or helpful. My own view is that using other classifiers 
such as flow IDs or DSCPs brings a lot of complexity to operations, 
because these are used for other purposes in various use-cases, so I am 
not yet clear that helps. L4S introduced an explicit signal (albeit 
on-trust) that the transport will gracefully react to CE marks in an 
appropriate way. I think L4S makes the two transport behaviours explicit 
to observers on the path, I liked that.

3. I don’t know the full set of uses for single queue AQMs that do ECN 
marking:  I would anticipate issues with any AQM that does not conform 
to current transport expectations - just as I anticipate issues with 
routers that configure old-style RED-based ECN marking using large 
thresholds and measuring over deep queues. Guidance will be needed.

4. ECN transport receiver reaction: It would help to have a way to 
reliably tell the receiving endpoint(s) whether the marks they see 
conform to their transport expectations - i.e. whether a packet was 
marked in the new way. SCE nicely makes the marking behaviour explicit, 
I like SCE from this respective.

5. We need to engineer transport methods that work: This is tsvwg’s main 
focus - I would expect to be able to update behaviour as we move forward 
with any work.  As an individual commentator, I expect significant work 
will finally be needed to arrive there (DCTCP was not designed for the 
Internet), but that I suggest we need to decide now on the path to take.

6. We must be able to deploy, manage, and measure the methods for paths 
involving tunnels and encapsulations: This is hard, encaps/tunnels are 
widespread, and tsvwg has devoted significant energy with INTAREA to 
update IETF specs in this area.This was one of the biggest activities 
that faced the proponents of deploying new ECN work, holding up the WG 
while this was done. Designing to work well with tunnels is crucial.

7. We still need to define BCPs that support operation: The use of 
scheduling/queueing fits in this area, but methods depend on the number 
of flows and the way you aggregate - what is applicable in a home CPE or 
WiFi base station is unlikely to be applicable in a service aggregation 
router or an enterprise WiFi deployment. That's why RFC7567 currently 
calls out scheduling, policing and protection as useful, but optional 
components. As an individual commentator, I can see some scenarios which 
benefit from policers, scheduling and/or protection mechanisms, and 
these may well become the accepted best practice for using ECN - and 
hope transports will evolve to mitigate issues, but I would expect 
details to remain scenario dependent.

Alas, I don’t see how we can do all using just the current ECN field, 
and that’s why I say we have a shortage of bits.

My own personal viewpoint is that we need to decide on the benefits for 
enabling ECN in the long term.

Best wishes,

Gorry Fairhurst
(not as a chair)

---

From: Joseph Touch <[email-masked]>
Date: Fri, 15 May 2020 09:49:10 -0700

Hi, all,

> On May 15, 2020, at 7:54 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
> Alas, I don’t see how we can do all using just the current ECN field, and that’s why I say we have a shortage of bits.

I agree with Gorry on the point above.

I also haven’t quite seen a brief summary of the utility of these bits as a signal in either direction (endpoint to net; net to endpoint) given the assumption that “everybody lies”.

IMO, lying is part of why DSCPs are useful only where either redundantly validated on network ingress or inside managed networks.

So my questions - not really experiments (as per one of the options) are:

	- assuming endpoints lie and/or network nodes lie (either by what they indicate or what they rewrite), are either of these options still safe?

	- assuming everybody lies (as above), are either of these options still useful? (i.e., more than just safe)

If not - and partly based on Gorry’s observation above, my inclination is not to define these bits for such uses at all.

Joe

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 15 May 2020 21:14:53 +0300

> On 15 May, 2020, at 7:49 pm, Joseph Touch <[email-masked]> wrote:
> 

It's an important question, so thank you for raising it.  Ultimately the answer to such questions boils down to what incentives an actor may have to abuse the mechanisms controlled by each codepoint.

Taking the question of lying endpoints first, SCE is reasonably robust in that respect.  The normal case is that an SCE sender originates ECT(0), the same as normal ECN.  If it instead originates ECT(1) or CE, it is equivalent to pre-applying a congestion signal and removing the network's ability to give the transport information.  The sender could already ignore that information without giving an explicit signal of doing so, so the latter is not incentivised.  Meanwhile the receiver may choose to ignore the ECT(1) signal and not feed it back to the sender - which is what existing RFC-3168 receivers do anyway, and this is allowed for in SCE's design.  Or they might falsely generate such a signal, but that would reduce its own throughput, so that is not incentivised.  Other forms of lying are already possible with RFC-3168, which SCE does not exacerbate.

In the L4S system, a sender may falsely mark a low-latency transport with ECT(0), or a conventional transport with ECT(1), to drop it into the "wrong" queue and AQM treatment regime at the bottleneck.  I think the former case is the most damaging one, because it would result in other traffic being forced out of the way of the misbehaving sender, which thereby gains a throughput advantage under congested conditions.  This scenario should be familiar from the usual concern about Diffserv PHBs.  I offer no opinion about misbehaving receivers in this context.

If we now turn to the prospect of lying middleboxes, SCE again inherits much existing analysis from RFC-3168.  The new distinction between ECT(0) and ECT(1) is meaningful and carries congestion information, and middleboxes are currently supposed to preserve that distinction (a rule originally intended to support Nonce Sum, a proposed prophylactic against ECN misbehaviour that proved to be unnecessary).  SCE permits changing ECT(0) to ECT(1) in the network, but not the reverse.  The latter would erase congestion information; this is however backstopped by the continued use of CE at a higher level of congestion, so is relatively benign.  A DoS scenario could employ falsely marking with ECT(1), but this could also be done (with greater efficiency) by falsely marking CE, and ultimately the same sorts of mitigations would apply as in RFC-3168.

It's also difficult to see how an attacker could gain benefit from manipulating the ECN field at a middlebox in the L4S system, besides the obvious ploy of simulating the sender falsely setting ECT(0) on L4S traffic.  In part this is because the behaviour of a correctly functioning L4S system is already so fragile, that it is not necessary to do much to disrupt its behaviour.  In particular, one of the worst things a middlebox can do is to simply ignore the ECT(1) codepoint and treat both classes of traffic as indistinguishable; but that is precisely what existing RFC-3168 AQMs are permitted, even encouraged to do.  Recent developments in TCP Prague were intended to detect this case at the endpoints, but actually fail to do so reliably.

I hope that at least begins to answer your question.

 - Jonathan Morton

---

From: Joseph Touch <[email-masked]>
Date: Fri, 15 May 2020 12:50:21 -0700

I guess it might be useful to highlight the following parts of my question:

> On May 15, 2020, at 11:14 AM, Jonathan Morton <[email-masked]> wrote:
> 
>> On 15 May, 2020, at 7:49 pm, Joseph Touch <[email-masked] <mailto:[email-masked]>> wrote:
>> 

Is there a brief summary of what you wrote?

---

From: Paul Vixie <[email-masked]>
Date: Fri, 15 May 2020 20:18:11 +0000

On Friday, 15 May 2020 16:49:10 UTC Joseph Touch wrote:
> > Alas, I don’t see how we can do all using just the current ECN field, and
> > that’s why I say we have a shortage of bits.

+1.

> I also haven’t quite seen a brief summary of the utility of these bits as a
> signal in either direction (endpoint to net; net to endpoint) given the
> assumption that “everybody lies”.
> 
> IMO, lying is part of why DSCPs are useful only where either redundantly
> validated on network ingress or inside managed networks.
> 
> 	- assuming endpoints lie and/or network nodes lie (either by what 
> they indicate or what they rewrite), are either of these options still safe?
> 
> 	- assuming everybody lies (as above), are either of these options 
> still useful? (i.e., more than just safe)

on a cost:benefit analysis, if ECT(1) is defined to be an output from the 
network the risk (to first and second parties) of lies when is higher, and the 
benefit (to the third) is lower. therefore if this WG were to assign meaning 
to the ECT(1) bit based on risk of lies to good guys and benefit of lies to 
bad guys, the output choice would prevail.

as an output that is intended to serially convey a changing fraction of path 
congestion, the benefit to a liar of lying about it would be to cause lower 
network utilization for affected flows. since the lie would have to be told by 
an on-path actor, this might even be within their established purview. here, 
the network is pressing for different endpoint behaviour.

as an input that is intended to select DCTCP-style shallow queues for some 
traffic, the benefit to the liar of lying about it would be to cause the 
liar's flows to be able to better compete against inside-the-DC flows. this is 
a denial-of-service vector at worst, and a nonconstructive competition form at 
best. here, the endpoint is pressing for different network behaviour.

> If not - and partly based on Gorry’s observation above, my inclination is
> not to define these bits for such uses at all.

enshrining the riskier design (ECT(1)-as-input) in all IP flows because it 
helps datacenter efficiency as long as that datacenter bleaches on ingress, 
strikes me as a poor tradeoff of cost and benefit for the last IP TOS bit.

perhaps the smart edge, dumb core model doesn't work perfectly for datacenter 
networks. but as fq_codel shows, it works extremely well for edge networks, 
who are vastly more numerous and which provide an at least equal share of the 
"network effect" value of internetworking.

so, i am +1 to joe's inclination as quoted above.

-- 
Paul

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 15 May 2020 23:38:20 +0300

> On 15 May, 2020, at 10:50 pm, Joseph Touch <[email-masked]> wrote:
> 

> 	- assuming endpoints lie and/or network nodes lie (either by what they indicate or what they rewrite), are either of these options still safe?

I believe SCE is safe under the assumption of lying endpoints and/or middleboxes, to approximately the same extent that RFC-3168 ECN is.  Either the lie has a benign effect, or it ultimately harms the liar, for each class of possible lie that has been brought to my attention.

Conversely, it is straightforward to demonstrate a case in L4S where lying gains a significant advantage for the liar: a sender may mark its traffic ECT(0) but then implement DCTCP-style congestion control.  The demonstrated in-network component of L4S does not protect against this, and at least some existing networks are also vulnerable to it.

> 	- assuming everybody lies (as above), are either of these options still useful? (i.e., more than just safe)

Under the assumption that everybody lies, RFC-3168 ECN is itself pretty useless.  But it is used and useful in practice, so the underlying assumption that everyone lies should be re-examined.  The non-deployment and ultimate retirement of Nonce Sum lends evidence to this viewpoint.

Under the weaker assumption that everybody *can* lie but would only do so given some incentive, I believe SCE is not only safe but remains useful in most circumstances.

Under this same weaker assumption, L4S is neither safe nor useful in my opinion.

 - Jonathan Morton

---

From: Paul Vixie <[email-masked]>
Date: Fri, 15 May 2020 20:48:59 +0000

On Friday, 15 May 2020 20:38:20 UTC Jonathan Morton wrote:
> ...
> Under the assumption that everybody lies, RFC-3168 ECN is itself pretty
> useless.  But it is used and useful in practice, so the underlying
> assumption that everyone lies should be re-examined.  The non-deployment
> and ultimate retirement of Nonce Sum lends evidence to this viewpoint.

"nothing which can be abused won't be" is my formulation of choice these days.

> Under the weaker assumption that everybody *can* lie but would only do so
> given some incentive, I believe SCE is not only safe but remains useful in
> most circumstances.
> 

i don't know enough to speak to the utility question. but if asymmetry in 
risk:benefit is to be our guide, i agree it would lead ECT(1) as an output.

-- 
Paul

---

From: Pete Heist <[email-masked]>
Date: Sat, 16 May 2020 00:14:51 +0200

> On May 15, 2020, at 10:38 PM, Jonathan Morton <[email-masked]> wrote:
> 

For a test result along these lines, the plot below shows CUBIC vs Prague when:
1) disabling bottleneck detection (a module parameter) and
2) using an iptables rule to change ECT(1) to ECT(0) on outgoing TCP segments without the SYN flag sent

http://sce.dnsmgr.net/results/ect1-2020-05-15T235113-s10-gaming-ect1-ect0/l4s-s10-gaming-ect1-ect0/l4s-s10-gaming-ect1-ect0-ns-cubic-vs-prague-dualpi2-20ms_tcp_delivery_with_rtt.svg

Capture files and so forth are here:

http://sce.dnsmgr.net/results/ect1-2020-05-15T235113-s10-gaming-ect1-ect0/l4s-s10-gaming-ect1-ect0/

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 16 May 2020 01:52:36 +0300

> On 16 May, 2020, at 1:14 am, Pete Heist <[email-masked]> wrote:
> 

Exactly.  What you see here is a simulated lying sender on a 20ms, 50Mbps path, starting up shortly after a TCP CUBIC flow that is not lying.  There is a DualQ AQM at the bottleneck which would normally be able to distinguish the Prague flow from CUBIC and apply the different congestion signalling that it needs.  Since that distinguishing identifier has been deliberately removed, everything reverts to DCTCP behaviour in which the CUBIC flow is squeezed out of the network, and the lying Prague sender gets 95%+ of the throughput.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 16 May 2020 12:44:40 +0100

I do not think this relates to the topics I raised in the subject line. You see. To have cut my points and started a new thread.

To be clear, I don’t see how the impact of a non-conformant flow, is a topic that concerns whether the ECN work can progress. I agree it is a design issue for transports irrespective of ECN  - since I also expect TCP Reno v. A non-Reno flow can do poorly in a loss-based queue, not just an ECN queue.

Gorry

Sent from my iPad

> On 15 May 2020, at 23:53, Jonathan Morton <[email-masked]> wrote:
> 
> 
>>

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 16 May 2020 23:32:30 +0300

> On 16 May, 2020, at 2:44 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
> I do not think this relates to the topics I raised in the subject line. You seem to have cut my points and started a new thread.

To be fair, I was just responding to Joe Touch.

I also think that the congestion safety questions he raised are central to the discussion as framed in your subject line.  I'm sure they would come up in later stages of IETF process anyway, as the security groups (for example) started to pay attention, and it's probably best to anticipate those objections beforehand.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 22 May 2020 23:44:04 +0100

Paul,

> the network is pressing for different endpoint behaviour.

[BB] I don't think any of the above is materially relevant to whether 
there should be one output or two. The number of outputs doesn't affect 
the network's ability to lie about one or both of them.

> best. here, the endpoint is pressing for different network behaviour.

[BB] The sender's congestion control (or lack thereof) is about 
behaviour. Misbehaviour and lying are orthogonal. So all this about 
lying or not is irrelevant to congestion control behaviour (as Gorry 
explained).

In Jonathan's example, an L4S sender sets ECT0, but behaves more 
aggressively in response to feedback (as if it was setting ECT1) which 
gives it more capacity share. Even if L4S had never existed, any sender 
can set ECT0 and behave more aggressively to attain more capacity share.

A sender can also set ECT1 to get classified into the L4S queue, and 
still behave more aggressively than other L4S sources to get more 
capacity share than them.

In summary, attaining capacity share is nothing to do with the truth of 
a codepoint, because capacity shares aren't determined by whatever 
codepoint the sender tells the network. Capacity shares aren't even 
determined by the network at all. They are determined by the sender's 
behaviour, irrespective of what codepoint it sets.

> strikes me as a poor tradeoff of cost and benefit for the last IP TOS bit.

[BB] Can you explain what made you say the second of these three lines, 
so we can try to unpick what's in your mind. There's surely some 
misconceptions behind these words. Where have you seen anything about 
helping datacenter efficiency? And what's this condition about bleaching 
on ingress to a DC?

> "network effect" value of internetworking.

[BB] Again, there seems to be some misunderstanding that L4S is about 
data centre networks. What's all this about?

It might be worth reading 
https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-06#section-6.3 
where I think you'll see that the primary deployment use-case for L4S is 
similar to that of FQ-CoDel. Nonetheless, being simpler, it is being 
picked up by those operating the downstream bottleneck in to the access 
link (the BNG, metro node, CMTS, etc), not just the upstream bottleneck 
(home gateway).

It's also simple enough for other possible bottlenecks, e.g. peering 
points, DC ToRs, hypervisors, DC access links, campus access links, and 
so on.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Paul Vixie <[email-masked]>
Date: Sat, 23 May 2020 03:10:00 +0000

bob,

On Friday, 22 May 2020 22:44:04 UTC Bob Briscoe wrote:
> >>> and

your arguments may fail by being too detailed and overlong, so whatever merits 
it had are not never noticed. brevity is the soul of relevance. in the old 
days we called this "proof by vigorous reassertion", but, we were joking.

> helping datacenter efficiency? And what's this condition about bleaching

in the context of that thread to that point, the operators and vendors who had 
spoken in favour of the input model had used short-RTT ("microseconds rather 
than milliseconds") examples, and had spoken extensively of switches 
specifically as handling these signal patterns not routers. this excludes the 
heterogeneous global WAN as a use case, and i believe this exclusion was 
deliberate, because so many local policies are already in place that new 
congestion related signalling can be expected to have a near-infinite tail in 
that environment.

tellingly, this means no operator or vendor will speak for the global WAN as a 
use case, because no operator or vendor makes most of their revenue that way; 
there is no value-add they can offer their customers that would have that 
reachability domain. that voice was by definition absent from the recent poll, 
other than random bit-heads like dave taht who seem to think that a rising 
tide could lift all boats. (that's not a compelling argument, as you saw.)

> similar to that of FQ-CoDel.

i'm not sure you meant to beg that question, but here it is. why isn't cake or 
fq-codel adequate, and can you answer using heterogeneous wide area examples?

also: noting that the L4S concept predates the bufferbloat project and its 
results such as modern age-based queue management at the endpoints, how has 
the L4S approach evolved to take advantage of this new situation and to focus 
on remaining unsolved problems -- which gets us back to: what are those?

> Nonetheless, being simpler, it is being

are you aware of the logical fallacies you're invoking here? that is, are you 
trying to use debate tricks, or have you been misled? (hint: L4S isn't 
simpler, and theoretic possibility does not speak to adoption likelihood.)

-- 
Paul

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 May 2020 12:57:56 +0100

Paul,

On 22/05/2020 23:44, Bob Briscoe wrote:
> there
> should be one output or two. The number of outputs doesn't affect the
> network's ability to lie about one or both of them.
>
>>
> behaviour.
> Misbehaviour and lying are orthogonal. So all this about lying or not is
> irrelevant to congestion control behaviour (as Gorry explained).
>
> gives it
> more capacity share. Even if L4S had never existed, any sender can set 
> ECT0
> and behave more aggressively to attain more capacity share.
>
> still
> behave more aggressively than other L4S sources to get more capacity 
> share
> than them.
>
> In summary, attaining capacity share is nothing to do with the truth of a
> codepoint, because capacity shares aren't determined by whatever 
> codepoint
> the sender tells the network. Capacity shares aren't even determined 
> by the
> network at all. They are determined by the sender's behaviour, 
> irrespective of
> what codepoint it sets.

[BB]
PV's argument: 15 lines.
BB's argument: 18 lines.

PV's response:

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 23 May 2020 16:07:23 +0100

Paul,

> congestion related signalling can be expected to have a near-infinite tail in

[BB] Data center had not been mentioned on that thread.
(And no operator or vendor had spoken on that thread.)
Whatever...

> reachability domain. that voice was by definition absent from the recent poll,

[BB] Nearly all the 35 operators and vendors that agreed to be listed in 
the recent interim mtg are planning to deploy L4S for access networks, 
mostly cable or mobile:
https://datatracker.ietf.org/meeting/interim-2020-tsvwg-03/materials/slides-interim-2020-tsvwg-03-sessa-3-ect1-slides.pdf#page=13

> tide could lift all boats. (that's not a compelling argument, as you saw.)

[BB] I see you hand out unprovoked unpleasantness to others as well. 
That's reassuring (not).
I will filter you out if you don't stop tho.

> fq-codel adequate, and can you answer using heterogeneous wide area examples?

[BB] For your FAQ, I recently prepared this answer:
http://bobbriscoe.net/presents/2004ietf/202005L4S-brief.pdf#page=4

Briefly, L4S has order of magnitude lower tail delay than FQ_CoDel, 
which is important for real-time interaction, e.g. applications like 
gaming, remote interaction and control, or this cloud-rendered 
interactive video:
     https://riteproject.eu/dctth/#1511dispatchwg

Also, L4S enables deployment of solutions to the throughput scaling 
problem with TCP congestion control (the root cause of the problem 
FQ_CoDel mitigated).

> simpler, and theoretic possibility does not speak to adoption likelihood.)

[BB] I don't think anyone except you is arguing that FQ_CoDel is simpler 
than the DualQ (because it isn't). You can compare this pseudocode 
yourself, which also links to the Linux code:
https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-11#appendix-A.1

The Coupled DualQ solely classifies at the IP layer (ECN field). It 
consists of two AQM instances:
* one instance of an AQM for the Classic queue, with similar complexity 
to /one/ of the per-flow CoDel instances.
* one instance of a brain-dead-simple stateless AQM for the L4S queue.

I'm seeing that vendors of larger scale equipment /have/ implemented it 
and operators /have/ plans to adopt it, whereas they didn't adopt 
FQ_CoDel. I'm suggesting an explanation for that (based on what they've 
told me), not talking theoretic possibilities.

Another explanation is that L4S enables new interactive applications 
that are currently infeasible over the WAN.

Regards

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 23 May 2020 17:33:23 +0200

Bob,

> On May 23, 2020, at 17:07, Bob Briscoe <[email-masked]> wrote:
> [...]
> 
> Another explanation is that L4S enables new interactive applications that are currently infeasible over the WAN.

	[SM] I have asked this before, but I am happy if I get a response this time: Which novel applications are possible now that the queuing delay has been reduced from ~10ms (fq_codel/cake, measured under real life traffic conditions on life internet access links) to ~1ms (L4S. dual queue coupled AQM under laboratory conditions)? This is the state of the art you compare against, not multiple 100s milliseconds of latency-under-load.
	Note, how for this question, I accept your claim of 1ms queueing delay even though testing indicates that this does not hold under all traffic patterns one is prone to encounter on a real internet access link (e.g. congestion on the reverse path). But really which application can not tolerate this additional 9ms of delay, yet is still a viable idea to deploy over the open internet? I note that real time control applications/reaction-time bound game services like google stadia and nvidia gforce now, are already deployed in today's internet, there really is nothing novel left as far as I can see
	Thanks in advance for the list of relevant new use-cases.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 23 May 2020 21:02:10 +0200

Dear Bob,

> [BB] I don't think anyone except you is arguing that FQ_CoDel is simpler than the DualQ (because it isn't). You can compare this pseudocode yourself, which also links to the Linux code:

	I note that the fq-codel RFC contains zero lines of pseudocode, so by this measure it must be infinitely simpler than the dual queue coupled AQM, no?

But joking aside, it seems like an odd choice to compare the complexity of an qdisc in the main-line kernel with an out of line qdisc under active development, but if we attempt to do this, surely 724 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_fq_codel.c (plus 310 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_codel.c) and 746 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_dualpi2.c do not seem to support your hypothesis, unless we want to quibble over individual lines of code here. And yes the runtime cost for fq_codel probably is a bit higher than for dualq, but I would be amazed if we would talk about an order of magnitude (base2) here...

My point is, please refrain from completely unhelpful complexity comparisons by virtue of pseudeocode, especially if not both qdiscs to be compared actually exist in pseudocode in the first place....

> 
> The Coupled DualQ solely classifies at the IP layer (ECN field). It consists of two AQM instances:

	[SM] True, until it becomes obvious that without the optional queue protection mechanism dualq severely lacks behind fq_codel in real-world usefulness. Once you add the queue protection to the requirements, your are looking at L4 headers again. That said, there is absolutely no reason, why fq_codel could not look only at the IP header for its hashing needs, if you really want that (and interestingly cake offers such flow-definition modes).

> * one instance of an AQM for the Classic queue, with similar complexity to /one/ of the per-flow CoDel instances.

	[SM] +1; it just turned out in the real world that the performance of a single queue AQM simply does not deliver what the end-users actually want from an AQM so that nobody went for a single queue CODEL. I see absolutely no reason, why this should change now with L4S. Now, I only talk to end-users, ISPs might see this differently (but ultimately will need to convince their users that L4S is actually an improvement).

> * one instance of a brain-dead-simple stateless AQM for the L4S queue.

	[SM] Since the two AQMs are coupled, the required state is simply kept on the non-LL side, claiming that it is stateless seems a bit disingenuous, no?

> 
> 
> I'm seeing that vendors of larger scale equipment /have/ implemented it and operators /have/ plans to adopt it, whereas they didn't adopt FQ_CoDel.

	[SM] Interestingly, end-users interested in low-latency and sharing their access links between, say gaming and streaming, opted for fq_codel and cake. Even though this requires post-bottleneck ingress shaping and is computationally relatively costly. So the relevant market has spoken and adopted FQ_CODEL. 

> I'm suggesting an explanation for that (based on what they've told me), not talking theoretic possibilities.

	[SM] Good marketing of the L4S effort? Unless you kick the tires yourself and simply believe the claims in the L4S drafts it seems like the perfect solution, optimal performance for low computational effort. It is only after you realize that this performance is neither robust nor reliable that questions will arise.

> Another explanation is that L4S enables new interactive applications that are currently infeasible over the WAN.

	[SM To repeat myself, none such applications that are both unfeasible today and are suitable for being run over the best-effort internet have so far been proposed. I am not claiming that L4S might not improve upon the state of the art at all, but this specific claim of currently impossible novel interactive applications seems wishful thinking. Case in point google satia, nvidie gforce now, shadow cloud gaming, all offer interactive real-time reaction-time bound games over the existing internet. That is your market, right there, already working over the existing internet, what are you going to offer?

Best Regards
	Sebastian

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Sun, 24 May 2020 00:36:44 +0300 (EEST)

On Sat, 23 May 2020, Sebastian Moeller wrote:

> > [BB] I don't think anyone except you is arguing that FQ_CoDel is 
> > simpler than the DualQ (because it isn't). You can compare this 
> > pseudocode yourself, which also links to the Linux code: 
> 	I note that the fq-codel RFC contains zero lines of pseudocode, so 
>       by this measure it must be infinitely simpler than the dual queue 
>       coupled AQM, no?
>
> But joking aside, it seems like an odd choice to compare the complexity 
> of an qdisc in the main-line kernel with an out of line qdisc under 
> active development, but if we attempt to do this, surely 724 lines for 
> https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_fq_codel.c 
> (plus 310 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_codel.c) 

And what about the header files that in this case actually implement part 
of CoDel (rather than just containing some func signatures)?

-- 
 i.

> and 746 lines for https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/net/sched/sch_dualpi2.c do not seem to support your hypothesis, unless we want to quibble over individual lines of code here. And yes the runtime cost for fq_codel probably is a bit higher than for dualq, but I would be amazed if we would talk about an order of magnitude (base2) here...    
> 
> My point is, please refrain from completely unhelpful complexity 
> comparisons by virtue of pseudeocode, especially if not both qdiscs to 
> be compared actually exist in pseudocode in the first place....
>

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 24 May 2020 00:00:22 +0200

HI Ilpo,

> On May 23, 2020, at 23:36, Ilpo Järvinen <[email-masked]> wrote:
> 

	[SM] Let me cite myself here:
"My point is, please refrain from completely unhelpful complexity comparisons by virtue of pseudeocode, especially if not both qdiscs to be compared actually exist in pseudocode in the first place...."

I was not so much trying to offer an academically correct complexity comparison, but to show that Bob's approach was/is a waste of everybody's time and not useful at all, and that it is rather simple to do a bit better, by at least comparing real code instead of pseudocode...

But sure if we want to go that unproductive route here are some missing headers:

164 lines: https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/include/net/codel.h
255 lines: https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/include/net/codel_impl.h
74 lines: https://github.com/L4STeam/sch_dualpi2_upstream/blob/sch_dualpi2/include/net/codel_qdisc.h

724+310+164+255+74 = 1527

versus 746 for the dualq, but that uses its own private PIE version, let's see how/if that survives upstreaming into the linux kernel.

Best Regards
	Sebastian

P.S.: I assume you actually had a look at the relevant files yourself and just omitted the numbers to keep things concise.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 24 May 2020 00:09:21 +0200

Hi Ilpo,

I just realized, that the dualQ alone is not really worth comparing to fq_codel, as it requires TCP Prague for proper functionality, so add at least 892 lines (https://github.com/L4STeam/linux/blob/testing/net/ipv4/tcp_prague.c) to the dualq ledger:
fq_codel: 724+310+164+255+74 = 1527
duaQ: 746+ 892 = 1638

I hope you agree that there is no clear significant complexity difference between L4S and fq_codel if measured by lines of code. I am not claiming that that is an ideal measure, but it sure beats lines of pseudocode.

Best Regards
	Sebastian

> On May 24, 2020, at 00:00, Sebastian Moeller <[email-masked]> wrote:
> 
> HI Ilpo,
> 
>

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 24 May 2020 01:52:47 +0300

> On 24 May, 2020, at 1:09 am, Sebastian Moeller <[email-masked]> wrote:
> 

There are definitely limitations in comparing implementation complexity by lines of code, especially as implementing certain algorithms in hardware requires more complexity than others, despite needing only similar lines of C.  But as long as we're doing this, let's compare apples to apples and use the nearest equivalents in SCE:

net/sched/sch_cnq_codel_af.c -  926 lines
net/sched/sch_lfq_cobalt.c   - 1049 lines

net/ipv4/tcp_dctcp_sce.c     -  224 lines
net/ipv4/tcp_reno_sce.c      -  223 lines
net/ipv4/tcp_cubic_sce.c     -  655 lines

The qdiscs above contain private copies of their AQMs and all relevant algorithms.  So from this we can assemble a simple SCE implementation with 926+224 = 1150 lines, or a more sophisticated one with 1049+655 = 1704 lines.

The latter includes such niceties as HyStart, improved scaling to high BDPs, flow isolation, and queue overload protection in addition to the basic AIMD and HFCC functionalities, for only a few more lines of code relative to the combination of DualQ and TCP Prague, which lack all of those additional features.

We could also include net/sched/sch_cake.c, which at 3143 lines represents an FQ-AQM system that is definitely more complex than anything previously mentioned in this thread, but has nevertheless been deployed in practice in its intended role.  Of course, nobody sensibly expects Cake to be deployed in high-speed core networks, but that's not what it was designed for.

 - Jonathan Morton

---

From: Paul Vixie <[email-masked]>
Date: Sat, 23 May 2020 23:49:39 +0000

On Saturday, 23 May 2020 22:09:21 UTC Sebastian Moeller wrote:
> ...
> 
> I hope you agree that there is no clear significant complexity difference
> between L4S and fq_codel if measured by lines of code. I am not claiming
> that that is an ideal measure, but it sure beats lines of pseudocode.

complexity is better measured in terms of state variables and transitions; 
there's nothing published i can find which compares these two competing 
approaches in those terms, and so i'm inclined to think in terms of network 
carried vs. edge carried complexity. it's the placement in this case, not the 
magnitude, that concerns me. i liked the fq_codel approach because it helped 
immediately and measurably at every increment of deployment, nobody had to 
deploy on faith as in ipv6 and dnssec that benefits would come later when 
others also deployed. so if you're counting fan boys, i'm an edge-complexity 
guy, even if the mass of all those edges each carrying all that flow state is 
higher overall than a smart-core approach.

i realize that nobody is proposing the fr/atm model (cir, br, per flow) but i 
still get cold chills down my spine when i hear a proposal for core complexity 
-- of whatever kind.

-- 
Paul

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 24 May 2020 00:53:51 +0100

Sebastian,

> 	[SM] I have asked this before, but I am happy if I get a response this time: Which novel applications are possible now that the queuing delay has been reduced from ~10ms (fq_codel/cake, measured u…

As explained in the email you are responding to, tail latency is 
important for real-time. The 10ms you report for FQ_CoDel is roughly its 
90th percentile queue latency, which is an irrelevant metric for 
real-time (if the stream was being rendered after the 90%ile delay, 10% 
of the packets would arrive too late to be rendered).

In http://bobbriscoe.net/presents/2004ietf/202005L4S-brief.pdf (slide 4) 
I've used the 99.9th percentile delay for the rendering deadline, which 
reduces the fraction of packets that cannot be used to 0.1%.
In the experiment reported in slide 3 you'll see the P99.9 latencies are 
32ms for FQ_CoDel and 4.5ms for L4S. On slide 4 this is translated into 
reach in fibre, taking the generally accepted 50ms max delay for 
'responsive feel' and subtracting application and OS delays.

See the link I put in the email you are responding to for the list, 
repeated here:
     http://bobbriscoe.net/presents/2004ietf/202005L4S-brief.pdf
To be clear, I'm not advocating these. I'm merely justifying my 
assertion about applications that are not currently possible. 
Personally, I'm not excited by gadgetry and social distancing.

Of course, we can't predict what other new applications will be 
developed once this amount of tail latency is removed over the wide 
area. Also, please bear in mind that I cannot generally pass on 
information about product plans.

The original L4S teams from Alcatel-Lucent and Simula demonstrated a 
number of these applications at the Multimedia System 2016 conference: 
“Ultra-Low Delay for All: Live Experience, Live Analysis 
<https://riteproject.files.wordpress.com/2015/10/uld4all-demo_mmsys.pdf>“, 
Proc. ACM Multimedia Systems; Demo Session (May 2016).
Particularly cloud-rendered panoramic camera control and cloud-rendered 
VR goggles, were run at the conference over DCTCP and L4S with emulated 
wide-area round trip delay.

Over today's state of the art AQMs, the delay is too inconsistent to 
rely on doing any of the following real-time interactive applications 
over wide area distances, or rendered remotely (e.g. in the cloud), 
rather than just confined to a LAN or metropolitan area:
* cloud-rendered interactive control (like this: 
https://riteproject.eu/dctth/#1511dispatchwg )
* remote presence,
* interactive real-time remote control or supervisory monitoring of 
cameras, drones, machinery (oil rigs, power plants, machine shops, 
cranes, ), medical procedures, using (high res.) video to monitor what 
is being controlled.
* interactive light-field experiences

> 	Note, how for this question, I accept your claim of 1ms queueing delay even though testing indicates that this does not hold under all traffic patterns one is prone to encounter on a real internet…

[BB] The novelty isn't the ability to demonstrate a "concept 
application" in a marketing video; it's large numbers of real people 
actually being able to use it over the wide area for extended periods, 
without getting stressed, sick, or exhausted. And with just their 
regular Internet, rather than having to set up special network 
arrangements, or a special reserved session.

In the case of industrial or professional applications, quite often it's 
the ability for expertise to be deployed to a far-flung or inaccessible 
site instantly that would be the novel aspect, without experts or 
supervisors having to already sit on-call close by the site.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 24 May 2020 07:21:53 +0300

> On 24 May, 2020, at 2:53 am, Bob Briscoe <[email-masked]> wrote:
> 
> * cloud-rendered interactive control (like this: https://riteproject.eu/dctth/#1511dispatchwg )

The demo linked there is a comparison between DCTCP and a dumb FIFO - of course the latter is going to lose.  But plain ECN (or SCE) is already capable of getting queue delay below one frame of video, so this is not a compelling case for L4S.

> * remote presence,

See above.  But also note that for properly functioning VR, developers have had trouble keeping the input-to-display latency sufficiently low to avoid motion sickness without ever leaving the single machine that both input and display are physically attached to.  Doing that over any sort of network is going to be extra hard.

For this particular category of applications, the total latency of the network path will govern the experience, not so much the variability as long as the latter is reasonably small.  If you're talking with someone who's 160ms of light-time plus video codec latency away, an extra 5ms will not be noticed.  If they're just the other side of the campus, a total of 5ms will not be noticed.  Do you see the pattern?

> * interactive real-time remote control or supervisory monitoring of cameras, drones, machinery (oil rigs, power plants, machine shops, cranes, ), medical procedures, using (high res.) video to monitor what is being controlled.

Are you seriously suggesting running realtime, safety-critical processes over a best-effort Internet path?  Because if so, that's the dumbest thing I've heard this week, and given how much the current POTUS is in the news, that's saying something.

> * interactive light-field experiences

I have absolutely no clue what practical application you might be referring to in this category.  But I suspect it basically falls into the same set of requirements as the first two items above, just with a different set of input and output devices.

I think I'll leave Paul and Sebastian to comment further in this vein.

 - Jonathan Morton

---

From: Paul Vixie <[email-masked]>
Date: Sun, 24 May 2020 05:16:08 +0000

> I think I'll leave Paul and Sebastian to comment further in this vein.

i can't do better than what you said here. the smoke seems thick and the 
mirrors well positioned. there is no way to thread the needle in a way that 
avoids the best-efforts and physical limits, has trusted end to end signals 
beyond an enterprise or cloud or isp, and won't be bleached.

no packet is undroppable, no matter how marked: when the last queue is full, 
some packet won't reach its destination.

in "rise of the stupid network" (isenberg) and "elements of networking style" 
(padlipsky) were were told not to make the network smart and we were told why. 
i feel like howard the duck ("trapped in a world he never made") watching this 
discussion unfold.

my dispute stands unresolved, on the process matters i described. one chair 
has spoken no-hats on that thread, which clarified very little.

-- 
Paul

---

From: Gorry Fairhurst <[email-masked]>
Date: Sun, 24 May 2020 09:01:33 +0100

> I think I'll leave Paul and Sebastian to comment further in this vein.

If you decide to comment more, could you please finish that very quickly 
- the WG has decided to work on L4S.

There are topics that the WG needs to address, and I think this 
discussion is not helping the WG make progress.

Gorry

-- 
G. Fairhurst, School of Engineering

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 24 May 2020 10:10:52 +0200

Hi Bob,

Tl, dr: L4S offers some quantitative improvements over the status quo, but nothing qualitative novel. 

> On May 24, 2020, at 01:53, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> As explained in the email you are responding to, tail latency is important for real-time.

	[SM] This is not wrong, but the fact of the matter is, that the three services I already listed, google stadia, nvidia gforce now, shadow cloud gaming, all are real-time sensitive applications that are DEPLOYED und OPERATE in today's internet. So even the status quo is good enough for real-time coordination between actions and resulting control flow changes to be convincing enough for people to spend money on. That is a bg part of your potential market, that clearly is OKAY today indicating that improving tail latency is not an urgent issue for internet-scale real-time applications.

> The 10ms you report for FQ_CoDel is roughly its 90th percentile queue latency, which is an irrelevant metric for real-time (if the stream was being rendered after the 90%ile delay, 10% of the packets would arrive too late to be rendered). 

	[SM] This is not wrong, assuming your total latency is below one frame period and you are rendering at >= 100 Hz (for the 10ms limit). Soft real-time applications like games already deal with such delay variations though (to over-come both information transfer and CPU/GPU-cycle limitations), so improving your PDV slightly is not going to bring qualitative changes, no matter how hard we might wish.

> 
> In http://bobbriscoe.net/presents/2004ietf/202005L4S-brief.pdf (slide 4) I've used the 99.9th percentile delay for the rendering deadline, which reduces the fraction of packets that cannot be used to 0.1%.

	[SM] Your assumptions for that plot are a bit simplistic, but let's work with those. A) I claim that L4S is in reality not going to work reliably in the green circles (which you made from extrapolating a lab measurement, not from real-world measurements). B) CDN/data centers tend to be closer to the red radii not the green ones, so the internet already "solved" that issue for you, and e.g. 5G proponents advertize moving data center nodes even into the base stations, so I see no tendency currently to re-centralize data centers, do you?

> In the experiment reported in slide 3 you'll see the P99.9 latencies are 32ms for FQ_CoDel and 4.5ms for L4S.
> On slide 4 this is translated into reach in fibre, taking the generally accepted 50ms max delay for 'responsive feel' and subtracting application and OS delays.

	[SM] The point is that 500km radius is already plenty sufficient, unless you are trying to sell L4S exclusively to operators of big data centers, so that they can re-centralize their sites.

> 
> 
> See the link I put in the email you are responding to for the list, repeated here:
> To be clear, I'm not advocating these. I'm merely justifying my assertion about applications that are not currently possible. Personally, I'm not excited by gadgetry and social distancing. 

	[SM] Again, which of these a) make sense over the internet and b) do not work already today? As far as I can see the intersection of that set is rather empty.
Let's face it, L4S offers nothing qualitative novel, it is (assuming it works) an quantitative improvement, and hence might make a few things work a bit better, but the claim for novel applications really is wishful thinking.

> 
> Of course, we can't predict what other new applications will be developed once this amount of tail latency is removed over the wide area.

	[SM] Well, I can offer a prediction: no novel applications will grow out of this.

> Also, please bear in mind that I cannot generally pass on information about product plans.

	[SM] Fair enough, but a product plan, does not make a successful new application, nor does it demonstrate that the application absolutely requires L4S like network feedback.

> The original L4S teams from Alcatel-Lucent and Simula demonstrated a number of these applications at the Multimedia System 2016 conference: “Ultra-Low Delay for All: Live Experience, Live Analysis“, Proc. ACM Multimedia Systems; Demo Session (May 2016).
> Particularly cloud-rendered panoramic camera control and cloud-rendered VR goggles, were run at the conference over DCTCP and L4S with emulated wide-area round trip delay. 

	[SM] People use shadow cloud gaming with oculus rift headsets to do cloud-rendered VR already today... L4S (if it works) might make shadow's job a bit easier, but that is simply nothing novel. As noted before shadow is not alone in that respect.

> 
> Over today's state of the art AQMs, the delay is too inconsistent to rely on doing any of the following real-time interactive applications over wide area distances, or rendered remotely (e.g. in the cloud), rather than just confined to a LAN or metropolitan area:

	[SM] A claim that either hinges on your definition of "wide area distances" or is simply already refuted by reality. As expected, as all you are offering is a quantitative improvement in one metric, queueing latency, while the proposed applications are limited by total latency and total latency variation.

> * cloud-rendered interactive control (like this: https://riteproject.eu/dctth/#1511dispatchwg )

	[SM] Shadow, stadia, gforce now, do this right now without any AQM on the consumer side/return channel, so clearly something is wrong with your analysis?

> * remote presence,

	[SM] Video conferencing already works today; interestingly this tends to work better with a competent home network set-up with airtime fairness on the wifi side and a properly debloated access link (the wifi tends to be the bigger issue for many users though, L4S is not going to help there, is it)?

> * interactive real-time remote control or supervisory monitoring of cameras, drones, machinery (oil rigs,

	[SM] This is asinine, but I will bite, these off-shore oil-rigs most likely are connected via satellite internet, and geostationary satellites at that, the variable queueing delay is just going to be drop in the latency bucket (~270ms one-way propagation delay); and good luck doing anything interactive over such a delay.

> power plants, machine shops, cranes, ), medical procedures, using (high res.) video to monitor what is being controlled. 

	[SM] Whoever thinks about routing important medical procedures information required to make that procedure successful over the open internet, needs to be separated from his/her medical license. Anything that is not important for the process, like a comented upon video feed can easily tolerate delays.

> * interactive light-field experiences

	[SM] Just like "cloud-rendered panoramic camera control" that seems to be something playing to L4S perceived strength, but that has currently zero market share behind it. Sort of like an on-line version of the laser-disc.

> [BB] The novelty isn't the ability to demonstrate a "concept application" in a marketing video;

	[SM] But that is all that you offer ATM; I asked explicitly to get some real world killer apps for L4S and all I got was generalities.

> it's large numbers of real people actually being able to use it over the wide area for extended periods, without getting stressed, sick, or exhausted. And with just their regular Internet, rather than having to set up special network arrangements, or a special reserved session.

	[SM] As before, because you keep ignoring the elephant in the room, like Google Stadia, NVidia GeForce Now, and shadow cloud gaming. This ship has sailed without L4S, all you can offer is incremental improvements.

> 
> In the case of industrial or professional applications, quite often it's the ability for expertise to be deployed to a far-flung or inaccessible site instantly that would be the novel aspect, without experts or supervisors having to already sit on-call close by the site.

	[SM] Bob, either the behavior can be conserved in a video, which can be easily and reliably transported over the best-effort internet to the remote expert to scrutinize, or we are back to the critically stupid and dangerous idea of running important real-time control loops with real life consequences over the best-effort internet (games are fine, the failure mode, you loose a round, is benign, as compared you blowing out the well-head of an oil-rig in the gulf of mexico, I assume you agree).

As before, my assessment of L4S boils down to "too little, too late". Too little improvement over the status quo (at too high side-effects/costs) almost a decade later than originally envisioned.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 24 May 2020 14:37:22 +0200

Hi Gorry,

>- the WG has decided to work on L4S.

[SM] On the danger of stating the obvious, that decision to work on L4S makes it quite important to highlight and discuss all the pain-points of the current L4S design, implementation and drafts. Unless I misunderstand what "work on L4S" is supposed to entail...

>discussion is not helping the WG make progress.

[SM] For better or worse, this WG will be held partially accountable, for L4S meeting it's claims, so I consider discussion of those claims, with the eventual goal of finally de-hyping the draft text, pretty on-topic and relevant for the mailing list.

BUT, since that is so obvious that I assume I must be misunderstanding your point, so please show me where/how.

Best Regards
        Sebastian

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 24 May 2020 16:54:55 +0100

Sebastian,

> My point is, please refrain from completely unhelpful complexity comparisons by virtue of pseudeocode, especially if not both qdiscs to be compared actually exist in pseudocode in the first place..…

[BB] There are a number of ways one can/should compare complexity. I was 
not suggesting anyone should count lines of code. I was suggesting Paul 
should /look/ at the pseudocode, or the code linked from it. You're the 
one that has turned this into a line counting exercise.

Code complexity (roughly related to lines of code) is important for 
minimizing bugs, and perhaps for minimizing memory footprint on 
cost-reduced devices like home gateways. However, for large-scale 
network equipment, ops per packet are particularly important, and 
minimizing run-time state is important for most hardware implementations.

> 	[SM] True, until it becomes obvious that without the optional queue protection mechanism dualq severely lacks behind fq_codel in real-world usefulness.

[BB] I think that statement needs to be tagged with "in the opinion of 
Sebastian Moeller who has not justified this statement".

> Once you add the queue protection to the requirements, your are looking at L4 headers again.

[BB] Indeed. Finding L4 headers was the most complex part of the whole 
Low Latency DOCSIS spec, because there are so many possible unusual 
cases to handle. The Linux coupled dualQ implementation handles overload 
in a much simpler way.

We shall see which turns out to be sufficient. I can understand that 
many operators will be cautious with a new approach, which is why I 
agreed to design per-flow QProt for Low Latency DOCSIS, even though I do 
not believe it will eventually prove necessary. That, to me, is one of 
the open questions that only an Internet-wide experiment can answer, so 
please refrain from giving your opinion again - we have all heard it 
many times already.

> That said, there is absolutely no reason, why fq_codel could not look only at the IP header for its hashing needs, if you really want that (and interestingly cake offers such flow-definition modes).
>
> 	[SM] +1; it just turned out in the real world that the performance of a single queue AQM simply does not deliver what the end-users actually want from an AQM so that nobody went for a single queue CODEL. I see absolutely no reason, why this should change now with L4S.

[BB] Because L4S replaces Reno-friendly congestion control, which was 
the root cause of the problem that made per-flow isolation useful.

> Now, I only talk to end-users, ISPs might see this differently (but ultimately will need to convince their users that L4S is actually an improvement).
>
>

[BB] The coupling uses a state variable that is already maintained for 
the non-LL AQM. But adding the LL AQM requires no additional state.

> 	[SM] Interestingly, end-users interested in low-latency and sharing their access links between, say gaming and streaming, opted for fq_codel and cake. Even though this requires post-bottleneck ing…

[BB] You can believe that story if you want. I think you will find that 
most end-users who use FQ_CoDel today were not aware they had a choice 
of anything else. I repeatedly had to pull up people who were 
relentlessly using the word FQ_CoDel in sentences where the word AQM was 
appropriate. I gave up very early on. It continued for years after that.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 24 May 2020 20:24:46 +0200

Dear list, dear Bob,

Tl; dr: In detail debate without much general impact.

> Begin forwarded message:
> 
> From: Bob Briscoe <[email-masked]>
> Subject: Re: [tsvwg] Gorry Fairhurst Individual thoughts on choosing whether/how to advance ECN work.
> Date: May 24, 2020 at 17:54:55 GMT+2
> To: Sebastian Moeller <[email-masked]>
> Cc: [email-masked]
> 
> Sebastian,
> 
> [BB] There are a number of ways one can/should compare complexity. I was not suggesting anyone should count lines of code. I was suggesting Paul should /look/ at the pseudocode, or the code linked from it. You're the one that has turned this into a line counting exercise.

	[SM] What else did you mean when you asked Paul to "compare this pseudocode yourself" after first mentioning both DualQ and fq_codel? I guess I misunderstood what you intended to say then.

> 
> Code complexity (roughly related to lines of code) is important for minimizing bugs, and perhaps for minimizing memory footprint on cost-reduced devices like home gateways. However, for large-scale network equipment, ops per packet are particularly important, and minimizing run-time state is important for most hardware implementations.

	[SM] That is why I wondered, you brought up code complexity in the first place.

> [BB] I think that statement needs to be tagged with "in the opinion of Sebastian Moeller who has not justified this statement".

	[SM] Well, it goes without saying that all of us can not speak "ex cathedra" here (with exception of the chairs with their official hats on) and hence voice our own opinions, me not less than you. Any specific tagging seems hence redundant, no? That said, I am supporting low-latency interested users as a small part of the SQM effort for several years now, so I am not a really a novice to the whole problem space.

> [BB] Indeed. Finding L4 headers was the most complex part of the whole Low Latency DOCSIS spec, because there are so many possible unusual cases to handle. 

	[SM] Except that at the ISP end-customer edge the only relevant cases are IPv4 and IPv6, but the linux kernel has quite powerful flow dissectors to help you, see e.g. https://elixir.bootlin.com/linux/v4.18.6/source/net/core/flow_dissector.c

> The Linux coupled dualQ implementation handles overload in a much simpler way.

	[SM] Unfortunately, as Koen's recent example demonstrated with rather limited real-world usefulness; in his example three unrelenting fixed rate UDP flows each got ~1/3 of the bottleneck capacity, while all concurrent TCP flows were basically starved. So I would say, the dualQ implementation handles overload in a too simple way, as so often with L4S, "too little, too late". 

> 
> We shall see which turns out to be sufficient. I can understand that many operators will be cautious with a new approach, which is why I agreed to design per-flow QProt for Low Latency DOCSIS, even though I do not believe it will eventually prove necessary. 

	[SM] And again you demonstrate that L4S safety considerations are mostly driven by hope and believe... I would have preferred a design that actually tried to test and robustify the performance under adversarial traffic patterns instead.

> That, to me, is one of the open questions that only an Internet-wide experiment can answer, 

	[SM] This is simply untrue, sure the real internet will generate such adversarial traffic patterns, better than one can do in the lab, but L4S did not even try.
> [BB] Because L4S replaces Reno-friendly congestion control, which was the root cause of the problem that made per-flow isolation useful.

	[SM] I object to this classification, that the only or even most value of per-flow-isolation is in isolating the normal TCP-sawtooths from each other. IMHO the value of FQ is that equatable sharing between all active flows is a) simple to reason about, b) avoids starvation and c) might not be the optimal solution for each traffic mix, but is exceedingly unlikely to be a pessimal solution either. Given that most network nodes lack the required information to actually optimize in any meaningful way, I will take alnost never pessimal any old day over L4S's anything goes. If asked why, I just point to Koen's example referenced above.

> [BB] The coupling uses a state variable that is already maintained for the non-LL AQM. But adding the LL AQM requires no additional state.

	[SM] That is legerdemain, you need that variable for both legs, so exclusively accounting it to the non-LL side is disingenuous, with equal or more basis in fact you could account it half to each side. But whatever.

> [BB] You can believe that story if you want. 

	[SM] This is not about what I believe or do not believe, it is that from supporting endusers in debloating their access links with SQM I had some insight in what behavior end-users desire, and currently the best all around solution seems to first share bandwidth equitably between all concurrently active end-user network internal IP addresses and inside of these aggregates share equitably between flows. That is a solution that solves most of the issues users report. Yes, this does not solve 100% of issues for 100% of users, but it is overall a pretty solid and conceptually simple solution, that gives predictability and relative robust and reliable performance, even under bi-directionally saturating loads.

> I think you will find that most end-users who use FQ_CoDel today were not aware they had a choice of anything else. 

	[SM] I have no opinion on the numbers, but the end-users of the SQM package generally gain some insight s in their options. And even on DOCSIS3.1 links often opt for fq_codel/cake on the uplink over the supposed default DOCSIS PIE instance...

> I repeatedly had to pull up people who were relentlessly using the word FQ_CoDel in sentences where the word AQM was appropriate. 

	[SM] These people might exist, I am not one of them so why bring this up?

> I gave up very early on. It continued for years after that.

	[SM] I do not remember you showing up in the relevant OpenWrt user forums at all, which handle did you use?

Best Regards
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 May 2020 08:46:55 +0100

Hello,

> BUT, since that is so obvious that I assume I must be misunderstanding your point, so please show me where/how.

I think it would be more normal to discuss or provide comments on a specific section of a document. If there are claims there, then maybe these do need to be looked at - it's not uncommon that the text around performance/merits in a spec need to be reviewed before a document completes WGLC.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 25 May 2020 10:13:11 +0200

Hi Gorry,

> On May 25, 2020, at 09:46, Gorry Fairhurst <[email-masked]> wrote:
> 
> Hello,
> 
> I think it would be more normal to discuss or provide comments on a specific section of a document.

	[SM] Well, I have tried and cited critical sections of the RFC where implemented reality and claim differ in the past only to receive basically silence on the mailing list. 

> If there are claims there, then maybe these do need to be looked at -

	[SM] This reads as if you would doubt that there are claims in the RFC drafts, is that really your position? I appreciate your attempts at keeping an objective perspective here, but that seems a bit too much.

> it's not uncommon that the text around performance/merits in a spec need to be reviewed before a document completes WGLC.

	[SM] Fair enough. Now, I can not help wondering, why we should only consider doing such corrections and toning-down after basically adopting the drafts, given that these overly optimistic/unrealistic performance predictions and safety assessments* might well have influences the choices voiced in the consensus process.

Best Regards
	Sebastian

*) Assessments is a strong word, "hopes" is a better description to L4S approach to safety/security, indicating a lack of actual testing under adversarial traffic patterns:

https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-11#page-21:
"Where the interests of users or flows might conflict, it could be
   necessary to police traffic to isolate any harm to the performance of
   individual flows.  However it is hard to avoid unintended side-
   effects with policing, and in a trusted environment policing is not
   necessary."

https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch-06:
"Like the Classic service, the L4S service relies on self-constraint -
   limiting rate in response to congestion.  In addition, the L4S
   service requires self-constraint in terms of limiting latency
   (burstiness).  It is hoped that self-interest and standardization of
   dynamic behaviour (especially flow start-up) will be sufficient to
   prevent transports from sending excessive bursts of L4S traffic,
   given the application's own latency will suffer most from such
   behaviour.

   Whether burst policing becomes necessary remains to be seen.  Without
   it, there will be potential for attacks on the low latency of the L4S
   service.  However it may only be necessary to apply such policing
   reactively, e.g. punitively targeted at any deployments of new bursty
   malware."

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 May 2020 09:39:05 +0100

So, this was your comment in Dual-Q:

>     effects with policing, and in a trusted environment policing is not

The general security considerations across section 4 seems reasonable to me.

Did you have issues with that text wrt security considerations?

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 25 May 2020 10:54:54 +0200

Hi Gorry,

> On May 25, 2020, at 10:39, Gorry Fairhurst <[email-masked]> wrote:
> 

	[SM] Yes, certainly I have issues with that. The use of the subjunctive "could" indicates a purely hypothetical concern, or rather that this issue was not actually tackled empirically, yet. Now, Bob argues* that experimental roll-out is required to test that hypothesis but at the same time fights nail and tooth to even add a MUST implement for his intended counter-measure against such a failure mode, the queue protection. As is this is safety-by-wishful-thinking, and that IMHO is not a solid basis for any IETF condoned experiment or standard. 
	Initially the L4S argument was, abusing L4S signaling is self-defeating and hence will not happen, due to lack of incentives to do so. As far as I understand that position was retracted (or at least not publicly repeated on the list) based on the discussion, but it has not been replaced with anything else explaining why safety mechanisms can and/or should stay optional, besides hopes and wishes.

Best Regards
	Sebastian

*) Bob: "I can understand that many operators will be cautious with a new approach, which is why I agreed to design per-flow QProt for Low Latency DOCSIS, even though I do not believe it will eventually prove necessary. That, to me, is one of the open questions that only an Internet-wide experiment can answer, so please refrain from giving your opinion again - we have all heard it many times already."

Note again, Bob's position is based on "believe" instead of harf cold numbers.

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 May 2020 10:02:11 +0100

See below.

On 25/05/2020 09:54, Sebastian Moeller wrote:
> 	[SM] Yes, certainly I have issues with that. The use of the subjunctive "could" indicates a purely hypothetical concern, or rather that this issue was not actually tackled empirically, yet.
I don't read "could" that way. I read this as this could-be-needed, 
interprettting this as a non-RFC2119 version of "MAY".
> Now, Bob argues* that experimental roll-out is required to test that hypothesis but at the same time fights nail and tooth to even add a MUST implement for his intended counter-measure against such a failure mode, the queue protection. As is this is safety-by-wishful-thinking, and that IMHO is not a solid basis for any IETF condoned experiment or standard.

I'll let the document editor decide if they wish to respond about 
whether they think more is needed in this section of the DualQ-draft.

> Note again, Bob's position is based on "believe" instead of hard cold numbers.
>
>> Gorry
Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 25 May 2020 12:00:30 +0200

Hi Gory,

more below in-line.

> On May 25, 2020, at 11:02, Gorry Fairhurst <[email-masked]> wrote:
> 
> See below.
> 
> I don't read "could" that way. I read this as this could-be-needed, interprettting this as a non-RFC2119 version of "MAY".

	[SM] Which boils down to the same more or less, IMHO. Point being instead of attempts at measuring realistic traffic scenarios (like e.g. flows using different initial window sizes as some CDN's are known to use, see https://www.comsys.rwth-aachen.de/fileadmin/papers/2019/2019-rueth-iwtnsm.pdf), all that is offered is a hope. I certainly also hope that this will not become an issue, but IMHO that does not take tho onus from the designers/developers of a solution to test under realistic adversarial traffic conditions, before simply relaying on hope. Why is that view even contentious? 
	Is everybody else convinced that L4S offers a sufficiently safe design and was tested with the to be expected level of adversarial traffic patterns to confirm the design's safety in the real-world?

Best Regards
	Sebastian

> I'll let the document editor decide if they wish to respond about whether they think more is needed in this section of the DualQ-draft.
> 
>> 
>> Best Regards
>> 	Sebastian
>> 
>>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 25 May 2020 13:10:13 +0300

> On 25 May, 2020, at 1:00 pm, Sebastian Moeller <[email-masked]> wrote:
> 

I am convinced of the opposite of this.  Hence the many technical concerns raised over the past few weeks.

Some results that Pete Heist posted recently demonstrate, if proof were needed, that traffic unresponsive to ECN signals can easily crowd out responsive traffic flows in DualQ, whereas straightforward implementations of flow-aware queue protection (ie. the BLUE component of COBALT used within CodelAF) are able to mitigate that sort of behaviour, even without an explicit FQ mechanism.  Earlier results also show easily triggerable bad results without requiring any intentional misbehaviour.

That, I think, lies at the root of Sebastian's line of argument.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 May 2020 11:22:47 +0100

See below, I'll try to explain how I hope this can focus on the documents.

On 25/05/2020 11:00, Sebastian Moeller wrote:
> 	[SM] Which boils down to the same more or less, IMHO.
Not really, this document is proposed as an AQM Spec. To me, traffic 
management, scheduling, etc belong in a draft about architecture or 
operations. So, a conversation needs to focus on the appropriate 
document and a specific section.
> Point being instead of attempts at measuring realistic traffic scenarios (like e.g. flows using different initial window sizes as some CDN's are known to use, see https://www.comsys.rwth-aachen.de/fileadmin/papers/2019/2019-rueth-iwtnsm.pdf), all that is offered is a hope. I certainly also hope that this will not become an issue, but IMHO that does not take tho onus from the designers/developers of a solution to test under realistic adversarial traffic conditions, before simply relaying on hope. Why is that view even contentious?

I do like data - that is good.

But, my email is urghing you to separate the pieces.

If for instance, one was to consider traffic unresponsive to ECN 
signals, then this is a particular case of non-conformant traffic - we 
all see non-conformant traffic in networks for various reasons and need 
to have tools and methods to deal with that - what a network decides to 
actually use will depend on many things. For example, it wouldn't be 
wise to suggest a scheduling-based approach eliminates all the problems 
- in some parts of the network this has always been the case, in other 
places it is not a useful technique. similarly circuit-breaker methods 
have their place, as likely does a policer.

Anyway, what I am suggesting is decide on a topic, relate this to a 
particular topic and a particular draft.

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 May 2020 11:43:51 +0100

> That, I think, lies at the root of Sebastian's line of argument.

Jonathan,

To me these seem like examples of reasonable operational tools when a network wants to provide flow-level management. Somewhere the documentation produced by the IETF should indeed describe this type of mechanism and how they can be deployed to manage traffic unresponsive to ECN signals. And I'd expect there are likely also other ways also to mitigate the risks from unresponsive traffic, and it could be posisble to offer a range of methods and decsribe their merits?

Gorry

---

From: Paul Vixie <[email-masked]>
Date: Mon, 25 May 2020 15:49:20 +0000

> Hi Gory,

i'm not gory, but you asked a (thetorical) question of "everybody".

> ... Is everybody else convinced that L4S offers a sufficiently
> safe design and was tested with the to be expected level of adversarial
> traffic patterns to confirm the design's safety in the real-world?

so, no, i am not convinced, but that doesn't matter. what matters is that if 
there is an input signal marked on a flow, datagram, packet, or segment which 
would cause my routing or switching equipment to behave differently, such as 
choosing one of two "dual queues", then i will likely never enable it, and if 
i do enable it i will bleach that mark out of inbound traffic from the WAN.

networks i operate, that would operate under that constraint, have included 
home, office, enterprise, and backbone. due to principle of least privilege, 
and not because of any known or suspected flaws in L4S per se. this is not 
different from filtering some kinds of ICMP and rate limiting other kinds. all 
policy is local, and possession is 90% of the law when it comes to ownership.

o.t.:

if my supply chain offers me something i can use, i may use it. but that use 
may not be the L4S (or SCE) system, i may favour some custom logic which makes 
novel use of the supply chain's new capabilities. so while all L4S (or SCE) 
may depend on some new network input signal, not all uses of that signal will 
be by L4S (or SCE). thus my focus on possible off-label uses of new signalling 
and my estimates of maximum risks and maximum benefits and to whom and with 
what kind of symmetry.

-- 
Paul

---

From: "C. M. Heard" <[email-masked]>
Date: Mon, 25 May 2020 09:21:17 -0700

On Mon, May 25, 2020 at 8:50 AM Paul Vixie wrote:
> so, no, i am not convinced, but that doesn't matter. what matters is that
> if there is an input signal marked on a flow, datagram, packet, or segment
> which would cause my routing or switching equipment to behave differently,
> such as choosing one of two "dual queues", then i will likely never enable
> it, and if i do enable it i will bleach that mark out of inbound traffic
> from the WAN.

Because it may inform the conversation, is your current policy to disable ECN
marking and/or to bleach ECN bits out of inbound traffic from the WAN? Per
RFC 3168, those bits are intended to affect drop vs mark behavior of routers.

Mike Heard

---

From: Paul Vixie <[email-masked]>
Date: Mon, 25 May 2020 16:33:10 +0000

On Monday, 25 May 2020 16:21:17 UTC C. M. Heard wrote:
> Because it may inform the conversation, is your current policy to disable
> ECN marking and/or to bleach ECN bits out of inbound traffic from the WAN?
> Per RFC 3168, those bits are intended to affect drop vs mark behavior of
> routers.

no. i'm following choice other, which is, don't enable those features.

-- 
Paul

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 25 May 2020 09:49:08 -0700 (PDT)

> RFC 3168, those bits are intended to affect drop vs mark behavior of routers.

Unsolicited, but useful data.

Netflix says they shall not turn on ECN in there services because they do
not trust the network, which is another case of the issue stated above.

Yandex says they shall not turn on ECN due to an unfairness issue, which
I am now starting to fully understand.  ECN can be gamed, plain and simple,
so ECN without a FQ or some other form of protection can lead to network
abuse.  It is rather simple to create a transport that claims to support
RFC3168 marking, and then have it simply ignore ECE.  Pete Heist, I believe,
has posted data on this, and no one has even responded to it.

So in answer to your question Mike, for me, I would feel at liberty to bleach
ECN entering my AS should I find it being gamed, and might just do so
out of safety until I can be sure that proper protections are in place.

IMHO, this WG should take a serious hard look at some of these issues
and possibilities, nothing in L4S removes those status quo's, and again
IMHO, this dooms it to the same fate as ECN in some rather large parts
of the internet.  SCE suffers from this less so in that it is almost
always coupled with either a FQ or AFQ AQM which does infact at least
mitigate the above issues.

It appears to me that one can not safely deply any ECN system without
some form of queue protection.   And yes, including RFC3168.

-- 
Rod Grimes                                                 [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 25 May 2020 18:08:55 +0100

> no. i'm following choice other, which is, don't enable those features.
Thanks. That's your choice, and I have seen networks that have made 
similar choices for various reasons, and ones that have not made those 
choices.

Gorry

---

From: Steven Blake <[email-masked]>
Date: Mon, 25 May 2020 16:14:58 -0400

On Mon, 2020-05-25 at 09:49 -0700, Rodney W. Grimes wrote:
> Netflix says they shall not turn on ECN in there services because
> they do not trust the network, which is another case of the issue
> stated above.

Source?

What is their concern, that downstream networks are setting CE more
frequently than necessary to lower Netflix's stream's thruput? Because
those same operators could just as easily drop packets.

Regards,

// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 25 May 2020 23:22:55 +0300

> On 25 May, 2020, at 11:14 pm, Steven Blake <[email-masked]> wrote:
> 

Which, in effect, is a thing that has actually happened to Netflix traffic in particular.  Certain cable companies didn't like them competing in their core market of video service, so they refused to provide the network capacity they needed at certain peering points.  So if Netflix says they "don't trust the network", I believe them.

I'm not sure how that translates into a fear of ECN from their perspective, though.

 - Jonathan Morton

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Mon, 25 May 2020 13:38:39 -0700 (PDT)

> > they do not trust the network, which is another case of the issue

Grenville Armitage during the Stanford Buffer Sizing Workshop.
Direct question from me to him while he was on stage.  It should
be in the video if there was one made.

> those same operators could just as easily drop packets.

Yes, they are concern that a networks could use CE to reduce Netflix's
stream goodput, and that it would "be easier to detect a drop" when
I pursued the same line of questioning you have here.

---

From: Paul Vixie <[email-masked]>
Date: Tue, 26 May 2020 17:52:43 +0000

> similar choices for various reasons, and ones that have not made those

i've called this a datacenter/cloud signal pattern and some of you have 
wondered why. one reason is trust. see also rod grimes' reply up thread 
<[email-masked]> mentioning yandex and netflix. 
but none of these decisions is simple; we have to look at potential added 
risks as well as potential added benefits. if the latter seems orders of 
magnitude greater than the former, then we may decide to accept the risks.

in dualq i'd perhaps see microsecond delivery on the fast path and millisecond 
delivery on the slow path. that difference could be relevant for traffic which 
is both sourced and sunk nearby or inside my network. however, it will never 
be relevant for arriving WAN traffic, whose speed-of-light constraints while 
reaching me will far exceed the difference between my slow and fast queues.

so even if i had no fears of having the configuration of my switches or 
routers set, however briefly (on a packet by packet basis) by a non-contracted 
unidentified third party, i would also see no benefit to allowing congestion 
signaling as input to my network other than from my own nearby/internal hosts. 
that's also true of output from the network to my endpoints, since congestion 
notification signals are also game-worthy. but less so, and i can imagine at 
least looking at the SCE running fraction in my transport, and considering it 
as a possible reason to reduce window size. that will never happen for the L4S 
pattern when it comes from the WAN, for reasons as given.

-- 
Paul

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 27 May 2020 10:58:39 +0200

Hi Gorry,

> On May 25, 2020, at 12:22, Gorry Fairhurst <[email-masked]> wrote:
> 

	[SM] Sure, I just want to note that I see the lack of properly assessing and addressing the security questions, not an issue about the wording of a few specific paragraphs, but rather a systematic decision that permeates the whole dratfs. Hence fixing these will not result in just a few changes to the security considerations section IMHO. Sure we can discuss these specifically, but I do not believe that we will be able to remedy that whole rather cavalier approach to security there.

> Not really, this document is proposed as an AQM Spec. To me, traffic management, scheduling, etc belong in a draft about architecture or operations.

	[SM] I disagree, I cited two closely related sections from both the dual queue coupled AQM and the L4S arch drafts, that both display the same cavalier approach to safety ("could be necessary", "it is hoped..."). I consider it essential to align all L4S drafts with reality.

> So, a conversation needs to focus on the appropriate document and a specific section.

	[SM] Let's focus on the big picture first, please. I do not intend to go on a quixotic quest against wind-mills, if the consensus in the WG is that none of these changes I argue for are necessary in the first place. (I will keep speaking my mind, but I will not embark of detailed text exegesis if I perceive this to be a fools errand).

> I do like data - that is good.

	[SM] That leaves me puzzled. Given the absolute lack of testing under realistic conditions that will be encountered at the ISP customer edge (including, bi-directionally saturating loads, asymmetric links, multi-hop paths across countries/oceans (where multiple nodes will add to latency and jitter), ... ), I fail to understand how we actually continue with text details, while the proof that the core promises actually are deliverable in the real-world is still missing? Rearranging the deck chairs of a certain ship of the White Star Line comes to mind.

> But, my email is urghing you to separate the pieces.

	[SM] Divide et impera, eh? ;)

> 
> If for instance, one was to consider traffic unresponsive to ECN signals, then this is a particular case of non-conformant traffic - we all see non-conformant traffic in networks for various reasons and need to have tools and methods to deal with that - what a network decides to actually use will depend on many things.

	[SM] We need to address the fact that L4S will make this issue more urgent, until now ECN is mostly an "also ran" so playing games with ECN will have little ROI, once the DOCSIS-edge is switched to L4S that value equation changes massively (and to their credit CableLabs created and mandated? the queue protection system to avoid this easy to predict pitfall).

> For example, it wouldn't be wise to suggest a scheduling-based approach eliminates all the problems

	[SM] +1; but in reality eleminating "all problems" typically is not feasible, but concluding from a perfect solution is not possible/desirable, that no solution is just as good, seems a bit counter-intuitive.

> - in some parts of the network this has always been the case, in other places it is not a useful technique. similarly circuit-breaker methods have their place, as likely does a policer.

	[SM] Gorry, please help understand, how a circuit breaker is going to help against the new class of attacks made possible by the L4S core design: non-paced bursts of ECT(1)-marked packets, that over a multi-dozends of millisecond window average out to below capacity rate, yet repeatedly drive the dualQ AQM into overload protection? This completely takes L4S, low-latency and low-loss promises off the table, yet is really hard to detect from just monitoring the incoming and outgoing aggregate traffic levels.
	Now, it s well possible that I misunderstand what a circuit breaker is and how sensitive it can be while still staying useful, but I can not help to think that the interactions between the promosed components have not yet been carefully thought through, let alone tested.

> 
> Anyway, what I am suggesting is decide on a topic, relate this to a particular topic and a particular draft.

	[SM] QUESTION: We have been promised a less enthusiastic worded version of these drafts, are these forth-coming? I would like to avoid spending time with a soon to be replaced version. (Not that I expect that the result of my text exegesis will actually flow back into the drafts, Bob has made it abundantly clear, that he emphatically does not share my risk assessments).

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 27 May 2020 11:04:09 +0200

Hi Paul,

thanks for your insights and rationale. 
I do share the doubts about how well L4S is going to be able to fulfill its promises with growing path length. 
	But, I am sure the L4S team probably already will have made measurement over the open internet and hence should be able to show how TCP Prague observed path RTT and jitter behave in dependence on path length in hops and how that correlates to the amount of ASs traversed. In the end it is the RTT and jitter observed by each flow that counts.

Best Regards
	Sebastian

> On May 25, 2020, at 17:49, Paul Vixie <[email-masked]> wrote:
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 27 May 2020 15:41:26 +0100

I'm going to leave others to pick-up on just a few details at the end.

On 27/05/2020 09:58, Sebastian Moeller wrote:
>> But, my email is urging you to separate the pieces.
> 	[SM] Gorry, please help understand, how a circuit breaker is going to help against the new class of attacks made possible by the L4S core design: non-paced bursts of ECT(1)-marked packets, that over a multi-dozens of millisecond window average out to below capacity rate, yet repeatedly drive the dualQ AQM into overload protection? This completely takes L4S, low-latency and low-loss promises off the table, yet is really hard to detect from just monitoring the incoming and outgoing aggregate traffic levels.

-- 

I would generally encourage people to separate security considerations 
from congestion control/traffic sharing issues.

Reducing claims about performance/features in drafts is always welcome. 
For the WGLC, we need to finally balance protocol design goals, against 
claims of what is offered, and doing this before WGLC is generally helpful.

The WG Chairs and ADs have been discussing what topics should best be 
included in which documents. My own suggestion would be to allow a few 
weeks for the document editors to digest this and then propose what 
needs to be done in the next steps to take this forward.

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 27 May 2020 19:47:08 +0300

> On 27 May, 2020, at 5:41 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
> I would generally encourage people to separate security considerations from congestion control/traffic sharing issues.
> 
> Reducing claims about performance/features in drafts is always welcome. For the WGLC, we need to finally balance protocol design goals, against claims of what is offered, and doing this before WGLC is generally helpful.
> 
> The WG Chairs and ADs have been discussing what topics should best be included in which documents. My own suggestion would be to allow a few weeks for the document editors to digest this and then propose what needs to be done in the next steps to take this forward.

Along these lines, these are the things I would expect to see addressed and considerably improved before a WGLC for L4S is even contemplated:

1: Interaction of minimally compliant L4S transport flows with existing network infrastructure and traffic.  This includes not only the incidence of single AQMs, but the relatively long paths and high jitter that exist in practice.  TCP Prague does not perform acceptably under these conditions, and neither does the proof-of-conceot adaptation of SCReAM to L4S-style signalling.

2: Performance of a minimally compliant L4S-aware middlebox under mixed traffic conditions, such that throughput of each class of traffic is "fair", under some widely accepted definition of "fair".  DualQ does not meet that standard.

3: Security and other operational considerations arising from the L4S specification, including the likely presence of bad actors.  These bad actors may ignore specifications entirely, or interpret specifications loosely, in an attempt to either gain an unfair advantage for their own traffic, or to disrupt service for others.  At the very least it is desirable to not worsen that situation; improving it is preferable if possible.

4: A sober and preferably independent appraisal of the actual performance obtained under varied conditions, for example according to RFC-5033.  Quoting from its introduction:

>    This document does not give hard-and-fast requirements for an
>    appropriate congestion control scheme.  Rather, the document provides
>    a set of criteria that should be considered and weighed by the IETF
>    in the context of each proposal.  The high-order criteria for any new
>    proposal is that a serious scientific study of the pros and cons of
>    the proposal needs to have been done such that the IETF has a well-
>    rounded set of information to consider.

The data thus far presented by the L4S team does not meet this standard, as it fails to cover scenarios that are less than ideal conditions for L4S, as demonstrated by the additional data provided by Pete Heist et al.

Personally, I am skeptical that L4S can meet these criteria without major architectural changes.  The most obvious of these would include a different use of the ECN field for congestion signalling (ie. not redefining CE, even conditionally).  The next most obvious would be to admit that L4S is not in fact suitable for widespread deployment on the public Internet, and instead confine it to specific networks that can be prepared for it; this would also permit using a DSCP as the classifier instead of ECT(1), freeing up the latter for alternative uses.

The onus is now on the L4S team to prove me wrong.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 27 May 2020 22:25:14 +0200

Hi Gorry,

please see below.

> On May 27, 2020, at 16:41, Gorry Fairhurst <[email-masked]> wrote:
> 
> Reducing claims about performance/features in drafts is always welcome. For the WGLC, we need to finally balance protocol design goals,

	[SM] Puzzled. L4S seems to offers no new protocol as part of the suite of RFC drafts. It does come with a few new requirements for protocols that want to get maximum ROI out of the L4S promises. Given that TCP Prague is not an RFC draft, what protocol design goals should we be concerned about here?

> against claims of what is offered,

	[SM] I am again puzzled, that very few members of this WG seem to have much issues with the L4S drafts promising the sky, while skimping on supplying robust evidence to support those claims.

> and doing this before WGLC is generally helpful.

	[SM] I would go further, it would have been helpful even before asking the ECT(1) as input or output question. Now, we are in the  weird situation, that L4S has basically gotten the "go" signal, with IMHO very little chance of not passing WGLC without any substantial changes to its core design. Verbiage, sure that might change a bit, but the core is by now pretty much accepted as set in stone (without sufficient evidence of it achieving its goals promises robustly and reliably).

> The WG Chairs and ADs have been discussing what topics should best be included in which documents. My own suggestion would be to allow a few weeks for the document editors to digest this and then p…

	[SM] Fair enough, IMHO there is no need to hurry any of this, or better no need to proceed at all before the requested robustness and reliability tests under to-be-expected conditions are performed, presented and evaluated, but I assume that that ship has sailed, and the process now will be based o the information and test results at hand?

Best Regards
	Sebastian

---

From: Tom Henderson <[email-masked]>
Date: Wed, 27 May 2020 16:52:28 -0700

On 5/27/20 7:41 AM, Gorry Fairhurst wrote:
(snip)

> needs to be done in the next steps to take this forward.

Gorry,
As part of this process, will the charter be updated to include working 
on low-latency service using ECT(1) as an input, or the L4S experiment 
specifically?

Also, regarding guidance to operators who do not participate in 
experiments, will the WG work on a recommendation for default router 
handling of ECT(1), outside of the L4S drafts?

- Tom

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 28 May 2020 04:30:20 +0100

See below.

On 28/05/2020 00:52, Tom Henderson wrote:
>> included in which documents. My own suggestion would be to allow a 
>> few weeks for the document editors to digest this and then propose 
>> what needs to be done in the next steps to take this forward.
>>
>
> Gorry,
> As part of this process, will the charter be updated to include 
> working on low-latency service using ECT(1) as an input, or the L4S 
> experiment specifically?
>
Our current charter has milestones relating to ECN propagation in 
tunnels and to development of L4S, these will be developed by the 
working group.
> handling of ECT(1), outside of the L4S drafts?

I'd hope that the WG will focus on developing the L4S specs - at least 
for the next series of meeting cycles. That work will include guidance 
on how ECN codepoints should be used and how equipment should be configured.

Gorry