Subject: [tsvwg] Pete Heist's rationale for ECT(1) as an output

From: Pete Heist <[email-masked]>
Date: Wed, 13 May 2020 13:57:24 +0200

My reasons for selecting ECT(1) as an output signal are primarily a two part
safety concern for the currently proposed input usage.

First, when we say "ECT(1) as an input", we need to clarify that the
present intention is to use the input to 1) change the meaning of the CE
codepoint and 2) select alternate treatment in the network.

*** Regarding #1 (changing the meaning of CE):

When two flows interpret the meaning of CE differently, it can lead to a
significant throughput imbalance (subject to our running definition of
starvation as "you know it when you see it"). There have been several arguments
made in opposition to this concern:

* Most of the AQMs deployed use FQ.

That may be true, but we shouldn't rely on FQ for a safety guarantee. None of
the FQ implementations in common use have collision resolution for their flow
hash tables, so if you choose any two incompatible flows, the chance of a
safety problem is one out of the number of hash buckets available, or 1024 in
the default case of fq_codel. It's one thing to rely on a hash table for flow
fairness, where a collision puts *compatible* flows in the same queue, and
another to rely on it for safety, where a collision puts *incompatible* flows
in the same queue. I will note that I recognize this as a challange for both
codepoint usage proposals, however, with the ECT(1) as an output proposal, it
only needs to be mitigated in future AQMs supporting the new signaling. With
the ECT(1) as an input proposal, it would need to be mitigated in all existing
deployed FQ implementations.

* We can use a heuristic to detect RFC3168 bottlenecks.

Experimental results suggest that this may be very difficult to do
(https://github.com/heistp/sce-l4s-ect1). In its present form, tightened AQM
settings in RFC 3168 bottlenecks can lead the heuristic to misidentify it as an
HFCC bottleneck, resulting in flow starvation. In the opposite direction, a few
milliseconds of jitter can lead the heuristic to misidentify an HFCC bottleneck
as RFC 3168, causing the benefits of HFCC to be lost, and incentivizing the
disablement of bottleneck detection.

* Worst-case throughput ratio is around 16:1.

Although it's questionable if 16:1 is an acceptable design limit, we appear to
see a higher ratio than that in this experiment:

http://sce.dnsmgr.net/results/ect1-2020-04-23-final/l4s-s2-twoflow/l4s-s2-twoflow-ns-cubic-vs-prague-codel1q_20ms_-80ms_tcp_delivery_with_rtt.svg

* We can bleach ECT(1) to Not-ECT.

While that may be one solution to the problem, it causes at least two new ones:
a) the benefits of HFCC are lost and b) we may lose ECT(1) for use in further
experiments should the ECT(1) as input experiment fail. We should think very
carefully before designing in the reliance on codepoint bleaching to fix
problems or set network policy.

*** Regarding #2 (selecting an alternate treatment in the network):

We can expect that endpoints and networks will use this to their maximum
advantage, setting and clearing ECT(1) whenever it best suits them to do so. 
The current proposal gives conditional latency priority to packets in the
low latency queue, creating a new class of service for Internet traffic.
Several concerns arise, a few of which are:

* How effective will DoS attacks be on the low latency queue?
* Will transit providers monetize the "low latency service", only allowing
  ECT(1) to pass if they're paid to do so? Will this lead to conflicts with
  network neutrality laws around network bias?
* As has been pointed out earlier on the tsvwg list, we have not yet tested the
  effects of VBR UDP flows on the low latency queue.

It is possible that some of the above concerns are unwarranted or have
solutions, but to date they seem to have gone largely unexplored by independent
parties, or the results of those explorations are not clear. Lastly, since
there is currently no implementation of queue protection to test, open
questions remain regarding the proposed codepoint usage in practice.

*** The process, and the problem statement

Among the key points raised by P Vixie, it was suggested that we should get
consensus on a problem statement before trying to do so on a codepoint. Based
on experimental results, I question whether the goal of "ultra-low latency for
all flows" is the right one for Internet traffic. Although we can agree that
reducing latency and jitter is a good thing, when the debate is framed around
that goal, we're led to draw key conclusions based on intra-flow latency plots
of capacity seeking flows made in closed environments. If we take a look at
what HFCC can actually achieve, what classes of Internet traffic actually
exist, and what conditions they face, we may find that the problem statement
and goals need realignment. If we manage to come to an agreement on that, we
may have a better chance of reaching consensus on the codepoint.

Finally, recognizing that the "ECT(1) as an output" proposal is still young and
needs work, we should avoid arguments that fall under the category of the sunk
cost fallacy, argument from authority or argument from popularity. This should
be an engineering decision that takes a long-term view rooted in good
engineering design. We can afford to wait, or roll something out in phases, as
long we reach a robust and well-designed solution in the end.

---

From: Steven Blake <[email-masked]>
Date: Wed, 13 May 2020 12:36:02 -0400

> http://sce.dnsmgr.net/results/ect1-2020-04-23-final/l4s-s2-twoflow/l4s-s2-twoflow-ns-cubic-vs-prague-codel1q_20ms_-80ms_tcp_delivery_with_rtt.svg

Hold that thought.

>   network neutrality laws around network bias?

If the "low latency service" in fact becomes a new Internet fast lane
(as you mentioned above), why wouldn't they? That's why operators
bleach DSCPs, right? This is not a knob that operators can turn that
immediately benefits all customers.

>   effects of VBR UDP flows on the low latency queue.

Regards,

// Steve

---

From: Pete Heist <[email-masked]>
Date: Wed, 13 May 2020 19:53:32 +0200

> On May 13, 2020, at 6:36 PM, Steven Blake <[email-masked]> wrote:
> 

The case linked to above was one where Codel’s AQM settings had been
tightened, so it's a special case in that sense, admittedly chosen as an example
of a “worst case” situation, although I can’t promise that’s the worst case.

Although I don’t think it was the intent, throughput unfairness can arise in
different situations that I’m not totally clear on, so it’s hard to predict if/where
the incentives would be beyond the low latency class. It does raise those
questions though.

---

From: Pete Heist <[email-masked]>
Date: Thu, 14 May 2020 09:11:11 +0200

Regarding the concern with hash collisions and FQ, I was kindly reminded that the birthday paradox applies to buckets in a hash table, so problems start to occur earlier than intuition would suggest. The details depend on the number of buckets, the flow mix, and the number of flows in the queue, so I’ve prepared a spreadsheet that attempts to quantify it:

https://docs.google.com/spreadsheets/d/1hOgTTZCKwR8f05Jjb3otJukh4gT6CKv-pnAJRqXEHLI/edit#gid=0

In a well loaded default fq_codel instance in a backhaul, for example, one might expect to see problems with around 1.76% of flows when 100 flows are present in the queue. It’s worth considering when relying on FQ for a safety guarantee.

> On May 13, 2020, at 1:57 PM, Pete Heist <[email-masked]> wrote:
>