Subject: [tsvwg] Jonathan Morton's rationale *against* ECT(1) as an input

From: Jonathan Morton <[email-masked]>
Date: Fri, 15 May 2020 06:06:20 +0300

When this consensus call was published, I immediately registered my position as being for ECT(1) as an output.  In part that is due to my involvement in the SCE proposal, which uses ECT(1) that way.  Naturally, not everyone agrees with me.  But at this stage I am more concerned that the dangerous L4S proposal should *not* go ahead; it matters much less if my own project's progress is delayed pending further development.

The L4S proponents claim that L4S is "mature" and that the open issues surrounding it "have been resolved".  Nothing could be further from the truth, and the fact that such falsehoods are being presented in this context should give rise to considerable concern.  But I should not baldly assert this claim without evidence, so I will lay some out below.

The claim of L4S' "maturity" is based primarily on the length of time it has been under development, with some claiming that this development time extends back ten years.  Bob Briscoe is slightly more modest in his recent posts, claiming that he started work on an enhancement of DCTCP in 2013 - seven years ago.  I have no reason to doubt this assertion.

But I prefer to use a different measure of maturity: the extent to which defects in the design and implementation have been ironed out.  Keen observers will remember the following plot, derived from L4S code published only two weeks before the most recent TSVWG virtual meeting, and publicly announced mere days beforehand:

This is a plot demonstrating a serious bug in the congestion response to loss in TCP Prague, a key component of L4S.  The network scenario triggering it is not an exotic one; it is a pair of flows (one CUBIC, one Prague) over a 50Mbps throughput, 80ms RTT path, with a PIE AQM at the bottleneck that has had ECN support disabled.  Such conditions could often be encountered at DOCSIS-3.1 cable modems, in which an implementation of PIE is mandatory.

You would expect that a test suite for a protocol expected to traverse the public Internet would cover this scenario, as well as the other scenarios which triggered the same bug (which, IIRC, also involved non-ECN bottlenecks of various kinds, unfortunately the most common type in today's Internet).

To their credit, the bug was swiftly fixed once it was pointed out.  My concern lies with the fact, however, that *they* did not detect it themselves, despite the voluminous test data the L4S team has published.  An implementation in which such defects can be found so easily cannot legitimately be described as "mature".

As for the L4S issue list, I note that it has been extended in recent days, rather than reduced, and the most serious of the original issues opened at Montreal last year (#16) was only addressed by running code (the "classic detection heuristic") in the latest release mentioned above - announced with clearly insufficient time to perform a proper evaluation of its performance and shortcomings.  Nevertheless we were paying sufficient attention to uncover some flaws immediately, and we have noticed some more in the time since the meeting.

Here is one of those new results:

The above is a test of what happens when all of the L4S components are deployed, exactly as its proponents wish, on a typical asymmetric last-mile cable link.  No artificial jitter or burstiness has been introduced here; L4S is, after all, associated with Low Latency DOCSIS which removes those properties from cable.  All we have done is apply a bidirectional load of four TCP Prague flows each way; a version of the RRUL test which we have previously used as a "mild" stress test while developing FQ-AQMs.  We have considerably more severe tests in our arsenal, but the results of this one are easier to interpret.

The results are that two of the downstream Prague flows appear to think they're running through a Classic AQM, when in fact they're running through the LL queue of DualPI2 in both directions.  This causes them to give way to the other two which continue to grow steadily, until they also apparently switch into "classic mode", exhibiting a sawtooth pattern precisely in-phase with the others.  There is also a definite imbalance between two pairs of flows in the more constrained upstream direction, though there is no obvious sawtooth symptom there to point to.

In conjunction with other results, we have to say that in our opinion, the "classic detection heuristic" lacks discriminative power and is therefore *not* a solution to issue #16.  That is what we predicted upon first reading the paper describing it, in roughly the Singapore timeframe, and that prediction has been borne out in full.

For comparison purposes, here are the same test conditions run using the current state of the art - TCP CUBIC, through a robust and widely deployed RFC-3168 FQ-AQM:

Yes, there is a sawtooth; the throughput of each flow oscillates around the average, and the overall total is a little less than the link capacity.  But there is also no sustained unfairness between flows, the throughput and latency performance is also pretty good, and the sawtooth cycles run out-of-phase with each other, which helps to maximise utilisation.

This is the standard against which you have to measure any radical new proposal. It is a comparison which L4S cannot (presently) withstand.  So how did L4S get here?

It began, of course, with DCTCP.  In a controlled environment, such as an isolated datacentre backplane, it is sometimes permitted to bend the rules a little bit, and that is what DCTCP did.  It was specifically intended to optimise performance in an environment where path delays and queue lengths are measured in microseconds, rather than the milliseconds that are more common on the wider Internet.  And it was quickly realised that if DCTCP shared a queue with a conventional TCP flow, the latter would be squeezed out.  So datacentre administrators knew they could deploy either DCTCP or conventional TCP, but not both, on a given network.

The fundamental sin of DCTCP was that it redefined the CE mark of ECN, from "please halve your congestion window" to "please subtract half a segment from your congestion window".  Hence a transport expecting only the former meaning will overreact to a signal carrying the latter meaning.  In a controlled environment, it is permitted to make such changes on a local basis and live with the consequences.  In the open Internet, you must also consider externalities, because you cannot expect everyone to adjust to the new meaning overnight.

So, how has L4S developed from DCTCP in the past seven years?  Considering only the demonstrated technology:

1: A new method of feeding back the CE marks to the sender - AccECN.  This probably has only a marginal effect on performance, but it could be viewed as a more robust feedback signal.  Its successful negotiation also acts as a proxy for determining whether both ends of the connection support L4S.

2: Senders now inform the network whether they should apply the AIMD signal or the DCTCP signal via CE, by setting ECT(1) at origin instead of ECT(0).  But there is no guarantee whatsoever that the network understands this directive; existing networks certainly do not, through they will pass it through unchanged in nearly all cases.

3: An AQM which does understand the above directive - DualQ-Coupled-PI2.  It is supposedly designed to be simple enough to implement in low-cost CPE and high-speed head-end equipment.  It has correspondingly weak performance characteristics compared to an FQ-AQM, and does not even attempt to enforce admission control on its new de-facto PHB; not even a dead-simple Deficit Round Robin between the two queues.

4: A heuristic to detect whether the latter AQM, or at least something compatible with it, is present at the path bottleneck.  As discussed above, this heuristic has proved to demonstrate insufficient discriminating power; it often identifies DualQ as "classic" and a plain RFC-3168 AQM as "L4S".

These are rather thin improvements for such a lengthy development cycle, and they have not actually succeeded in "taming" DCTCP for safe use over the Internet.  Two additional pieces of technology are often mentioned (Paced Chirping and Queue Protection) but have not been demonstrated in running code.

What's more, the ECT(1) "classifier" acts so much like a Diffserv mechanism that it really should be subject to the same sort of scrutiny.  Indeed some observers are already suggesting management techniques which sound awfully like those usually associated with Diffserv.  Bleaching, in particular, is a Diffserv management technique that the ECT(1) codepoint was supposedly chosen specifically to avoid.  It is certainly not a useful way to "add safety to" the L4S system, by effectively disabling it on networks which are both sufficiently aware of its misbehaviour to do something about it, and yet reluctant to actually deploy it themselves.

Now, I fully understand if the TSVWG is reluctant to assign ECT(1) to the relatively young SCE, which has only been actively working in this particular problem space for a little over a year.  We still have much work to do, and the working environment is presently difficult for a number of reasons.  But I also think that, based on demonstrated results, the L4S project is due for retirement, not promotion.

I think this article is worth reading in this context.  It was written for a different context, but the lessons apply:  http://www.ganssle.com/item/guide-to-managing-firmware-teams.htm

So in summary, I primarily support allocating ECT(1) to be an output from the network.  But I would settle for merely *not* allocating it for use as an input that changes the meaning of CE.

 - Jonathan Morton