Subject: Re: [tsvwg] Consensus call on ECT(1)

From: "K. K. Ramakrishnan" <[email-masked]>
Date: Sun, 17 May 2020 21:52:06 -0700

This is in response to the request to share opinions on the topic of 
which option of ECT(1) is preferable.

I would prefer the third option proposed in the email, and the following 
summarizes my view on the topic.

--

It is good to see the IETF actively pursuing work to implement ECN in 
the Internet. While it is important to re-examine previous design 
decisions as technology, workloads, and application needs change, it is 
also necessary to verify that new designs are equally applicable over a 
wide dynamic range to accommodate use cases as the Internet continues to 
evolve. The current debate between L4S and SCE is helpful to bring forth 
to the forefront some of the key considerations that go into the design 
of the ECN control loop.

The Internet is a diverse environment with widely varying, dynamic 
conditions, especially with regard to bandwidth and latency, and 
congestion control continues to have to strike a balance across multiple 
considerations. One of the critical aspects in congestion control is the 
need to be able to tolerate large round-trip delays in the Internet. 
Using the instantaneous queue as a main input to signal congestion is 
helpful when the feedback delays are quite small (relative to the buffer 
capacity in the routers and the packet arrival rate dynamics), but (as I 
found out at the beginning of my work on ECN in the mid-1980s) it is 
much more difficult to control congestion and have reasonable individual 
flow behavior when feedback delays get substantially larger. Active 
queue management techniques have been studied for a long time as a way 
to overcome this, in conjunction with a congestion signal that is not 
dependent on the highly varying, transient behavior of the instantaneous 
queue.

Additionally, congestion control has an almost equally important goal of 
maintaining fairness as far as possible across flows that have diverse 
characteristics. We all recognize that there is some level of 
RTT-unfairness with the way TCP congestion control operates. While the 
specific fairness goal to be achieved has been (and probably will 
continue to be) debated, it is very helpful to not exacerbate the level 
of RTT-unfairness we observe now, with the new designs. Additionally, it 
is important to be robust to loss of packets that carry feedback 
information for congestion control.

The considerations that went into the design of RFC 3168, including 
backwards compatibility to existing loss-based congestion control and 
providing the appropriate incentives, and avoiding dis-incentives, for 
adoption of ECN, may or may not apply as strongly to the current 
decision making. Nonetheless, a design that seeks to redo an important 
mechanism such as congestion control, that impacts so many, should 
carefully consider the issue of backwards compatibility to existing 
congestion control mechanisms that have been so widely deployed.

I would suggest that we carefully look at the existing proposals and 
require careful demonstration, through evaluations, across all these 
dimensions. Some additional analysis and evaluation appear to be needed.

A specific suggestion is to have emulations/simulations and experiments 
with implementations that carefully evaluate and successfully 
demonstrate reasonably stable throughput for long-lived flows and 
responsive to short-lived flows, adequate link utilization, and control 
the queue to mitigate congestion-induced delays and fairness 
(intra-protocol and inter-protocol) across connections with long and 
short RTT flows. Particularly critical is the need to be able to 
tolerate large RTTs that are representative of what we see on the 
Internet. This will help us understand the fundamental characteristics 
of the individual proposals and their backward compatibility.
--

Thanks,

  K. K. Ramakrishnan

-- 
K. K. Ramakrishnan
Professor
Dept. of Computer Science and Engineering
University of California, Riverside
Rm. 332, Winston Chung Hall
Tel: (951) 827-2480
Web Page: http://www.cs.ucr.edu/~kk/

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 19 May 2020 02:37:53 +0100

KK,

Thank you very much for your thoughts. I am wondering whether your 
request for more testing is in addition to all the testing we have done, 
or whether perhaps you are not aware of some or all of it. Pls see 
inline, particularly about large RTTs...

On 18/05/2020 05:52, K. K. Ramakrishnan wrote:
> I would prefer the third option proposed in the email, and the 
> following summarizes my view on the topic.
>
> -- 
>
> decisions as technology, workloads, and application needs change, it 
> is also necessary to verify that new designs are equally applicable 
> over a wide dynamic range to accommodate use cases as the Internet 
> continues to evolve. The current debate between L4S and SCE is helpful 
> to bring forth to the forefront some of the key considerations that go 
> into the design of the ECN control loop.
>
> congestion control continues to have to strike a balance across 
> multiple considerations. One of the critical aspects in congestion 
> control is the need to be able to tolerate large round-trip delays in 
> the Internet. Using the instantaneous queue as a main input to signal 
> congestion is helpful when the feedback delays are quite small 
> (relative to the buffer capacity in the routers and the packet arrival 
> rate dynamics), but (as I found out at the beginning of my work on ECN 
> in the mid-1980s) it is much more difficult to control congestion and 
> have reasonable individual flow behavior when feedback delays get 
> substantially larger. Active queue management techniques have been 
> studied for a long time as a way to overcome this, in conjunction with 
> a congestion signal that is not dependent on the highly varying, 
> transient behavior of the instantaneous queue.

[BB] One of the top two motivations for L4S is to shift the 
RTT-dependent aspect of the feedback loop out of the network to the end 
system. The signal is still smoothed, but by each congestion control 
algo at each sender, not by the AQM.

Senders know their own RTT, so each can smooth / filter over a timescale 
related to its own RTT. In contrast, the network doesn't know the RTTs 
of all the flows, and it would be hard for it to ever know. So smoothing 
in the network has always had to be conducted assuming there might be 
some worst-case flows with the largest likely RTT.

Actually smoothing in the network tends to not to be configured for the 
absolute worst-case, which would be far too sluggish. For instance, 
100ms is typically considered to be a 'reasonable' max RTT. But whenever 
an RTT exceeds 100ms (half-way round the earth and back in glass = 
200ms), it is outside the stability range of the traditional AQM. In 
contrast, when the sender smooths based on its own RTT, there is no 
assumed default RTT, so it can never be outside the default range - it 
automatically gets the RTT right.

Another upside of this shift to each sender, is that the sender knows 
which phase of congestion control it is in, and whether it is filling 
the pipe, etc. E.g. in slow start it need not smooth the signal, so it 
can respond immediately. Whereas in congestion avoidance, it does need 
to smooth the signal (dependent on its own RTT). In contrast, when the 
network is smoothing, it delays the signal by a worst-case RTT for all 
flows, whether the delay is needed or not.

The downside is that each flow sees only its own part of the aggregate 
signal, so there is less aggregation that would otherwise remove more noise.

This insight about shifting the RTT-dependent part of the control loop 
was first pointed out in section 5 of [Kuzmanovic05]. It was also 
rediscovered in [Bhandarkar07]. A theoretical analysis of this shift is 
in [Alizadeh11], particularly section 5.

Despite sounding intuitively correct, as you rightly say, such a radical 
shift needs to be exhaustively tested. Before we brought L4S to the IRTF 
in early 2015, we had conducted repeated experiments in a wide range of 
thousands of scenarios with different link rates, mixtures of RTTs, 
numbers of flows, patterns of short and long flows, etc, etc. Since 
then, we have always ran a set of these tests to check for regression 
every time we change anything.

We ran the original experiments over real broadband equipment, then we 
validated that our lab testbed gave repeatable results. So nowadays, we 
rely on the lab testbed.

Each run automatically produces a set of plots summarizing the 
statistical properties of the experiments. In the past we have tacked 
all these plots onto a copy of a paper that we intend to submit to a 
journal (which documents all the experimental conditions, etc). We are 
just producing a new one. Prior to that, the most recent one (Jul 2019) 
was here:
     http://bobbriscoe.net/pubs.html#DCttH_TR (also linked vai the L4S 
landing page at https://riteproject.eu/dctth/#papers )
  (in fact the original evaluation paper from 2015 is just below it, but 
since then we have improved the visualization of the large number of 
experiments).

Please see if you are satisfied with the long RTT cases, and let us know 
if there are specific tests you would rather we had done.

Of course, ideally we want to measure under more realistic conditions as 
well. But that needs greater certainty that the codepoint can be used on 
production networks, which needs the IETF's blessing.

[Kuzmanovic05] Kuzmanovic, A., "The Power of Explicit Congestion 
Notification," Proc. ACM SIGCOMM'05, Computer Communication Review 35(4) 
In: SIGCOMM '05: Proceedings of the 2005 conference on Applications, 
technologies, architectures, and protocols for computer communications 
Vol.35 No.4 ACM Press (2005)

[Bhandarkar07] Bhandarkar, S., Reddy, A.L.N., Zhang, Y. & Loguinov, D., 
"Emulating AQM from End Hosts," ACM SIGCOMM Computer Communication 
Review 37(4):349–360 Association for Computing Machinery (August 2007)

[Alizadeh11] Alizadeh, M., Javanmard, A. & Prabhakar, B., "Analysis of 
DCTCP: Stability, Convergence, and Fairness," In: Proc. ACM 
SIGMETRICS'11 (2011)

>
> Additionally, congestion control has an almost equally important goal 
> of maintaining fairness as far as possible across flows that have 
> diverse characteristics. We all recognize that there is some level of 
> continue to be) debated, it is very helpful to not exacerbate the 
> level of RTT-unfairness we observe now, with the new designs. 

[BB] At the February interim tsvwg meeting, Koen presented the new 
RTT-independent variant of TCP Prague.
https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/materials/slides-interim-2020-tsvwg-01-sessa-l4s-tcp-prague-update
 From memory, Koen switched to a live system to demonstrate it and show 
results, but I can't immediately find a link to a recording of that 
session. I believe he and Olivier are writing up evaluation results of 
that code. And next time we post one of our regression tests, we need to 
extend the scenarios to exercise more of the lower end of the RTT 
spectrum as well.

Of course, I also want to encourage others to validate these tests, and 
do their own.

> Additionally, it is important to be robust to loss of packets that 
> carry feedback information for congestion control.

[BB] Are you aware of the design of Accurate ECN TCP feedback that is a 
prerequisite for L4S (if using TCP)? It is designed to be robust to ack 
loss or a certain degree of deliberate thinning:
     https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn

IETF QUIC (as opposed to Google QUIC) includes precise robust ECN 
feedback as standard.

These satisfy the requirement for robust feedback in the draft L4S spec:
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4

> backwards compatibility to existing loss-based congestion control 

[BB] Note that all the L4S congestion controls fall back to 
Reno-Friendly in response to loss (another requirement in the same draft 
L4S spec).

> and providing the appropriate incentives, and avoiding dis-incentives, 
> for adoption of ECN, may or may not apply as strongly to the current 

[BB] I'm not sure whether you are saying that specific evaluation is 
lacking, or that it is likely that more evaluation is needed. I know 
it's difficult to know everything that has and hasn't been done. But I 
think the question the chairs have posed is whether there is any 
specific testing needed before a decision can be made between ECT(1) as 
an input or as an output.

You might have seen Pete Heist's recent tests of the Fallback algorithm, 
and our site recording a wide range of tests of this algorithm: 
http://folk.uio.no/asadsa/ecn-fbk

>
> A specific suggestion is to have emulations/simulations and 
> experiments with implementations that carefully evaluate and 
> successfully demonstrate reasonably stable throughput for long-lived 
> flows and responsive to short-lived flows, adequate link utilization, 
> and control the queue to mitigate congestion-induced delays and 
> fairness (intra-protocol and inter-protocol) across connections with 
> long and short RTT flows. Particularly critical is the need to be able 
> to tolerate large RTTs that are representative of what we see on the 

[BB] These specific suggestions seem to be covered by tests I've pointed 
to earlier in this email. Can you say why those particular tests are 
lacking. There are many reasons I can think of, not least that they have 
so far been primarily conducted over fixed networks. But I want to check 
that you are aware of what has been done, because testing goal posts can 
always be moved closer to the horizon, and at some point we have to be 
able to say that there's been sufficient lab testing. So we can move on 
to start testing on production networks, which is the stage we're trying 
to get past at the moment, I believe.

Best regards, and thank you again,

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/