Subject: [tsvwg] Suggestion of using classic ECN as early congestion signal

From: daihuichen <[email-masked]>
Date: Wed, 28 Apr 2021 13:16:11 +0000

Hello folks,

I have been following the discussions of L4S and guard DSCP, from which I learnt the huge difficulty in improving of a standard that has been deployed.
The extensions or replacements of classic ECN [RFC3168], including L4S, SCE (Some Congestion Experience) and AccECN, intend to provide more semantics of the congestion, rather than simply a '0-1' signal.
In parallel to the development of richer congestion information, IMHO, we also need to think about how to satisfy the ever increasing performance requirements from the business, using off-the-shelf techniques.
>From our experience, I would like to suggest using the classic ECN AQM, accompanied by a new window adjustment algorithm, to achieve high-performance transport. We have proven its effectiveness in high-speed datacenter networks.

An early congestion notification is very important for congestion control, as it allows senders to react before severe congestion happens. RED, since its advent, aims to provide early congestion signal (as suggested by its name Random Early Detection). If the queue size is between Kmin and Kmax, the marking probability is proportional to the queue size, and these marks are the "early" congestion signals. If the queue size is above Kmax, all the packets are marked, and we refer to these marks as "late" congestion signals. In our algorithm, an important modification to the receiver is, it acts like a mirror of the ECN marks in data packets (copies the ECN signal in data packets to ACKs directly). This is different from classic TCP that limits the feedback to one ECN-event per RTT. In this way, ECN signals are fed back to the senders "as-is". Consequently, simply using different Kmin and Kmax (rather than Kmin==Kmax) for ECN is able to feed back early congestion signals. The range of [Kmin, Kmax] is determined by how early one wants to signal the congestion. In our practice, we care very much about the latency so we set Kmin=5KB, Kmax=100KB, very low thresholds. Meanwhile we can well control the queue jitter to prevent queue underflow.

The next question is if senders need to distinguish "early" and "late" congestion signals, and react differently. Our response is No. A simple and unified operation for all the ECN marks is enough, as it can respond to the "degree" of congestion. The operation is to reduce the window size by a fixed step beta (0<beta<1) upon each Echoed-CE (ECE) flag. The reason is as follows. Within one RTT, if only sporadic ECE flags are echoed to the sender, then the window decreases a little bit to mitigate the transient congestion. As the queue size approaches Kmax, there are more marks so the window decreases more sharply. If the queue size goes beyond Kmax, all the data packets are marked then the window is decreased by beta*cw, a multiplicative decrease to quickly slow down the sending rate (if beta=0.5, this leads to a half-cut of cw). Therefore, with un-equal Kmin and Kmax, the number of received ECE flags in one RTT reflects the "degree" of congestion, so reducing the window by a constant beta for each ECE naturally responds to the congestion degree - simple and effective.
For ACKs without ECE flag, we still let it increase the window by alpha/cw (alpha is parameter), which is additive increase. With AIMD, all flows will converge to fairness eventually.
In summary, our suggestions are:
1) Using different Kmin and Kmax values for ECN marking thresholds;
2) Receivers echo ECN signals "as-is";
3) Senders decrease the window by a fixed step beta (0<beta<1) upon each ECE flag.

Details of the above algorithm can be found in this draft: https://tools.ietf.org/html/draft-dai-tsvwg-pfc-free-congestion-control-01
We refer to this new algorithm as LDCP. It's worth noting that this is an algorithm proposal. The required codepoints for implementation can vary from protocol to protocol, e.g., RoCEv2 are over UDP, there is no TCP header in the packets. The congestion info can be encoded into the CNP packets, or RoCE BTH (basic transport header) of ACK packets.

This algorithm has been safely running in a *lossy* datacenter network for more than one year (with RoCEv2), and we observed well controlled queue switch size, almost no packet loss and high-throughput.
Indeed, datacenter networks is where tight congestion control can happen as it features low hardware latency, availability of smart NICs, etc. I summarize the differences at the end of this email.

Backward compatibility and guard DSCP:
Cloud datacenter networks is our primary target scenario, but we believe this algorithm is also applicable to the Internet.
Recent discussion of the guard DSCP reminds me the importance of restricted experimental domain, incremental deployment and backward compatibility over the Internet.
If a classic bystander TCP sender receives an early ECN signal, it cuts the window by half directly, while LDCP senders only reduce the window by beta, leading to unfairness between flows.
We could use a guard DSCP for LDCP traffic to restrict them inside the experimental domain. Meanwhile, we could use WRED to configure two sets of ECN marking thresholds, one set is Kmin==Kmax for classic TCP traffic, the other is Kmin < Kmax, for DSCP labelled LDCP traffic. (WRED supports at least 3 set of ECN marking thresholds: green, yellow, red). A mapping between DSCP and WRED marking thresholds needs configured on the switches/routers.
These are general rules, and I leave the details to later discussion if anyone is interested.

Please let me know what you think about this algorithm, your responses are very important to improve this work. Thanks~

Best,
/Huichen
----------------------------------------------------------------------------------------------------------------

Today, the Internet and datacenter networks are diverging in many ways:
1. Forwarding delay and bandwidth: the switching/forwarding delay of a packet on a datacenter switch becomes extremely small, e.g., 300ns, while such delay of a router is usually in microseconds. Meanwhile, 100Gbps are common in datacenter networks, and 200Gbps are being gradually deployed.
2. Path hops: the path hops of data center networks is much smaller than that of the Internet, so is the end-to-end hardware latency. The RTT of datacenter networks is at the scale of microseconds or 10s of microseconds. Such short latency makes "tight" congestion control possible. As we know, a small congestion feedback delay is very beneficial to the control stability of the congestion control loop.
3. Smart NICs: for high-throughput and low-latency considerations, transport in data-center networks is offloaded to smart NICs, including ToE (TCP offloading Engine) and RoCE. Smart NICs usually have hardware packet encapsulation and de-capsulation logic, thus can generate and parse ACK packets much faster than CPU. Therefore, ACK (which carries congestion signals) returns much more timely and the smart NIC can also process it in-time.
4. Congestion control complexity: the low-latency transport congestion control algorithm is expected to be simple (avoid multiplication and division as much as possible), to reduce circuit complexity (if ASIC) and the processing delay.
5. Deployment of new protocols: as datacenter networks are under the control of individual companies/organizations, it is easy to deployment new protocols and isolate the new traffic by a separate queue. However, incremental deployment of the Internet is extremely complicated and time-consuming.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 28 Apr 2021 18:11:38 +0300

> On 28 Apr, 2021, at 4:16 pm, daihuichen <[email-masked]> wrote:
> 
> 2) Receivers echo ECN signals “as-is”;

What you propose is very similar to the existing DCTCP, and would have much the same problems.  It cannot be deployed on the general Internet, because it would fiercely outcompete conventional TCP (and all the other protocols that have been carefully engineered to coexist nicely with conventional TCP) at any ECN-marking node that did not enforce per-flow fairness or could not distinguish the two flows.

L4S attempts to solve this problem by pre-identifying its traffic and directing it to a separate queue and AQM.  Unfortunately it doesn't solve the problem at existing AQM nodes that don't know about the new identifier, and that is the basic premise of the current contention in the WG.

SCE solves the problem by by putting the new "early" congestion signal, named "Some Congestion Experienced", on a different ECN codepoint than the existing "Congestion Experienced" signal, and adding a separate feedback channel for it (using one spare TCP header flag).  Existing receivers and transports transparently ignore the new signal, and it can be arranged so that only nodes that are prepared to compensate for the fairness issues will produce the new signal (as well as the old one at higher congestion levels).  The crucial point is that receivers and transports are in no doubt as to which type of congestion signal they have received from the network, unlike DCTCP or L4S.

Please take these existing developments into consideration when making proposals in this area.  If you think yours still has merit, please take the time to explain how it is different to the above and handles the known problem areas.

 - Jonathan Morton

---

From: daihuichen <[email-masked]>
Date: Fri, 30 Apr 2021 03:55:43 +0000

Hi Jonathan,

Thank you very much for your comments. Pls see my response below, labeled with [Huichen].

-----Original Message-----
From: Jonathan Morton [mailto:[email-masked]] 
Sent: 2021年4月28日 11:12 PM
To: daihuichen <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] Suggestion of using classic ECN as early congestion signal

[Huichen] Jonathan, I can see that your focus is on the deployment issue in current Internet, and I am not ignoring existing developments, that's why I propose to use guard DSCP. But what I define is an algorithm that specifies how early congestion signal can be generated, and how to react to the congestion signals (not the codepoints). It has been verified by good results in datacenter networks, which is our target scenario. As you may notice, the datacenter networks and the Internet have been different in many aspects. And the datacenter networks do not have incremental deployment issues (at least not as much as the Internet), also datacenters require much higher performance, so it's reasonable that datacenter networks go ahead with more "tight" congestion control algorithms. An abundant of work exist in this area(NDP[1], DCQCN[2], TIMELY[3], Swift[4], HPCC[5]), and keep evolving. We cannot ignore the development of this area also.

The SCE scheme, as you said, opens up a separate feedback channel. If our algorithm is to be deployed over the Internet, can we make use of this channel to feed back our congestion signal? Thanks.

[1]Handley, M., Raiciu, C., Agache, A., Voinescu, A., Moore, A. W., Antichi, G., and Wójcik, M. Re-architecting datacenter networks and stacks for low latency and high performance. In Proceedings of the ACM SIGCOMM (2017), ACM, pp. 29–42.
[2] Zhu, Y., Eran, H., Firestone, D., Guo, C., Lipshteyn, M., Liron, Y., Padhye, J., Raindel, S., Yahia, M. H., and Zhang, M. Congestion control for large-scale rdma deployments. In Proceedings of the ACM SIGCOMM’15 (2015), ACM, pp. 523–536.
[3] Mittal, R., Dukkipati, N., Blem, E., Wassel, H., Ghobadi, M., Vahdat, A., Wang, Y., Wetherall, D., Zats, D., et al. Timely: RTT-based congestion control for the datacenter. In Proceedings of the 2015
ACM SIGCOMM (2015), ACM, pp. 537–550.
[4] Kumar, G., Dukkipati, N., Jang, K., Wassel, H. M. G., Wu, X., Montazeri, B., Wang, Y., Springborn, K., Alfeld, C., Ryan, M., Wetherall, D., and Vahdat, A. Swift: Delay is simple and effective for congestion control in the datacenter. In Proceedings of ACM SIGCOMM (New York, NY, USA, 2020), SIGCOMM’20, Association for Computing Machinery, p. 514–528.
[5] Li, Y., Miao, R., Liu, H. H., Zhuang, Y., Feng, F., Tang, L., Cao, Z., Zhang, M., Kelly, F., Alizadeh, M., and et al. Hpcc: High precision congestion control. In Proceedings of the ACM Special Interest Group
on Data Communication, SIGCOMM’19 (New York, NY, USA, 2019), ACM.

/Huichen

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 30 Apr 2021 09:03:36 +0300

> On 30 Apr, 2021, at 6:55 am, daihuichen <[email-masked]> wrote:
> 

Yes, I think it does pretty much exactly what you need.

In fact, you might look at our reference implementation for SCE, which includes an example of how to implement a DCTCP-like response to high-fidelity congestion signals, overlaid on a NewReno-type response to conventional congestion signals (both ECN and loss).

https://github.com/chromi/sce/blob/sce/net/ipv4/tcp_dctcp_sce.c

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Thu, 29 Apr 2021 23:44:20 -0700

I am always interested in data on new congestion control algorithms.

Got sources for linux?

Do you have any sample packet captures of what things look like under
contention? One of my favorite tests for stressing
a network is the flent "rrul" test which not only generates a useful
set of graphs but an experienced eye can pull much from captures
via tcptrace, xplot.org, wireshark, etc.

So the output of a flent rrul, or rtt_fair_var to multiple
destinations on the same wire or switch (think a 10GB switch going to
a 1GB switch with 10 hosts on it), with a packet capture would be
fascinating. Thx!

---

From: daihuichen <[email-masked]>
Date: Fri, 30 Apr 2021 08:29:30 +0000

Hello Dave,

Thanks for your interest.
This algorithm is currently implemented in the firmware for an RoCE NIC. I don’t know if the "rrul" test is applicable to RoCE?

/Huichen

-----Original Message-----
From: Dave Taht [mailto:[email-masked]] 
Sent: 2021年4月30日 2:44 PM
To: daihuichen <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] Suggestion of using classic ECN as early congestion signal

Got sources for linux?

Do you have any sample packet captures of what things look like under contention? One of my favorite tests for stressing a network is the flent "rrul" test which not only generates a useful set of graphs but an experienced eye can pull much from captures via tcptrace, xplot.org, wireshark, etc.

So the output of a flent rrul, or rtt_fair_var to multiple destinations on the same wire or switch (think a 10GB switch going to a 1GB switch with 10 hosts on it), with a packet capture would be fascinating. Thx!