Subject: [tsvwg] Deprecating RFC 3168 for future ECN experimentation

From: Steven Blake <[email-masked]>
Date: Fri, 26 Mar 2021 13:01:42 -0400

A lot (not all) of the recent arguments revolve around the assumption
by some that RFC 3168 ECN deployment barely exists in the Internet, and
the few networks where it does can be safely ignored, or cleaned out,
or be expected to take proactive measures to protect themselves, which
may in practice require them to lobby their router vendors to spin
patch releases to enable (some of) the mitigation measures detailed in
-l4ops-02 Sec. 5.

If that is the WG consensus, then I *strongly urge* the WG to do the
following:

1. Push to move RFC 3168 ECN to Historic

2. Adopt the following "New ECN" signals for future ECN
experimentation:

- Not-ECT
- ECT
- CE-a
- CE-b

This second step would allow for two sets of experiments. The semantics
of CE-a and CE-b for the first set of experiments would be as follows:

- CE-a: "Decelerate"
- CE-b: "Decelerate harder" (multiplicative decrease)

The exact behavior elicited by the "Decelerate" signal would be the
subject of investigation. Since we are certain that any remaining RFC
3168 deployments can be safely ignored, then ECT/CE-a/CE-b can be used
as unambiguous signals to steer packets into a low-latency queue, if
desired.

The semantics of CE-a and CE-b for the second set of experiments would
be as follows:

- CE-a: "Decelerate"
- CE-b: "Accelerate"

An aggressive fraction (100%?) of CE-b marked packets traversing a
queue not in "Accelerate" state would be re-marked to either CE-a or
ECT. Any packet discard (or detection of high delay variation?) must
disable the transport's "Accelerate" mechanism for some interval and
should cause the transport to revert to "TCP-friendly" behavior for
some (different?) interval. The exact behaviors of "Accelerate" and
"Decelerate" signals would be the subject of investigation. Again,
since we are certain that any remaining RFC 3168 deployments can be
safely ignored, then ECT/CE-a/CE-b can be used as unambiguous signals
to steer packets into a low-latency queue.

The differences between these two sets of experiments hinge on whether
there is more utility in an "Accelerate" signal coupled with a
"Decelerate" signal, or with two separate levels of "Decelerate"
signals. Since it is WG consensus that the RFC 3168 ECN experiment
failed after two decades, we probably only get one more chance to get
this right, so careful and exhaustive experimentation which explores
the design space is in order.

Obviously, both sets of experiments cannot be run simultaneously on
intersecting parts of the Internet. I leave the options for safely
isolating these experiments as an exercise for the reader. Since we are
certain that any remaining RFC 3168 ECN deployments can be safely
ignored, I suggest choosing bit assignments for the four signals that
induce maximum pain in the obstinate minority that might still deploy
RFC 3168 ECN.

Now, *if it is not WG consensus* that any existing RFC 3168 ECN
deployments can be safely ignored, then I *strongly urge* the WG *to
not adopt* experimental proposals that place burden and/or risk on
networks that have deployed it.

TL;DR: Either RFC 3168 ECN exists in the Internet, or it doesn't.
Decide, and act appropriately.

Regards,

// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 26 Mar 2021 19:05:41 +0200

> On 26 Mar, 2021, at 7:01 pm, Steven Blake <[email-masked]> wrote:
> 

As is probably no surprise, the above is the position I support.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Sat, 27 Mar 2021 15:41:18 +0100

I agree overall. If we want to introduce a proposal that's incompatible
with RFC3168, we should first make it historic.

Before we do that though, we should make sure that the current CE is
not actually useful. Figure 5 in this paper suggests some benefit to
two bits of signal as opposed to one:
http://buffer-workshop.stanford.edu/papers/paper34.pdf

A second signal provides a harder backoff without packet loss, for
example during capacity changes or flow introductions. It wouldn't be
ideal to deprecate RFC3168, only to find out that another bit of signal
in line with CE, along with ABE in RFC8511, or something similarly
deployable with today's equipment, is still useful.

It's also my position that we can't ignore existing RFC3168
bottlenecks, not just for safety but also for performance. The recent
ISP study we did suggested RFC3168 AQMs may be present on ~10% of
Internet paths there. Prior to that we heard 5% elsewhere. Whatever the
number is exactly, these AQMs do exist and mark in response to both
ECT(0) and ECT(1). If you introduce traffic that backs off much less in
response to CE, the AQMs may operate sub-optimally, since they weren't
designed with that kind of traffic in mind
(https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes).

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 28 Mar 2021 18:36:11 +0000

Hi
I am not sure about the purpose of this discussion but it is free speech.

I any case it could be worth stating (for the general audience) that the 
charter of the WG is still as given by
https://tools.ietf.org/wg/tsvwg/charters
In particular
  Oct 2021 - Submit "Low Latency, Low Loss, Scalable Throughput (L4S) Internet 
Service: Architecture"
     as an Informational RFC
  Oct 2021 - Submit "DualQ Coupled AQM for Low Latency, Low Loss and Scalable 
Throughput" as an Experimental RFC
  Oct 2021 - Submit "Identifying Modified Explicit Congestion Notification 
(ECN) Semantics for Ultra-Low Queuing Delay"
     as an Experimental RFC

The ideas below have likely been floated before, after all they are not rocket 
science. In particular the accelerate/decelerate was recently presented in a 
Stanford or MIT paper (not sure which) and I am pretty sure that it was 
discussed even before that.
There are reasons to why we are at the present proposed solution.

/Ingemar

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Pete Heist
> Sent: den 27 mars 2021 15:41
> To: Steven Blake <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [tsvwg] Deprecating RFC 3168 for future ECN experimentation
>
> I agree overall. If we want to introduce a proposal that's incompatible with
> RFC3168, we should first make it historic.
>
> Before we do that though, we should make sure that the current CE is not
> actually useful. Figure 5 in this paper suggests some benefit to two bits of
> signal as opposed to one:
> A second signal provides a harder backoff without packet loss, for example
> during capacity changes or flow introductions. It wouldn't be ideal to
> deprecate RFC3168, only to find out that another bit of signal in line with 
> CE,
> along with ABE in RFC8511, or something similarly deployable with today's
> equipment, is still useful.
>
> It's also my position that we can't ignore existing RFC3168 bottlenecks, not
> just for safety but also for performance. The recent ISP study we did
> suggested RFC3168 AQMs may be present on ~10% of Internet paths there.
> Prior to that we heard 5% elsewhere. Whatever the number is exactly, these
> AQMs do exist and mark in response to both
> (https://protect2.fireeye.com/v1/url?k=d314eb2b-8c8fd268-d314abb0-
> 861fcb972bfc-578ed7f13c74c412&q=1&e=ebe2f7ed-6fe3-4c6b-8b9d-
> 07170e08ad77&u=https%3A%2F%2Fgithub.com%2Fheistp%2Fl4s-
> tests%2F%23intra-flow-latency-spikes).
>
> > by some that RFC 3168 ECN deployment barely exists in the Internet,
> > and the few networks where it does can be safely ignored, or cleaned
> > out, or be expected to take proactive measures to protect themselves,
> > which may in practice require them to lobby their router vendors to
> > spin patch releases to enable (some of) the mitigation measures
> > detailed in
> > -l4ops-02 Sec. 5.
> >
> > This second step would allow for two sets of experiments. The
> > semantics of CE-a and CE-b for the first set of experiments would be as
> follows:
> >
> > - CE-a: "Decelerate"
> > isolating these experiments as an exercise for the reader. Since we
> > are certain that any remaining RFC 3168 ECN deployments can be safely

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 28 Mar 2021 22:13:52 +0200

Dear All,

Ingemar helpfully directed my to our charter. Where I find among other's:

"(C) Maintenance of IP Differentiated Services (DiffServ) mechanisms,
    which involves mostly advisory documents on the use of DiffServ in
    specific application scenarios. Other work items related to DiffServ
    require Area Director approval."

Based on this, I would say, mandating a guard DSCP for the duration of the L4S experiment (and after depending on the experiments evaluation?) seems to be fully within this WG charter, at worst after talking to the respective AD. 

Best Regards
	Sebastian

> On Mar 28, 2021, at 20:36, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 29 Mar 2021 10:26:30 +0200

Hi,

> with RFC3168, we should first make it historic.

I just want to add that RFC 3168 is a proposed standard, whereas we
want to introduce L4S as an _experiment_. All the recent work of the
AQM WG would be also void and I thought that vendors are actually
implementing this in home routers. I think that we must be really sure
about RFC3168 (non-)deployment before deprecating RFC3168. Even RFC 8311
still stated: "Forwarding behavior as described in RFC 3168 remains the
preferred approach for routers that are not involved in ECN experiments,".

> deployable with today's equipment, is still useful.

I'm also in favor of having better (i.e., more fine grained)
explicit congestion signals as end-systems may use them to
come up with better decisions. I expressed this clearly in the
ECT(1) input cs. output signal discussion. Now, if we come up
with a DSCP as additional safeguard qualifier for L4S,
which I support, then it makes sense to revisit the earlier
WG decision and use ECT(1) as additional congestion signal.

> (https://github.com/heistp/l4s-tests/#intra-flow-latency-spikes).

In case home routers are using RFC 3168 and congestion is in
downstream direction one would only be able to detect this by seeing
ECE in the other direction. Maybe servers from large content providers
could also provide data about this...

Regards,
  Roland

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 29 Mar 2021 10:48:52 +0100

If you have good deployment data on devices with ECN enabled by default, 
then please share it.

It would be extremely helpful to also know if the home routers you 
mention were deployed using FQ.

Gorry

---

From: Pete Heist <[email-masked]>
Date: Tue, 30 Mar 2021 12:56:59 +0200

...

> preferred approach for routers that are not involved in ECN

Just to clarify, although I think it *would* be the right approach to
deprecate RFC3168 in order to redefine its semantics, I don't
personally support doing that. Its benefits to typical Internet traffic
may not be dramatic, but on the other hand it is fully compatible with
non-ECN traffic, and avoids some packet loss and retransmissions, while
serving as a reliable base to build upon.

I also think recent discussion suggests that there is enough support
for the integrity of RFC3168 that traffic from new proposals, if not
guarded by a DSCP, must be fully compatible with existing deployed 3168
middleboxes, effectively including those with single queue AQMs.

> ECT(1) input cs. output signal discussion. 

> Now, if we come up

> could also provide data about this...

That's also what we saw in the ISP study at the gateway, that ECE is a
more reliable indicator of AQM deployment than CE:

https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-02.html#name-tcp-initiated-from-lan-to-w

Pete

> > > and
> > > semantics
> > > are

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 30 Mar 2021 12:15:46 +0100

Please see below:

> https://www.ietf.org/archive/id/draft-heist-tsvwg-ecn-deployment-observations-02.html#name-tcp-initiated-from-lan-to-w

Thanks, I think would agree on your summary of the "non dramatic" 
benefits for existing using the currently standardised RFC-3168 marking, 
and hopefully what you say is not far from the summary we arrived at in 
RFC 8087.

I see arguments in recent discussions for retaining backwards 
compatibility with this and also arguments stressing other aspects.

Gorry

---

From: Pete Heist <[email-masked]>
Date: Tue, 30 Mar 2021 21:43:07 +0200

More comments below...

On Tue, 2021-03-30 at 12:15 +0100, Gorry Fairhurst wrote:
> > > about RFC3168 (non-)deployment before deprecating RFC3168. Even
> > > RFC
> > > 8311
> > > > Before we do that though, we should make sure that the current
> > > > CE
> > > > is
> > > > not actually useful. Figure 5 in this paper suggests some
> > > > benefit
> > > > to
> > > > example during capacity changes or flow introductions. It
> > > > wouldn't
> > > > be
> > > > ideal to deprecate RFC3168, only to find out that another bit
> > > > of
> > > > signal
> > > > Internet paths there. Prior to that we heard 5% elsewhere.
> > > > Whatever
> > > > the
> > > > ECT(0) and ECT(1). If you introduce traffic that backs off much
> > > > less
> > > > in

RFC8087 lists more benefits for RFC3168 than I did. What I also
appreciate about 3168 is that its compatibility with non-ECN traffic
makes it readily deployable, which is good engineering.

> compatibility with this and also arguments stressing other aspects.

That does seem to describe the situation. It also suggests that when
there are arguments "for and against" backwards compatibility, rough
consensus on any consequential change to RFC3168 is likely to be
difficult. If so, it will remain a proposed standard, deployments will
continue, and full 3168 compatibility remains important.

Pete

> > > > > the few networks where it does can be safely ignored, or
> > > > > cleaned
> > > > > out,
> > > > > or be expected to take proactive measures to protect
> > > > > themselves,
> > > > > which
> > > > > patch releases to enable (some of) the mitigation measures
> > > > > detailed
> > > > > in
> > > > > -l4ops-02 Sec. 5.
> > > > > 
> > > > > If that is the WG consensus, then I *strongly urge* the WG to
> > > > > do
> > > > > the
> > > > > following:
> > > > > 
> > > > > subject of investigation. Since we are certain that any
> > > > > remaining
> > > > > RFC
> > > > > 3168 deployments can be safely ignored, then ECT/CE-a/CE-b
> > > > > can be
> > > > > used
> > > > > as unambiguous signals to steer packets into a low-latency
> > > > > queue,
> > > > > if
> > > > > desired.
> > > > > 
> > > > > The semantics of CE-a and CE-b for the second set of
> > > > > experiments
> > > > > would
> > > > > be as follows:
> > > > > 
> > > > > - CE-a: "Decelerate"
> > > > > - CE-b: "Accelerate"
> > > > > 
> > > > > queue not in "Accelerate" state would be re-marked to either
> > > > > CE-a
> > > > > or
> > > > > ECT. Any packet discard (or detection of high delay
> > > > > variation?)
> > > > > must
> > > > > disable the transport's "Accelerate" mechanism for some
> > > > > interval
> > > > > and
> > > > > The differences between these two sets of experiments hinge
> > > > > on
> > > > > whether
> > > > > failed after two decades, we probably only get one more
> > > > > chance to
> > > > > get
> > > > > ignored, I suggest choosing bit assignments for the four
> > > > > signals
> > > > > that
> > > > > induce maximum pain in the obstinate minority that might
> > > > > still
> > > > > deploy
> > > > > RFC 3168 ECN.
> > > > > 
> > > > > deployments can be safely ignored, then I *strongly urge* the
> > > > > WG
> > > > > *to

---

From: daihuichen <[email-masked]>
Date: Wed, 31 Mar 2021 03:15:22 +0000

Hello,

The discussion in this thread, as well as the discussion in the thread of L4S adoption call, seems only consider the Internet scenario. 
Today, ECN has been widely deployed in data center networks for low-latency transports like RoCE, I think we should also consider the results to data center networks if deprecating RFC 3168.
I don’t have a proposal, just a reminder that we cannot leave data center networks out.

Regards,
/Huichen

-----Original Message-----
From: tsvwg [mailto:[email-masked]] On Behalf Of Steven Blake
Sent: 2021年3月27日 1:02 AM
To: [email-masked]
Subject: [tsvwg] Deprecating RFC 3168 for future ECN experimentation

A lot (not all) of the recent arguments revolve around the assumption by some that RFC 3168 ECN deployment barely exists in the Internet, and the few networks where it does can be safely ignored, or cleaned out, or be expected to take proactive measures to protect themselves, which may in practice require them to lobby their router vendors to spin patch releases to enable (some of) the mitigation measures detailed in
-l4ops-02 Sec. 5.

- Not-ECT
- ECT
- CE-a
- CE-b

This second step would allow for two sets of experiments. The semantics of CE-a and CE-b for the first set of experiments would be as follows:

- CE-a: "Decelerate"

The exact behavior elicited by the "Decelerate" signal would be the subject of investigation. Since we are certain that any remaining RFC
3168 deployments can be safely ignored, then ECT/CE-a/CE-b can be used as unambiguous signals to steer packets into a low-latency queue, if desired.

The semantics of CE-a and CE-b for the second set of experiments would be as follows:

- CE-a: "Decelerate"
- CE-b: "Accelerate"

An aggressive fraction (100%?) of CE-b marked packets traversing a queue not in "Accelerate" state would be re-marked to either CE-a or ECT. Any packet discard (or detection of high delay variation?) must disable the transport's "Accelerate" mechanism for some interval and should cause the transport to revert to "TCP-friendly" behavior for some (different?) interval. The exact behaviors of "Accelerate" and "Decelerate" signals would be the subject of investigation. Again, since we are certain that any remaining RFC 3168 deployments can be safely ignored, then ECT/CE-a/CE-b can be used as unambiguous signals to steer packets into a low-latency queue.

The differences between these two sets of experiments hinge on whether there is more utility in an "Accelerate" signal coupled with a "Decelerate" signal, or with two separate levels of "Decelerate"
signals. Since it is WG consensus that the RFC 3168 ECN experiment failed after two decades, we probably only get one more chance to get this right, so careful and exhaustive experimentation which explores the design space is in order.

Obviously, both sets of experiments cannot be run simultaneously on intersecting parts of the Internet. I leave the options for safely isolating these experiments as an exercise for the reader. Since we are certain that any remaining RFC 3168 ECN deployments can be safely ignored, I suggest choosing bit assignments for the four signals that induce maximum pain in the obstinate minority that might still deploy RFC 3168 ECN.

Now, *if it is not WG consensus* that any existing RFC 3168 ECN deployments can be safely ignored, then I *strongly urge* the WG *to not adopt* experimental proposals that place burden and/or risk on networks that have deployed it.

Regards,

// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 31 Mar 2021 06:19:02 +0300

> On 31 Mar, 2021, at 6:15 am, daihuichen <[email-masked]> wrote:
> 
> Today, ECN has been widely deployed in data center networks…

May I ask, purely out of idle curiosity, whether those deployments are primarily of the single-queue variety, or of the flow-isolating type?

 - Jonathan Morton

---

From: daihuichen <[email-masked]>
Date: Wed, 31 Mar 2021 03:38:01 +0000

Jonathan,

Only the queue for low-latency traffic, e.g., roce packets, is ECN enabled.
The rest queues are usually FIFOs with tail-drop policy. 

/Huichen

-----Original Message-----
From: Jonathan Morton [mailto:[email-masked]] 
Sent: 2021年3月31日 11:19 AM
To: daihuichen <[email-masked]>
Cc: Steven Blake <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] Deprecating RFC 3168 for future ECN experimentation

 - Jonathan Morton

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Wed, 31 Mar 2021 05:05:55 -0700 (PDT)

Huichen,

What type of queues are used for the low-latency ROCE traffic?

> Sent: 2021?3?31? 11:19 AM
> > Today, ECN has been widely deployed in data center networks?
> 
-- 
Rod Grimes                                                 [email-masked]

---

From: daihuichen <[email-masked]>
Date: Wed, 31 Mar 2021 12:20:59 +0000

Rod,

Mostly RED/ECN queues with very small marking thresholds, e.g., Kmin=5KB and Kmax=120KB, the exact setting differs from algorithm to algorithm. 
The purpose is to achieve very high link utilization (close to 100%) and at the same time maintains small queue size to minimize queueing delay.
The RoCE clusters in cloud have been using RED/ECN for years, and at scale.

A new algorithm even propose to use WRED to speed up the first RTT transmission rate.
https://dl.acm.org/doi/10.1145/3387514.3405878

Regards,
/Huichen

-----Original Message-----
From: Rodney W. Grimes [mailto:[email-masked]] 
Sent: 2021年3月31日 8:06 PM
To: daihuichen <[email-masked]>
Cc: Jonathan Morton <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] Deprecating RFC 3168 for future ECN experimentation

Huichen,