Subject: [tsvwg] Neal Cardwell's rationale for supporting ECT(1) as an input/L4S signal

From: Neal Cardwell <[email-masked]>
Date: Fri, 8 May 2020 11:19:57 -0400

As part of the discussion around the consensus call on ECT(1), I wanted to
post my personal rationale for supporting ECT(1) as an input/classification
signal, along the lines of the L4S proposals.

This is my personal opinion; I'm not speaking for any company or
institution.

Many of these points will echo the excellent points that were made at the
TSVWG interim on April 27, by Greg White, Bob Briscoe, Jana Iyengar, Stuart
Cheshire, and Andrew McGregor; I agree with all of the points they made.
But for the sake of discussion and debate I wanted to amplify what they
said and enumerate the particular considerations that are most compelling
to me.

Here they are:

- SCE is basically an extension of RFC3168 ECN: senders mark their packets
ECT(0), and respond to CE like RFC 3168 says they should, by halving cwnd.
This would not work well for large sites with a substantial installed base
of shallow-threshold (DCTCP-style) ECN: if these sites marked their traffic
with ECT(0) to try to use SCE, their local switches configured for
shallow-threshold ECN would make CE marks at low queue occupancies, which
would cause large RFC3681-style reductions in cwnd, which would cause
underutilization. To the best of my knowledge this applies to several large
Internet services.

- Much of the installed base of switch hardware around the world can
already be configured to signal shallow-threshold congestion with
DCTCP/L4S-style CE marks. Many have CE-marking mechanisms baked into
hardware, and would need to be replaced in order to deploy SCE.

- Encapsulation/decapsulation is a widely prevalent and important
technology today in production networks. With the installed base of
encap/decap mechanisms, it is likely that for many implementations any SCE
marking applied to packets would just get stripped off with the outer
header when decapsulated.

- To get low latency in the public Internet, we need multiple queues. With
SCE AFAICT there's not yet a compelling story about how to classify traffic
to maintain a low-delay queue and  a classic queue, since there is no ECN
code point to identify the low-delay traffic. AFAICT with SCE you likely
must either (a) have a single high-delay queue with a jumble of Reno,
CUBIC, classic RFC 3168, and SCE-respecting RFC 3168 senders, (b)  use fair
queuing (tricky in hardware), (c) use DSCP to classify (academically pure
but unrealistic across multiple ASes in the public Internet), (d) use a
dual-queue system that heuristically classifies all flows into the
low-delay queue by default but kicks them out using a queue-protection
algorithm.

- SCE seems to involve an ecosystem with a more complex and more
experimental CC (with two different kinds of ECN signal) and little
real-world/production experience yet.  L4S seems to involve an ecosystem
that provides a queue that  is basically a single-threshold,
shallow-threshold, DCTCP-style, ECN ecosystem, which is simpler and for
which the world has a lot of accumulated academic research and
real-world/production experience over the last decade.

- L4S flows potentially causing unfairness in RFC3168 ECN bottlenecks has
been mentioned as a potential concern. However, a robust RFC3168 ECN
bottleneck should already have a mechanism to avoid unfairness caused by
flows that are marked as ECT(0|1) and yet not performing RFC3168 responses.
In particular, many of the large sources of known deployments of  RFC3168
--  Linux fq_codel and cake -- are already deployed with fair queueing. In
such bottlenecks L4S traffic should not cause harm to other non-L4S flows.
Furthermore, if there really are ISPs with deployments of RFC3168
bottlenecks that have neither FQ nor any other protection from
non-RFC3168-ECT(1) flows, then they can bleach incoming ECT(1) code points
to Not-ECT and treat L4S as Not-ECT (ISPs typically already transform the
DSCP byte at their ingress anyway). So I do not see harm to RFC3168 ECN
bottlenecks as a prohibitive concern.

- More generally, if there is any problem discovered with the L4S
experiment, either the algorithm or particular implementations, bottlenecks
can easily identify L4S traffic and bleach it into Not-ECT, and treat it
like Reno/CUBIC traffic.

Best regards,
Neal

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 8 May 2020 18:10:47 +0100

> underutilization. To the best of my knowledge this applies to several large

Wait, that doesn't match my understanding.  In such an example site
using SCE I would expect ECT(0) marking only at "large" queue
occupancies (and resulting in cwnd halving when seen), but more
prevalently ECT(1) marking at "small" queue occupancies - resulting
in some smaller percentage (than 50) decrease in cwnd when seen.

What did I miss?

-- 
Cheers,
  Jeremy

---

From: Roland Bless <[email-masked]>
Date: Fri, 8 May 2020 19:12:27 +0200

Hi Neal and all,

see inline.

On 08.05.20 at 17:19 Neal Cardwell wrote:
> As part of the discussion around the consensus call on ECT(1), I
> wanted to post my personal rationale for supporting ECT(1) as an
> input/classification signal, along the lines of the L4S proposals.
>
> Many of these points will echo the excellent points that were made at
> the TSVWG interim on April 27, by Greg White, Bob Briscoe, Jana
> Iyengar, Stuart Cheshire, and Andrew McGregor; I agree with all of the
> points they made. But for the sake of discussion and debate I wanted
> to amplify what they said and enumerate the particular considerations
> that are most compelling to me.

I had the feeling that this was more a discussion around L4S vs. SCE
proposals and not about the
ECT(1) codepoint in question. Several arguments were along the lines of
"L4S is more advanced
and needs to be finished as it already took too long". I don't think
that this is a good argument as
the L4S proposal could also use ECT(1) as output and use a DSCP as
classifier. See below.

>
> Here they are:
>
> - SCE is basically an extension of RFC3168 ECN: senders mark their
> packets ECT(0), and respond to CE like RFC 3168 says they should, by
> halving cwnd. This would not work well for large sites with a
> substantial installed base of shallow-threshold (DCTCP-style) ECN: if
> these sites marked their traffic with ECT(0) to try to use SCE, their
> local switches configured for shallow-threshold ECN would make CE
> marks at low queue occupancies, which would cause large RFC3681-style
> reductions in cwnd, which would cause underutilization. To the best of
> my knowledge this applies to several large Internet services.

Let's leave the classifier issue aside for a moment:
using ECT(1) as mark for "shallow-threshold exceeded (DCTCP-style)"
would make it
an unambiguous congestion signal. CE would still mean: use RFC3681-style
reduction.
These could even be mixed along a path (CE after ECT(1), but CE will
stay CE).
Sure: shallow-threshold switches need to use ECT(1) instead of CE then;
I have no clue how difficult this would be to change in the existing
deployed gear.
So the situation you are describing is caused by the semantic overload
for CE used for both types of marking (RFC3681-style and DCTCP-style).
The current solution is to distinguish the semantics by context (e.g.,
controlled environment/pure in-DC use), which doesn't allow DCTCP-style
marking and RFC3681-style marking for an
end-to-end connection at the same time, but using ECT(1) as output would
allow this.

> hardware, and would need to be replaced in order to deploy SCE.
The question is: could they easily support using ECT(1) instead of CE
for the shallow-threshold marking?
Supporting L4S's dual queue and classification by ECT(1) would IMHO
require even more extensive hardware changes.
> encap/decap mechanisms, it is likely that for many implementations any
> SCE marking applied to packets would just get stripped off with the
> outer header when decapsulated.
>
I'm not sure about this one. L4S could be affected equally or do you
think that CE is handled differently? I wasn't able to find a quick
answer by looking at
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-encap-guidelines-13
> - To get low latency in the public Internet, we need multiple queues.
> With SCE AFAICT there's not yet a compelling story about how to
> classify traffic to maintain a low-delay queue and  a classic queue,
> since there is no ECN code point to identify the low-delay traffic.
> AFAICT with SCE you likely must either (a) have a single high-delay
> queue with a jumble of Reno, CUBIC, classic RFC 3168, and
> SCE-respecting RFC 3168 senders, (b)  use fair queuing (tricky in
> hardware), (c) use DSCP to classify (academically pure but unrealistic
> across multiple ASes in the public Internet), (d) use a dual-queue
> system that heuristically classifies all flows into the low-delay
> queue by default but kicks them out using a queue-protection algorithm.

I agree here and queue separation would also be essential for SCE. But
using a DSCP for classification would
be the more extensible solution as ECT(1) wouldn't always mean to use
the L4S queue semantics. Instead both
solutions could use the ECT(1) mark with nearly the same congestion
signal semantics. Moreover, both solutions have
problems with "misbehaving" flows in the wrong queue.

> real-world/production experience yet..  L4S seems to involve an
> ecosystem that provides a queue that  is basically a single-threshold,
> shallow-threshold, DCTCP-style, ECN ecosystem, which is simpler and
> for which the world has a lot of accumulated academic research and
> - L4S flows potentially causing unfairness in RFC3168 ECN bottlenecks
> has been mentioned as a potential concern. However, a robust RFC3168
> ECN bottleneck should already have a mechanism to avoid unfairness
> caused by flows that are marked as ECT(0|1) and yet not performing
> RFC3168 responses. In particular, many of the large sources of known
> deployments of  RFC3168 --  Linux fq_codel and cake -- are already
> deployed with fair queueing. In such bottlenecks L4S traffic should
> not cause harm to other non-L4S flows. Furthermore, if there really
> are ISPs with deployments of RFC3168 bottlenecks that have neither FQ
> nor any other protection from non-RFC3168-ECT(1) flows, then they can
> bleach incoming ECT(1) code points to Not-ECT and treat L4S as Not-ECT
> (ISPs typically already transform the DSCP byte at their ingress
> anyway). So I do not see harm to RFC3168 ECN bottlenecks as a
> prohibitive concern.
>
> experiment, either the algorithm or particular implementations,
> bottlenecks can easily identify L4S traffic and bleach it into
> Not-ECT, and treat it like Reno/CUBIC traffic.
>
> Best regards,
> Neal
>
Regards,
 Roland

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 8 May 2020 20:53:03 +0300

> in some smaller percentage (than 50) decrease in cwnd when seen.

I think there are two points to clear up here.

1: SCE proposes keeping the existing meanings of ECT(0) (ie. ECN Capable Transport) and CE (ie. Congestion Experienced).  At an SCE AQM, CE marks would be expected at relatively deep queuing, and SCE marks - ECT(1) - would be applied at the shallower depth.  The AQM would not "introduce" ECT(0) marks; that codepoint is set at the packet's origin.

2: Neal is referring to an existing installation which signals congestion only with CE.  SCE would react to that in the same way as existing ECN transports.  In particular, SCE does not introduce any special problems with such deployments that do not already exist with RFC-3168.

The shallow queues referred to are intended for deployment in tightly controlled datacentre environments, where the typical path RTT is measured in microseconds rather than milliseconds.  It would not be surprising if a transport passing through such a queue as part of a typical Internet path of many milliseconds experienced poor utilisation.  But that, I think, should be taken as a misconfiguration of the network - one which SCE deals with identically to existing ECN transports.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 8 May 2020 22:02:22 +0000

Hi Neal,

Thanks for posting this, it’s a helpful explanation.

A couple of points I wanted to respond to:

From: Neal Cardwell <ncardwell=[email-masked]>
> - Much of the installed base of switch hardware around the world can already be configured to signal shallow-threshold congestion with DCTCP/L4S-style CE marks. Many have CE-marking mechanisms baked into hardware, and would need to be replaced in order to deploy SCE.

I wanted to check to make sure I understand:
To what extent do you think it's an L4S goal to accommodate the
existing installed base of switch hardware that’s currently in use
for DCTCP in datacenters?

I thought the existing deployments generally wouldn’t be compliant
L4S-compatible dualq devices suitable for general internet traffic
anyway, and would continue to need traffic isolation the way they do
now.  Is that different from your understanding?

> - More generally, if there is any problem discovered with the L4S experiment, either the algorithm or particular implementations, bottlenecks can easily identify L4S traffic and bleach it into Not-ECT, and treat it like Reno/CUBIC traffic.

To repeat in what's maybe a better thread for it:

I agree that bleaching is an option where there’s RFC 3168 trouble,
that’s a good point and thanks for mentioning it.

I’d consider it a poor choice to land there if there's other options,
because it works against the success of existing or under-way
deployments of RFC 3168 queues by imposing a new bleaching requirement
that may not be trivial for all networks with marking queues

It would also be an unfortunate choice to endorse bleaching because it
would lose much of the potential value in the code point, as David
recently mentioned.

But I do agree it would work, and that it’s a worthwhile mitigation
that should be mentioned in the L4S docs if we end up not finding a
way to support robust 3168 compatibility.

> - Encapsulation/decapsulation is a widely prevalent and important technology today in production networks. With the installed base of encap/decap mechanisms, it is likely that for many implementations any SCE marking applied to packets would just get stripped off with the outer header when decapsulated.

I'd be interested in more information about this.  Is there anything
publicly known about the deployment footprint of different tunnel
implementations in the places likely to deploy L4S dualqs, and
whether they're managed by the same entities who control the queues?

I'm not sure it changes my position much, but it seems at least
useful to know how big a lift it would be to do a tunnel
decapsulation change, if that's a necessary piece of a safe
approach to L4S that can achieve low latency effectively.

(I know Bob gave a long list of tunneling protocols not long ago, but
it seems like only some of them are likely to be relevant to the L4S
question, and probably only in a reasonably specific set of
implementations for at least the early stages...)

Thanks and regards,
Jake

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 9 May 2020 12:52:58 +0200

Hi Neal,

> On May 8, 2020, at 17:19, Neal Cardwell <ncardwell=[email-masked]> wrote:
> [...]
> 
> - SCE seems to involve an ecosystem with a more complex and more experimental CC (with two different kinds of ECN signal) and little real-world/production experience yet..  L4S seems to involve an ecosystem that provides a queue that  is basically a single-threshold, shallow-threshold, DCTCP-style, ECN ecosystem, which is simpler and for which the world has a lot of accumulated academic research and real-world/production experience over the last decade.

	[SM] Interestingly, I take it as a considerable downside that in a decade of work L4S has not managed to come up with robust and reliable solutions to its challenges. "Too little, too late", comes to mind as much as "robust solution after years of diligent engineering", but which one it is is still an open question.
One more downside of the long-winding development is that the change of reference protocol from DCTCP to TCP Prague basically devalues the old DCTCPs measurements as proof of safety.
	My point is, it seems odd, using indirect measures like accumulated development time and magnitude of conducted tests as proxies for the quality of L4S instead of actually looking closely into the RFCs and compare their claims with the existing data. I am not saying that my assessment of L4S' implementation not being close to its promises is the only conclusion one can come to, but I would hope that everybody chiming into this consensus questions actually takes the time to look at that closely for themselves. It is easy to promise the sky, delivery & execution however...

> - L4S flows potentially causing unfairness in RFC3168 ECN bottlenecks has been mentioned as a potential concern. However, a robust RFC3168 ECN bottleneck should already have a mechanism to avoid unfairness caused by flows that are marked as ECT(0|1) and yet not performing RFC3168 responses.

	[SM] That essentially declares all non-FQ AQMs to be fair game, no? Because if they wanted better isolation they could get it (at a cost). That seems at odds with the extra mile L4S goes to avoid using FQ solutions even for a problem that is exceptionally well suited for FQ. Because that can easily be turned around, why not demand the same level of robustness from L4S instead, it being the newcomer and all? Say, require L4S to monitor flow behavior and make its classification based on observed behavior instead of a simple assertion by the sender (ECT(1) is nothing more than that, it is at best a classification on intent, while the thing that should be classified is behavior.) In the context of another thread it seems clear that pure intent signaling is actually expected to be abused:

To cite Tom Herbert paraphrasing Joe Touch  from ([tsvwg] Comment on draft-ietf-tsvwg-transport-encrypt-13)
"It was not previously mentioned in the context of extension headers.
This is a general consideration for any unauthenticated plaintext data
in a packet that an intermediate node chooses to consume. As Joe said,
in the absence of any requirement or contract, it's the prerogative of
the host to manipulate packet contents as it sees fit to gain an
advantage (where sometimes the "advantage" is just that packets get
delivered and not dropped)."

While I do not fully agree that every sender rightfully should try to abuse the network at all costs, I accept that the potential is there and solutions need to take this into account in their threat modeling (and IMHO L4S has not done so sufficiently, simply claiming without supporting evidence that ECT(1) can not be abused is either naively optimistic or intentionally misguided).

> In particular, many of the large sources of known deployments of  RFC3168 --  Linux fq_codel and cake -- are already deployed with fair queueing. In such bottlenecks L4S traffic should not cause harm to other non-L4S flows.

	[SM] Mmmh, that requires active defenses by existing network to accommodate a newcomer; sure it might ameliorate the fall-out from L4S, that would be akin to haphazardly handle flu virus samples in a public kitchen instead of a S3-bio safety laboratory, as the population should be vaccinated and hence immune already, so what is the harm? That seems not like a great idea IMHO, in neither case.

> Furthermore, if there really are ISPs with deployments of RFC3168 bottlenecks that have neither FQ nor any other protection from non-RFC3168-ECT(1) flows, then they can bleach incoming ECT(1) code points to Not-ECT and treat L4S as Not-ECT (ISPs typically already transform the DSCP byte at their ingress anyway). So I do not see harm to RFC3168 ECN bottlenecks as a prohibitive concern.
> 

	[SM] At that point L4S regresses into a relative boring pie-derivative (albeit with decreased burst-tolerance*) single queue AQM at those nodes where the dual queue coupled AQM was deployed, sure getting rid of the imprecise/unsafe coupling is going to be a win, but having to spend the last ECN codepoint seems a rather steep cost to get to such pedestrian a result, no?
	More importantly, why not first do the due diligence research to assess the probability of this outcome for the L4S experiment first, before roll-out/elevation to experimental RFC status?

Best Regards
	Sebastian

*) Not the best design when operating in drop only mode for Reno/CUBIC style CCs, no?

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 12 May 2020 15:47:30 -0400

> in some smaller percentage (than 50) decrease in cwnd when seen.

Sorry if my scenario was not clearly expressed. In the scenario I'm
talking about here, there is a substantial installed base of
shallow-threshold (DCTCP-style) ECN switches in the datacenter in
which a server/sender sits. Those switches have hardware that only
knows how to mark with CE marks, and the ECN marking thresholds are
set to be shallow, because that's what allows nice, low latency for
DCTCP-style algorithms, so that's how they've been running for years.
(And the switch hardware is unable to mark with ECT(1) as a congestion
signal.)

If a server/sender in such a context attempts to participate in an SCE
conversation with a client/receiver on the public Internet, there is
the likelihood that shallow queues in the local datacenter bottlenecks
will result in the local switches making CE marks that will be
reflected by the receiver and interpreted as requiring a maximum-sized
cwnd cut (50% for Reno, 30% for CUBIC). That would cause poor
throughput.

That's the basic concern for this scenario.

best,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 12 May 2020 16:16:29 -0400

Hi Roland,

Please see some thoughts in-line below...

> the L4S proposal could also use ECT(1) as output and use a DSCP as

I think it's overly optimistic to think that DSCP could be used as an
end-to-end classifier for a new congestion control class across the
public Internet. Rewriting DSCP bits at ingress into an AS is a
ubiquitous mechanism, and such an undertaking would require hundreds
or thousands of ASes to rearrange their custom internal DSCP/QoS
architecture in compatible ways. And there is a combinatorial
explosion if an AS needs to deal with each of its existing DSCP/QoS
levels needing to support both the new congestion control class and
classic congestion controls. This doesn't seem like a realistic path
to me.

> for the shallow-threshold marking?

I suspect that in most high-speed switches and routers, the
functionality to use CE for ECN markings is etched into ASIC hardware,
and could not be changed.

> require even more extensive hardware changes.

Yes. But dual-queue is not a required component of L4S inside a
datacenter. So the existing installed base of datacenter switches
would not need to be changed at all to participate in L4S. Those
switches are designed to mark any ECT(0) or ECT(1) packets with CE
when the queue passes a shallow threshold. That works for their
current use case, and it works for L4S. No changes would be needed, to
hardware or firmware.

> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-encap-guidelines-13

Here my understanding was based on a description of this issue by Bob
Briscoe. I'm not sure what the original data source is for that issue.
(And apologies if I have misinterpreted the issue.)

> problems with "misbehaving" flows in the wrong queue.

As mentioned above, I think using DSCP for congestion control
classification, while theoretically clean and extensible, is not a
realistic/feasible option.

Best regards,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 12 May 2020 16:29:46 -0400

On Fri, May 8, 2020 at 1:54 PM Jonathan Morton <[email-masked]> wrote:
> The shallow queues referred to are intended for deployment in tightly
> controlled datacentre environments, where the typical path RTT is measured in
> microseconds rather than milliseconds. It would not be surprising if a
> transport passing through such a queue as part of a typical Internet path of
> many milliseconds experienced poor utilisation.

I think the issue is that such paths do not currently suffer from
throughput problems, but could if they attempt to participate in an
SCE conversation.

> But that, I think, should be
> taken as a misconfiguration of the network - one which SCE deals with
> identically to existing ECN transports.

I think that depends on what is included in "existing ECN transports".
If we include ECN transports like RFC8257/DCTCP, then SCE deals with
this scenario differently than such transports.

neal

---

From: Jeremy Harris <[email-masked]>
Date: Tue, 12 May 2020 21:38:38 +0100

> throughput problems, but could if they attempt to participate in an

They would if they tried to participate in a plain 3168 conversation,
right?  I don't see you can dislike SCE for that.

How does it behave for L4S, without changing the datacentre hardware?
-- 
Cheers,
  Jeremy

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 12 May 2020 23:38:59 +0300

> On 12 May, 2020, at 11:29 pm, Neal Cardwell <[email-masked]> wrote:
> 

Except that they *do* already suffer from throughput problems if they attempt to participate in an RFC-3168 ECN conversation, to precisely the same extent, and for precisely the same reasons.

 - Jonathan Morton

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 12 May 2020 16:55:22 -0400

> From: Neal Cardwell <ncardwell=[email-masked]>
..
> for DCTCP in datacenters?

I'm not sure if accommodating the existing installed base of switch
hardware was an explicit goal of L4S or not, but it is a nice property
that seems to help greatly in making L4S easier to deploy than it
might otherwise be.

> now.  Is that different from your understanding?

My understanding is that dualq is not a required component of
implementing L4S, and definitely would not be required at every hop or
potential bottleneck along the network path. My understanding is that
there would be sites that don't want to change the qdiscs on their
senders/servers, and don't want to change their datacenter switches,
but would like their connections over the public Internet to be able
to use L4S.

> >   bottlenecks can easily identify L4S traffic and bleach it into Not-ECT,
> >   and treat it like Reno/CUBIC traffic.
>

Yes, I agree with all those points; bleaching is very far from ideal.

> >   SCE marking applied to packets would just get stripped off with the outer

As I mentioned with Roland, here my understanding was based on a
description of this issue by Bob Briscoe. I'm not sure what the
original data source is for that issue. (And apologies if I have
misinterpreted the issue.) But perhaps Bob will have time to fill in
more of the details or pointers.

Best regards,
neal

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 12 May 2020 17:30:48 -0400

Hi Sebastian,

Some thoughts in-line below...

On Sat, May 9, 2020 at 6:53 AM Sebastian Moeller <[email-masked]> wrote:
> >   real-world/production experience yet.. L4S seems to involve an ecosystem
> [SM] Interestingly, I take it as a considerable downside that in a decade of
> work L4S has not managed to come up with robust and reliable solutions to
> its challenges. "Too little, too late", comes to mind as much as "robust
> solution after years of diligent engineering", but which one it is is still
> an open question.
>
> One more downside of the long-winding development is that the change of
> reference protocol from DCTCP to TCP Prague basically devalues the old DCTCPs
> measurements as proof of safety.
<
> My point is, it seems odd, using indirect measures like accumulated
> development time and magnitude of conducted tests as proxies for the quality
> of L4S instead of actually looking closely into the RFCs and compare their
> claims with the existing data. I am not saying that my assessment of L4S'
> implementation not being close to its promises is the only conclusion one
> can come to, but I would hope that everybody chiming into this consensus
> questions actually takes the time to look at that closely for themselves. It
> is easy to promise the sky, delivery & execution however...

Both L4S and SCE have algorithms and implementations that are works in
progress, and not set in stone at this point. Since they are works in
progress, I think it's worthwhile to focus on the core question we are
facing here, which is about the interpretation of the ECT(1) code
point. I think it's useful to distinguish between what is inherent in
the interpretation of the code point from what is incidental in the
current algorithms/implementations on either side.

> [SM] That essentially declares all non-FQ AQMs to be fair game, no?

No, there are ways to deal with abusive flows that do not require fair queuing.

>  Because
>  if they wanted better isolation they could get it (at a cost). That seems at
>  odds with the extra mile L4S goes to avoid using FQ solutions even for a
>  problem that is exceptionally well suited for FQ. Because that can easily be
>  turned around, why not demand the same level of robustness from L4S instead,
>  it being the newcomer and all? Say, require L4S to monitor flow behavior and
>  make its classification based on observed behavior instead of a simple
>  assertion by the sender (ECT(1) is nothing more than that, it is at best a
>  classification on intent, while the thing that should be classified is
>  behavior.) In the context of another thread it seems clear that pure intent
>  signaling is actually expected to be abused:
...
> While I do not fully agree that every sender rightfully should try to abuse
> the network at all costs, I accept that the potential is there and solutions
> need to take this into account in their threat modeling (and IMHO L4S has not
> done so sufficiently, simply claiming without supporting evidence that ECT(1)
> can not be abused is either naively optimistic or intentionally misguided).

L4S does not claim that ECT(1) cannot be abused. Rather, it has a
rather well-developed story for detecting and dealing with abuse of
the ECT(1) code point with queue protection algorithms. Please see:

  https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-04#section-8.2
  https://tools.ietf.org/html/draft-briscoe-docsis-q-protection-00

> > In particular, many of the large sources of known deployments of RFC3168 --
> > Linux fq_codel and cake -- are already deployed with fair queueing. In such
> > bottlenecks L4S traffic should not cause harm to other non-L4S flows.
>
> [SM] Mmmh, that requires active defenses by existing network to
> accommodate a newcomer...

It's not perfect, but we can't let the perfect be the enemy of the
good, and need to evaluate all the trade-offs of the alternatives
holistically.

Best regards,
neal

---

From: Steven Blake <[email-masked]>
Date: Tue, 12 May 2020 17:47:56 -0400

On Tue, 2020-05-12 at 23:38 +0300, Jonathan Morton wrote:
> Except that they *do* already suffer from throughput problems if they
> attempt to participate in an RFC-3168 ECN conversation, to precisely
> the same extent, and for precisely the same reasons.

In (the vast majority of) datacenter switches, all queues are shallow
(10s of microseconds) relative to typical Internet RTTs (tens of
milliseconds). It is completely orthogonal to any AQM or ECN scheme.

If you want your Internet-facing servers buried within a datacenter
network, you are well advised to carve out a BW capacity slice for that
traffic sufficient to avoid congestion.

Regards,

// Steve

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 13 May 2020 00:04:01 +0200

Hi Neal,

thanks for your input, more below in-line.

> On May 12, 2020, at 23:30, Neal Cardwell <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] That would have been an interesting discussion, but as far as I can tell a discussion we did not have.

> No, there are ways to deal with abusive flows that do not require fair queuing.

	[SM] Could you please elaborate on this? For example L4S advisory queue protection scheme basically is just a bad implementation of fair queuing... combining most of the cost with few of the upsides (for example, it only triggers after the fact and packets of the offending flow already in the LL-queue stay there, so if I can change the 5-tuple of an offendin flow intended to disturb LL-services often enough, queue protection will constantly run behind me).

> L4S does not claim that ECT(1) cannot be abused.

	[SM] It is missing a realistic discussion about how it wants to deal with t

> Rather, it has a

	[SM] "Such a queue protection function is not considered a necessary part
   of the L4S architecture, which works without it (in a similar way to
   how the Internet works without per-flow rate policing)."

Yeah that might be a solution, but being purely advisory is not going to cut it, the main rationale seems to be wishful thinking instead:
" It is
   hoped that self-interest and standardisation of dynamic behaviour
   (cf.  TCP slow-start) will be sufficient to prevent transports from
   sending excessive bursts of L4S traffic, given the application's own
   latency will suffer most from such behaviour.

Whether burst policing becomes necessary remains to be seen.  Without
   it, there will be potential for attacks on the low latency of the L4S
   service.  However it may only be necessary to apply such policing
   reactively, e.g. punitively targeted at any deployments of new bursty
   malware."

It is not "claimed", but "hoped" that it is not abused, not a sign of safe engineering, as anything that can be abused will be abused.

>  https://tools.ietf.org/html/draft-briscoe-docsis-q-protection-00

	[SM] I have a) read this, and argued that that basically drags in L4-header inspection and keeping (limited) per flow state, if we are willing to pay this price we are actually better of going FQ all the way IMHO, because that would solve a number of warts in the L4S design. 
And b) I have noted that that is not really a hindrance for abuse, it will only mildly push back against totally barren attempts to take over the LL-queue too hastily. But if the whole goal is disruption of the LL-queue it makes my limited burst DOS, auto-homing on my victims probably most latency sensitive flows, and as long as my attack is doing its deed not by continuous high rates, but simply by being sufficiently bursty I bet I can do lots of damage on the nominal LL side of L4S without triggering any circuit breakers or queue protection. 

As I mentioned before, I seems that the L4S design has not been tested with many adversarial traffic patterns yet, which after a decade of development is an odd thing to observe, no?

> good, and need to evaluate all the trade-offs of the alternatives

	[SM] Erm, my complaints mostly come from comparing the promises in the L4S drafts with the reality of the L4S implementation after a ~decade of working on it. This is not about perfection, but about demonstration that the promises can actually be robustly and reliably delivered. The "perfect versus good" argument does IMHO not apply here, as I am rarely comparing L4S against a theoretically superior hypothetical alternative. 

Best Regards
	Sebastian

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 12 May 2020 18:29:55 -0400

> right?  I don't see you can dislike SCE for that.

Yes, they would hit these issues if they tried to participate in a
plain RFC3168 conversation, but I think that's somewhat academic,
since they don't use RFC3168. I think the useful comparison is L4S vs
SCE.

> How does it behave for L4S, without changing the datacentre hardware?

I think for the L4S case, the point is that the sender would mark the
packets as ECT(1), even shallow queues in the datacenter switch would
be marked as CE, the receiver would reflect the CE information, and
the sender would see the CE but would know that the queue might well
be shallow, so would not necessarily cut its cwnd by 50% (Reno) or 30%
(CUBIC), until/unless the CE marks were sustained/prevalent.

Best regards,
neal

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 13 May 2020 05:07:50 +0300

> On 13 May, 2020, at 1:29 am, Neal Cardwell <[email-masked]> wrote:
> 

If you want to compare L4S to SCE, do it properly.

Look at the big picture.  How many existing, RFC-compliant networks does each proposal break?  How many of the ones that do break are RFC-ignorant to begin with?  How severe is the breakage; does it violate the principle of effective congestion control, or merely degrade gracefully to a more basic level of functionality?

Look at the data produced by both sides, not just one, and question why one set appears to contradict the other set.  Is there a difference in the focus of each set of tests?  Are they looking for the optimistic scenario, as desired for marketing, or for the pessimistic scenario, as required for real engineering?

Look at the arguments being presented.  Do they invoke appeals to authority, or fall into the sunk cost fallacy?  Or are they trying desperately to function as a meritocracy in an environment that seems strangely hostile to that concept?

> (CUBIC), until/unless the CE marks were sustained/prevalent.

So basically, from your perspective, it's fine to discard the existing ECN deployment in favour of making everything run DCTCP.  You will ignore all requests to negotiate RFC-3168, and only respond affirmatively to AccECN.  Welcome to the brave new world of L4S.  Safe as helicopters.  I'm *so* glad I'm a Beta.

What do you do about all those flows which, denied the chance to run ECN, have fallen back to packet loss as a congestion signal?  You still have the same problem of coping with their AIMD behaviour, and their application latency is increased by approximately one RTT every time they have to retransmit one of those lost packets.  So much for the L4S promise of "ultra low latency for everyone".

And what of everyone else who is already running, or thinking about deploying, AQMs and RFC-3168 ECN?  L4S throws a huge spanner into those works, which have taken twenty years to get moving to any appreciable degree.  Personally, I think the tipping point has been the development of AQM algorithms that are correctly configurable by mere mortals - a state of affairs to which I have contributed several years of my time.

There will be plenty of people who deliberately disable L4S support because their own network interacts badly with it - including everyone connecting over wifi.  I wonder what percentage of ordinary consumers even know what an Ethernet cable is?  SCE still offers reasonable performance to those users, and I'd be perfectly happy if they carried on using plain old ECN.  Any AQM at all is an improvement over a dumb, two-second-deep FIFO.

L4S still has too many problems - fundamental ones to which no robust solution has even been outlined - to commit the last ECN codepoint to.  I will freely admit that SCE also still needs work, but in some areas we are clearly ahead, and we make an effort to avoid over-promising.  We've had one year to get this far.  Given another year, what can we achieve?

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 14 May 2020 00:13:11 +0100

Neal, Jake,

> but would like their connections over the public Internet to be able

[BB] Heterogeneous DC networks was actually the original use-case that 
motivated the work that became known as L4S.

Back in 2013, when I worked for BT, most of my work was for BT's finance 
sector customers (BT operated what used to be the Thomson-Reuters 
Radianz network, which is the largest finance network in the world, 
interconnecting all the finance centres and stock markets). We did an 
analysis of the solutions that gave the greatest latency gains with the 
least pain (deployment cost). DCTCP had great potential, but we couldn't 
deploy it because it needed a single flag-day deployment, whiach wasn't 
feasible on this private internetwork operated by hundreds of 
independent administrators. Glenn Judd at Morgan Stanley was using 
traditional Diffserv for isolation [Judd15], but the whole Radianz 
traffic matrix was too unpredictable to configure static partitions.

That's why I started on the problem of coexistence between classic TCP 
and DCTCP [Kuehlewind14], which was just as applicable to the public 
Internet as to private DC networks. These remained the two strongest 
use-cases as we developed the DualQ Coupled AQM within the EU-funded 
RITE project, alongside Koen's team from Al-Lu, who were more focused on 
broadband access, which incidentally led to BT's network architect 
sponsoring the work internally in addition to the finance sector 
business unit.

After I left BT, I lost the link with the finance sector clients, but 
the DC use-case was still very relevant at Simula Research. Indeed, in 
2016 the Curvy RED pseudocode in draft-briscoe-tsvwg-aqm-dualq-coupled 
was picked up for implementation in merchant switch silicon.

However, once I started with CableLabs in 2017, the core L4S team no 
longer included anyone directly working on DCs, which is probably why 
you aren't aware of the DC heritage behind L4S.

____________
Quoting from the Use cases section of the L4S architecture:

    o  Private networks of heterogeneous data centres, where there is no
       single administrator that can arrange for all the simultaneous
       changes to senders, receivers and network needed to deploy DCTCP:

       *  a set of private data centres interconnected over a wide area
          with separate administrations, but within the same company

       *  a set of data centres operated by separate companies
          interconnected by a community of interest network (e.g. for the
          finance sector)

       *  multi-tenant (cloud) data centres where tenants choose their
          operating system stack (Infrastructure as a Service - IaaS)

And quoting from the Scope section of draft-ietf-tsvwg-aqm-dualq-coupled:

    ... it is believed the
    Coupled AQM would be applicable and easy to deploy in all types of
    buffers; buffers in cost-reduced mass-market residential equipment;
    buffers in end-system stacks; buffers in carrier-scale equipment
    including remote access servers, routers, firewalls and Ethernet
    switches; buffers in network interface cards, buffers in virtualized
    network appliances, hypervisors, and so on.

[Kühlewind14] Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, 
B., "Using Data Center TCP (DCTCP) in the Internet," In: Proc. Third 
IEEE Globecom Workshop on Telecommunications Standards: From Research to 
Standards pp.583-588 (December 2014)

[Judd15] Judd, G., "Attaining the Promise and Avoiding the Pitfalls of 
TCP in the Datacenter," In: 12th USENIX Symposium on Networked Systems 
Design and Implementation (NSDI 15) pp.145-157 USENIX Association (May 2015)

more...

> Yes, I agree with all those points; bleaching is very far from ideal.

[BB] It's important not to change ECT(1) to Not-ECT, and there's no need 
to, as long as you can configure AQMs to treat ECT(1) as Not-ECT.

[I-D.ietf-tsvwg-ecn-l4s-id] says

    "the ECT(1) codepoint MUST NOT be
    changed to any other codepoint than CE"

See also 
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-5.4.1.2

5.4.1.2. Exclusion of Traffic From L4S Treatment
...
    The operator MUST NOT alter the end-to-end L4S ECN identifier from
    L4S to Classic, because its decision to exclude certain traffic from
    L4S treatment is local-only.  The end-to-end L4S identifier then
    survives for other operators to use, or indeed, they can apply their
    own policy, independently based on their own choice of locally-used
    identifiers.  This approach also allows any operator to remove its
    locally-applied exclusions in future, e.g.  if it wishes to widen the
    benefit of the L4S treatment to all its customers.

>> implementations for at least the early stages...)

All the tunnels and encapsulations that I mentioned are surely highly 
relevant to any "two-output" marking scheme (whether SCE or your 
proposal). I've updated the 2-slide briefing I posted a couple of weeks 
ago to add a third slide about how an AQM applying your proposed marking 
scheme within a tunnel would just get stripped:
http://bobbriscoe.net/presents/2004ietf/2020-04ecn-tunnel-brief.pdf

Actually, my long list was in response to an offlist email from you. So 
I ought to repeat it here for the list:

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
Jake,

[Context: marking ECT(1) -> ECT(0)...]
I'm afraid there are two very strong reasons why we didn't do this:

#1/ That will be stripped by all stds track ECN tunnels and 
encapsulations, e.g. the following that specify ECN processing specifically:

  * RFC3168,
  * IPSec [RFC4301]
  * those tunnels that have since been updated to RFC6040,
  * All IP/UDP/IP encaps [RFC8085]
  * MPLS ECN [RFC 5129],
  * CAPWAP [RFC5415]
  * LISP [RFC6830]
  * VXLAN [RFC7349]

And the following stds track drafts that are widely already implemented 
because they are close to RFC:

  * TRILL (altho that's waiting on the L4S drafts in the RFC Editor
    queue, even tho it's stds track).
  * Geneve [draft-ietf-nvo3-geneve-16]
  * GUE [draft-ietf-intarea-gue-09]

It would also be stripped by all the other stds track encaps that some 
implementers could have written to comply with either of RFC3168 or 
RFC6040, e.g.

  * L2TP [RFC3931]
  * GRE [RFC2784]
  * GTP [3GPP]
  * Teredo [RFC4380]

#2/ [...snip]

Bob
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 28 May 2020 06:14:58 +0000

Hi Neal and Bob,

On 5/12/20, 1:55 PM, "Neal Cardwell" <ncardwell=[email-masked]> wrote:
On Fri, May 8, 2020 at 6:02 PM Holland, Jake
> but would like their connections over the public Internet to be able

I'm confused how this would operate in a way that's compatible with both
existing hardware and internet traffic, but I can think of 2 possible
interpretations.  Are you saying that:

(1) For internet traffic, classic markings (ECT(0) and NECT) would be
segregated to a different path, but L4S traffic (ECT(1) and CE) would be
sent through an existing marking low-threshold queue as currently
configured for DCTCP so it could mark the internet L4S traffic without a
hardware upgrade? Or

(2) The datacenter traffic and the internet traffic would be completely
segregated, and the existing datacenter switches wouldn't be involved in
marking the L4S traffic from the internet?

The point being: I don't think the existing switches would be able to
mark internet traffic as in (1) without segregation, because otherwise
the classic traffic would back off aggressively to the low-threshold
signal from the existing queues.

So it sounds to me like this idea is something like dualq but uncoupled,
with separate devices as the different queues to avoid a hardware upgrade
in order to use L4S?  (Otherwise, as in (2), I'm not clear on how the
existing hardware makes a difference.)

Just want to make sure I understand the datacenter angle on this.
Thanks in advance for clarifying, if you get a chance.

I tried to figure out which of these you meant (or whether it was a 3rd
option), by re-reading these messages:
https://mailarchive.ietf.org/arch/msg/tsvwg/Io3UlijUMEtjwyyxEwFSFBFpqIQ/
https://mailarchive.ietf.org/arch/msg/tsvwg/IWZNcqQCAwjX7AQT-QGJpLAqeEQ/
https://mailarchive.ietf.org/arch/msg/tsvwg/ZdNy965gDsqdqXz94rpENWaogwM/
https://mailarchive.ietf.org/arch/msg/tsvwg/1MlFf36hNSMqzG40uYGn-CpsqcY/

And it looked like you were thinking option (1) as far as I could tell,
but you didn't seem to have any mention of the need for segregating the
L4S vs. classic traffic from the internet, so I'm worried there might
have been a misunderstanding in the explanation you got from Bob.

As far as I could tell from Bob's response so far, he is claiming that
datacenter switches WOULD need to upgrade hardware to have dualq support
in order to participate in marking L4S traffic from the internet, unless
maybe they were marking ONLY L4S traffic, though it was a little hard to
be sure:
https://mailarchive.ietf.org/arch/msg/tsvwg/1cU3g2KfBFiWOAJZnUOKybeOrGk/

Or Bob, maybe you have some clarifying remarks on this point?  It's
possible I'm misunderstanding something.

(I'm granting the significance of the tunnel issues you mentioned in
the last referenced message (thanks for including your tunneling
protocol list and analysis), and asking just for your position on whether
existing switches can participate in marking mixed internet traffic
without segregating the L4S traffic.)

Best regards,
Jake

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 28 May 2020 09:28:04 -0400

Hi Jake and Bob,

> option), by re-reading these messages:

I'm thinking of a 3rd approach not listed above:

(3) There is no RFC3168 ECN traffic, because the site does not use
RFC3168 ECN. All TCP/UDP traffic would be allowed to share a queue:
non-ECN public Internet traffic, L4S public Internet traffic, and
low-threshold-ECN internal datacenter traffic. During conditions of
congestion, the switches would mark using shallow-threshold ECN marks,
and thus the non-ECN public Internet traffic would receive a higher
share of bandwidth than L4S or internal traffic in that queue, but
this is deemed acceptable, since public Internet traffic is already
deemed to be highest-priority. In this approach, the L4S traffic is
added into the mix using existing queues in existing datacenter switch
hardware, without introducing new devices, new queues, or dualq.

best,
neal

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 28 May 2020 17:11:36 +0000

Hi Neal,

> hardware, without introducing new devices, new queues, or dualq.

After a bit of searching, this makes sense to me now.

It looks like this approach can work with multifield classifiers, or
maybe even without if using Juniper's version of WRED which to my
surprise seems to say it doesn't drop NECT at WRED thresholds when the
queue has ECN enabled (and if you don't mind adding delay to the L4S
flows), so TIL:
https://www.juniper.net/documentation/en_US/junos/topics/concept/cos-qfx-series-tail-drop-understanding.html#jd0e290

Cisco and the tc-based ones I'm pretty sure do the thing I think
RFC 3168 describes and drop NECT packets that would have been marked.

Which is why I was at first confused by your explanation, since it
seems like that would wreck the NECT internet traffic in your setup
if you didn't split up the queues.  But now I think I see how it
could work with the right config, and this usage hadn't occurred
to me.

So I very much appreciate the clarification, thanks.

Best regards,
Jake