Subject: [tsvwg] Update to Position Statement on ECT(1)

From: "Holland, Jake" <[email-masked]>
Date: Fri, 8 May 2020 19:42:07 +0000

Hi tsvwg,

As promised in my original position statement[1], if I substantially changed
my views on the ECT(1) question I would post an update.

It has come to my attention that a technical fix is possible for my safety
concerns with "ECT(1) as an input" by changing the signaling scheme.

Although I still stand by all my claims in the original position statement,
the existence of a safe signaling scheme that uses ECT(1) as a network input
has changed my conclusion on the input/output question.

New position:
- I slightly prefer ECT(1) as an input (with qualifiers, given below)

(Note that this position should not be taken as an endorsement of L4S's
safety as currently proposed.)

The new signaling scheme that drove my change of position is this:

- ECT(1) is set from sender after negotiating endpoint support
- On-path devices change ECT(1)->ECT(0) to signal low levels of congestion
- On-path devices continue to use CE, as in RFC3168, to signal high levels
  of congestion, resulting in a required multiplicative decrease response.

Under this scheme, for any path with a single AQM that's dualq, ECT(1)
remains a very good classifier for that AQM.  Since this covers most
relevant paths that aren't within a controlled environment like datacenters,
it has a low downside.

Under this scheme, ECT(0) becomes the 1/p signal for dualq+TCP Prague, and
CE becomes the 1/sqrt(p) signal from the classic queue if the LL queue
overflows, and results in multiplicative decrease from the sender.

This would make L4S compatible with RFC 3168 without relying on a fragile
classic queue detection algorithm, so it would address my safety concerns.

As with all available signaling schemes, I acknowledge that this approach
is not perfect, and comes with tradeoffs.  A few of the known tradeoffs
would include (with thanks to Bob, Koen, and Kyle for explaining some of
these to me offline):
- existing tunneling decapsulation specs would often lose non-CE signals
- the existing accecn spec would often lose non-CE signals
- For paths with multiple AQMs, the classifier partially loses integrity in
  later AQMs when earlier AQMs are loaded.  (Note also the worse downside
  that increasing deployment of new AQMs potentially reduces the fidelity
  further.)

In spite of the downside from these tradeoffs (and the work that would be
necessary to fix the specs and their deployment to capture the most value
from L4S), a signaling scheme with the backward compatibility that this
approach provides is what would make the key difference between a safely
deployable L4S and not, IMO.

As I said, I still stand by my previous claims.  In particular, I still
believe that DSCP is a reasonable and appropriate classifier for L4S
traffic at this stage of its maturity.

However, I also acknowledge that there's value in getting a quicker wide
deployment, as long as it can be done safely.  Since I believe ECT(1) with
the above signaling scheme can do so, I now think it's as reasonable a
choice as DSCP, but carries substantial benefits.

Since this approach would give almost the same benefits as "ECT(1) as
output", and also provides a classifier that can serve dualq's needs well
in most of the deployment scenarios, "ECT(1) as input" is my current
preference, because of my new belief that it can be made compatible
with RFC 3168 queues and still mostly get the classification job done.

I remain opposed to moving L4S forward in a way that's not compatible with
RFC 3168 queues, as it's currently proposed.

I also remain skeptical that it's possible to get the classic queue
detection working robustly, I think that's probably a dead end. And I
have become more skeptical of the viability of the queue protection
mechanisms mentioned, because those seem to require access to the layer 4
packet contents, which has been flagged as too hard to be practical.

So I remain skeptical of the safety stories told so far for the current
L4S proposal, because it has no MD fallback signal except loss or detection.

Best,
Jake

PS: I also have some mostly-supportive comments on Kyle's remarks about
the input/output question that might be relevant.  Thread is here and should
soon get a new message:
https://mailarchive.ietf.org/arch/msg/tsvwg/VhgCiE9dF6F2Z-eN9wkpeVG2LX0/

[1] Jake's original position on ECT(1):
https://mailarchive.ietf.org/arch/msg/tsvwg/Zrk7Up6g9BwfnJLjKD44K0riAg4/

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 8 May 2020 22:58:53 +0000 (UTC)

Hi Jake, tsvwg,

If we're thinking of alternate encodings, it seems to me that there are three theoretically possible:

A) Jake's encoding: ECT(0) as 1/p, CE as RFC3168
 Advantages and disadvantages as mentioned already.

B) As A), but with ECT(0) and CE switched.
This retains the classification properties of L4S (both 1/p marked and unmarked packets classify as low latency) but doesn't resolve the ambiguity of CE versus RFC3168, and therefore doesn't obviously provide a benefit.

C) ECT(1) as 1/p, CE as RFC3168

Without a classifier, this has the problems with tunnels etc that people have mentioned.

However it occurs to me that the ECT(1) (and CE) could still be used as a classifier in the following way:
Have the Prague sender premark 50% of packets with CE, in an order known to the receiver (eg, alternating).
The L4S AQM and the RFC3168 would then only be able to mark different packets, so  the receiver can then determine with good confidence whether any CE marked packet was 'originally' ECT(1), and if so, it must have been marked by a RFC3168 
AQM. (At this point the sender should revert to RFC3168 and stop sending premarked CE packets).

This has the advantage of over A) that the classification signal is not lost after the first bottleneck. However it has the same problem with tunnels, and I can see some additional issues:

 Both the RFC3168 and L4S congestion signals would be degraded.  For RFC3168 this degradation would be only until the RFC3168 AQM was detected, which under conditions of congestion should be with a well defined probability per RTT. It is not obvious how much a 50% loss of bandwidth for the 1/p signal would weaken the latency 

Middleboxes which rewrite tcp (eg, by changing packet boundaries) would cause problems. Work would be required to check that they can be detected reliably.

Although the classification would be preserved across L4S AQMs, conditions of an L4S AQM followed by an RFC3168 AQM (or vice versa) could cause some congestion marks to be "undone". However this could only occur until the RFC3168 AQM was detected. IT woudl not be caused my multiple AQMS of the same type.

CE has up till now been a fixed point (no transitions away from CE) so there may be other network elements which are surprised by transitions away from it.

Anyway, hope that is of interest.

(I will not be taking a  position on the consensus call, as I do not currently work in networking and don't have time to do the necessary due diligence).

Alex

On Friday, May 8, 2020, 8:51:26 PM GMT+1, Holland, Jake <jholland=[email-masked]> wrote: 

Hi tsvwg,

New position:

Best,
Jake

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 9 May 2020 00:17:22 +0000

Hi Alex,

> This has the advantage of over A) that the classification signal is not lost after the first bottleneck. However it has the same problem with tunnels, and I can see some additional issues:

I'm not sure I follow the specifics here--neither a L4S AQM nor a classic
queue would re-mark anything that's CE.  Did you mean that there would be
ECT(0) in a known part of the packet stream that would be used as an active
probe for classic queue-marking?

An ongoing active probe is a potentially interesting idea I haven't seen
considered.  I think there are a lot of potential complications there, but
I can see that it might be possible to get a better signal on presence of a
classic queue at a bottleneck than from a purely heuristic detection algorithm.

> Although the classification would be preserved across L4S AQMs, conditions of an L4S AQM followed by an RFC3168 AQM (or vice versa) could cause some congestion marks to be "undone". However this co…

Are you imagining an ongoing probe that permanently cuts over to a classic
marking style as soon as a classic queue is detected?

> CE has up till now been a fixed point (no transitions away from CE) so there may be other network elements which are surprised by transitions away from it.

Yes, if something transitions away from CE, it seems incompatible with
RFC 3168 queue markings.  I'm not sure offhand what the pathologies would
look like, but I'd be similarly skeptical it could end well.

> Anyway, hope that is of interest.

Thanks for the comment, I think there's maybe an interesting idea in there,
though I'm not sure I quite grasped how it would work.

Best,
Jake

---

From: "[email-masked]" <[email-masked]>
Date: Sat, 9 May 2020 01:24:28 +0000 (UTC)

Hi Jake,

replies inline (marked [AB]).

On Saturday, May 9, 2020, 1:34:29 AM GMT+1, Holland, Jake <[email-masked]> wrote: 

Hi Alex,

[AB] Not exactly. The existing L4S scheme mark congestion with an ECT(1) to CE transition. In the scheme C in my previous mail, an L4S AQM would be defined to  mark it instead with the opposite, a CE to ECT(1) transition, while the RFC3168 AQM continues to mark using the ECT(1) to CE transition.

[AB] Hence, the scheme requires 50% of packets to be pre-marked with CE by the sender, since otherwise there are unlikely to be enough CE marked packets to allow the 1/p signal to be sent.  This allows 1/p and RFC3168 signals to be distinguished by the receiver - which is assumed to know which packets were premarked CE -  albeit with some degradation to each. 50% of RFC3168 marks would be unobservable to the receiver initially, but as soon as it observed one, the receiver would signal the sender to stop premarking packets with CE, allowing all marks to be observed. Probably the receiver should count all RFC3168 double until then.  

[AB] Probing with ECT(0) is mentioned in Bob's paper, I'm afraid I didn't read the test results closely enough to remember if 
it was included in the method tested.

[AB] Yes, except as described above, not using additional ECT(0) packets.

[AB] Indeed, I am not able to foresee clearly what difficulties it could encounter.  I mention it largely for the sake of completeness.

[AB] Thanks, hopefully I have made it a bit clearer.

Alex

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 9 May 2020 12:19:04 +0300

> On 9 May, 2020, at 4:24 am, [email-masked] wrote:
> 
> [AB] Not exactly. The existing L4S scheme mark congestion with an ECT(1) to CE transition. In the scheme C in my previous mail, an L4S AQM would be defined to  mark it instead with the opposite, a CE to ECT(1) transition, while the RFC3168 AQM continues to mark using the ECT(1) to CE transition..

I don't think this idea is workable.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Sat, 9 May 2020 13:50:22 +0000

Ah yes, I remember now that you point it out, section 5.  Thanks for
reminding me.

Now that I looked again I recall skimming that and thinking it had
some troubles, like that just running a naive active probe with ECT(0)
would capture the classic half of the dualq, and that as written it
depended on the heuristic half of the queue detection to try to avoid
the overhead, so it seemed a secondary thing that was still subject to
failures in the main detection.

But you're right, it's there and the general strategy might have some
promise for improving the detection, with some costs.  Totally forgot
that I had already seen active probes mentioned after all.

-Jake

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 10 May 2020 07:36:56 -0700

> concerns with "ECT(1) as an input" by changing the signaling scheme.
[...]

> classic queue detection algorithm, so it would address my safety concerns.

Thank you for presenting this idea. If the details can be worked out it
looks very promising.

> - existing tunneling decapsulation specs would often lose non-CE signals

Yes, combinations marked (**) below would have to changed from RFC 6040:

            +---------+------------------------------------------------+
            |Arriving |            Arriving Outer Header               |
            |   Inner +---------+------------+------------+------------+
            |  Header | Not-ECT | ECT(0)     | ECT(1)     |     CE     |
            +---------+---------+------------+------------+------------+
            | Not-ECT | Not-ECT |Not-ECT(!!!)|Not-ECT(!!!)| <drop>(!!!)|
            |  ECT(0) |  ECT(0) | ECT(0)     | ECT(0) (**)|     CE     |
            |  ECT(1) |  ECT(1) | ECT(0) (**)| ECT(1)     |     CE     |
            |    CE   |      CE |     CE     |     CE(!!!)|     CE     |
            +---------+---------+------------+------------+------------+

Similar changes would be needed for draft-ietf-tsvwg-rfc6040update-shim and
draft-ietf-tsvwg-ecn-encap-guidelines.

Clearly, the need to get such changes deployed would be a barrier to
barrier to adoption.

Actually, I would go farther and say that something rather different from
the existing AccECN draft would be needed. AccECN provides accurate
feedback of the number of CE marks observed. Under the proposed scheme L4S
would need to getting accurate feedback of the number of ECT(0)
(pre-congestion / some congestion) marks. AccECN would need to be
re-worked to provide both that and, in addition, either the existing
ECE/CWR handshake or something else that performs the equivalent function.
The most obvious solution would be to repurpose NS and one or  more
currently reserved flag bits (or use other ideas from RFC 7560 Sec 5.2) and
leave ECE/CWR unchaged. I note in passing the SCE proposal would have to do
something along the same lines (though AFAICT that has not yet been fully
fleshed out).

>   that increasing deployment of new AQMs potentially reduces the fidelity

If I understand what is being said, this is because ECT(0) would become
ambiguous, as it can appear either on an L4S packet with a pre-congestion
marking, or a non-L4S packet.  Doesn't the same issue exist with the
current L4S proposal for CE-marked packets?

> deployable L4S and not, IMO.

+1

> with RFC 3168 queues and still mostly get the classification job done.

Actually, it seems to me that this approach would yield exactly the same
congestion signaling capability as using ECT(1) as a  pre-congestion / some
congestion mark. All that has been done is to reverse the role of ECT(1)
and ECT(0) compared to what the SCE draft and RFC 6040 envisioned. In other
words:

      +-----+-----+
      | ECN FIELD |
      +-----+-----+
         0     0        Not-ECT
         0     1        ECT(1) - L4S/SCE Capable AND No Congestion
         1     0        ECT(0) - Some Congestion OR RFC 3168 ECN Capable
         1     1        CE

Jake, you said that the three issues discussed above -- tunnels, AccECN,
and multiple AQMs in the path are "a few of the known tradeoffs." What are
the others?

Mike Heard

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 11 May 2020 00:09:29 +0000

Hi Mike,

From: "C. M. Heard" <[email-masked]>
> Yes, combinations marked (**) below would have to changed from RFC 6040:
...
> Similar changes would be needed for draft-ietf-tsvwg-rfc6040update-shim and draft-ietf-tsvwg-ecn-encap-guidelines.
>
> Clearly, the need to get such changes deployed would be a barrier to barrier to adoption.

Yes.  I think in a recent thread I heard it confirmed that current tunnel
handling of these is kind of spotty today, specifically:

  > as an endpoint we'll be dealing with weird inconsistencies that basically
  > never fully understood anything beyond "don't lose CE marks" and maybe
  > "loss is acceptable in confusing cases", if we're lucky.

  [BB] See reply to Dave Taht just now, which pretty much confirms what 
  you've just said. 

that's from here:
https://mailarchive.ietf.org/arch/msg/tsvwg/QudqLu1RTQZCVnS4HNf8jnZ_kIY/

(I think the Dave Taht reply he was referencing was this one:
https://mailarchive.ietf.org/arch/msg/tsvwg/2ElPK72IiFg2gHJZ_rLMUKTDfCI/ )

I'm beginning to think the reason I've come down differently than the
L4S team on the judgement call for this approach being better, in light
of the state of tunneling encapsulation deployments, might boil down to
a disagreement over the answer to a question like:
  Which is better:
  - losing the safety response of MD from a loaded classic queue, or
  - losing some of the reliability on the low-latency response when there
    is a dualq on-path?

I'm beginning to think we might be stuck with one of those options for
tunneled paths until tunnel decap implementations can be widely upgraded
in deployment, however this lands.

> Actually, I would go farther and say that something rather different from the existing AccECN draft would be needed. AccECN provides accurate feedback of the number of CE marks observed. Under the proposed scheme L4S would need to getting accurate feedback of the number of ECT(0)  (pre-congestion / some congestion) marks. AccECN would need to be re-worked to provide both that and, in addition, either the existing ECE/CWR handshake or something else that performs the equivalent function. The most obvious solution would be to repurpose NS and one or  more currently reserved flag bits (or use other ideas from RFC 7560 Sec 5.2) and leave ECE/CWR unchaged. I note in passing the SCE proposal would have to do something along the same lines (though AFAICT that has not yet been fully fleshed out).

Agreed, I think a different feedback than AccECN would be smarter if the
ECT(1)->ECT(0) approach goes forward, and I like the NS reflection approach
that SCE's implementation started with.  (Although it might lose fidelity
from some ack aggregation responses, I'd expect usually that the marking
rate maintains proportionality on the low-congestion signal, and where that
fails, the standard ECE response is at least reliable, so it covers the
safety considerations the same way as classic marking.)

However, I also think for anyone who disagrees, other viable approaches
for the feedback might be possible.  But in that direction, I do think
it probably would need to differ from AccECN.  This would of course need to
be nailed down in the end, though I didn't get into it in the first email.
But the potential complexity here is one of the reasons I rate the
suggestion as perhaps a major architectural change for L4S.

I tend to think that the per-ECT(0) reflection in NS is the best way,
but I don't think it would change the rest of the argument if that position
turned out wrong.

> If I understand what is being said, this is because ECT(0) would become ambiguous, as it can appear either on an L4S packet with a pre-congestion marking, or a non-L4S packet.  Doesn't the same issue exist with the current L4S proposal for CE-marked packets?

Yes, but the L4S specs go over this and walk through the reasoning for
why they landed on classifying CE into the LL queue, and the net result
in that case is  that the ECT(1)->CE marking strategy that L4S currently
follows will keep the 1/p packet signals in the LL queue for the later
hops.

(The potential problems were mostly limited to mis-classified marked
classic traffic, which will tend to be fewer in number and also less
severe given that the flow is slated to back off anyway, plus a review of
the main implementations suggested they wouldn't be doing double-backoffs
if the CE packet was out of order, IIRC.)

It might be better to phrase this not as "loses integrity", but rather as
"might systematically increase the latency experienced" for the 1/p signal,
since when there are multiple dualqs in line, those packets (but not others
from L4S flows) will land in the classic queue on the later dualqs.  This
is arguably a worse downside than the classification failure from putting
CE-marked classic traffic in the LL queue.

> Actually, it seems to me that this approach would yield exactly the same congestion signaling capability as using ECT(1) as a  pre-congestion / some congestion mark. All that has been done is to reverse the role of ECT(1) and ECT(0) compared to what the SCE draft and RFC 6040 envisioned. In other words:
>
>      +-----+-----+
>      | ECN FIELD |
>      +-----+-----+
>         0     0        Not-ECT

Yes, that's my understanding.  I think the whole proposal can reasonably be
summarized as "SCE with the bits flipped".

> Jake, you said that the three issues discussed above -- tunnels, AccECN, and multiple AQMs in the path are "a few of the known tradeoffs." What are the others?

These are all the ones I know of yet, but I think Bob and Koen might have
some others they already know.  I'm not sure I got the whole story yet.

Thanks for your comments.

Best regards,
Jake

---

From: Martin Duke <[email-masked]>
Date: Mon, 18 May 2020 15:11:44 -0700

Jake,

I'm intrigued by this discussion of the ECT(1)->ECT(0) proposal, as
something that could definitively solve the safety concerns. I'll make two
unrelated points:
1) If the current L4S proposal is in need of an MD signal, there's always
dropping the packet.  Although packet loss is bad, maybe some drops at the
end of slow start is the tradeoff we have to make to get low latency.
Implementations really concerned about loss can be less aggressive during
slow start.
2) Clearly, there is no AccECN signaling problem for ECT(1)->ECT(0) for
QUIC, and for TCP paths where the option gets through. It this is an issue
of the three ACe bits, I think one codepoint in ACE would be sufficient to
indicate that a CE mark was received, which IMO would trump whatever other
feedback is in that header. Unless there's some sort of performance cliff
in not being able to encode 7 ECT(0) marks, this seems like a non-problem

Martin (as an individual)

On Sun, May 10, 2020 at 5:09 PM Holland, Jake <jholland=
[email-masked]> wrote:

> > Similar changes would be needed for draft-ietf-tsvwg-rfc6040update-shim
> and draft-ietf-tsvwg-ecn-encap-guidelines.
> >
> > Actually, I would go farther and say that something rather different
> from the existing AccECN draft would be needed. AccECN provides accurate

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 19 May 2020 02:56:27 +0000

Hi Martin,

Thanks, I think it’s an interesting direction also, and I’m glad to get some discussion on it.  Here’s hoping it turns out useful, or at least informative.

Some responses inline with <JH></JH>

From: Martin Duke <[email-masked]>
Date: Monday, May 18, 2020 at 3:12 PM
To: "Holland, Jake" <[email-masked]>
Cc: "C. M. Heard" <[email-masked]>, TSVWG <[email-masked]>
Subject: Re: [tsvwg] Update to Position Statement on ECT(1)

Jake,

I'm intrigued by this discussion of the ECT(1)->ECT(0) proposal, as something that could definitively solve the safety concerns. I'll make two unrelated points:
1) If the current L4S proposal is in need of an MD signal, there's always dropping the packet.  Although packet loss is bad, maybe some drops at the end of slow start is the tradeoff we have to make to get low latency. Implementations really concerned about loss can be less aggressive during slow start.

<JH>
To me, the problem with loss as the only (viable) MD signal is the existing devices that don’t drop until well after exceeding reasonable fairness bounds from an unresponsive flow, so for TCP Prague to maintain compatibility with classic queues, it would still need a successful classic queue detection mechanism.

For example, Cisco’s AFD approach is a modern classic queue in a high-speed device that doesn’t need tuning like wred, and seems to be helpful, and allows ECN marking:
https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-738488.html
(h/t Dave Täht on the make-wifi-fast list, without whose comment I probably wouldn’t have noticed this)

There’s likewise some very interesting ongoing work on CoDel and PIE in P4 that might be relevant:
- https://ieeexplore.ieee.org/document/4813399
- https://p4.org/assets/P4WS_2019/p4workshop19-final10.pdf

The impact there might be lower, since they can perhaps upgrade to treat ECT(1) differently more easily, and perhaps even to dualq with perhaps a faster update cycle than replacing those nexus 9ks.  (But also because they haven’t quite gotten to marking yet, though they seem to intend to: https://github.com/PIFO-TM/ns3-bmv2/blob/master/traffic-control/examples/p4-src/pie/pie.p4#L186 )

RFC 7567 seems to encourage turning stuff like this on and enabling marking, but we’d expect it to interact poorly with L4S’s failure to treat the marks as the MD signal intended by the classic algorithms.  By the time multiple L4S flows competing with multiple classic marking flows causes loss across devices like these, based on past results we’d expect to sometimes see something between gross unfairness and near starvation for the classic flows.

Some people have argued that classic ECN had a long time to roll out without much motion, and so we should stop worrying about it.  Maybe that’s a good consensus question to check—I don’t agree with the claim, but I might be in the rough.

If consensus suggests that we should deprecate that 7567 advice and call classic ECN a failed protocol and make it historical (or deprecated without FQ), then I guess we should get that process started as soon as possible, with appropriate outreach to make for a smoother transition for those who have adopted it or are in progress working on it.

If we did that and didn’t get substantial pushback during the outreach, it would clear the way for incompatible proposals.  It’s certainly possible to land on a consensus that RFC 8311 didn’t go far enough in being permissive toward not worrying about shared classic marking queues, as I’ve heard expressed a few times.

But for now, without such a deprecation, it seems to me like a blocker for L4S as a viable experiment at internet scope if it can’t interoperate with existing devices reasonably, when the current BCP says to turn those devices on.  Even if they’re not quite turned on yet.
</JH>

2) Clearly, there is no AccECN signaling problem for ECT(1)->ECT(0) for QUIC, and for TCP paths where the option gets through. It this is an issue of the three ACe bits, I think one codepoint in ACE would be sufficient to indicate that a CE mark was received, which IMO would trump whatever other feedback is in that header.. Unless there's some sort of performance cliff in not being able to encode 7 ECT(0) marks, this seems like a non-problem

<JH>
I’m not sure I follow this suggestion.

If you’re imagining a “latch-on” behavior like ECE that waits for CWR from sender before moving along, I think the sender no longer has a CWR if they’ve negotiated ACE, so they can’t respond in a way that turns off the latch.  But if you’re just using 1 packet with 1 codepoint to indicate “CE observed”, you’ll miss the report of CE observed at sender if the ack bearing that signal is dropped or thinned or whatever, so I think neither of those would work.

But I guess you could maybe use 1 codepoint for a latched-on ECE and another codepoint for CWR, and still have a 6-slot counter for the ECT(0) signal (which is superseded by the ECE codepoint while active).

I do also think it makes good sense to use the SCE approach of reflecting each received ECT(0) in NS (or ESCE, you could still rename it), while keeping the ECE/CWR behavior the same and just not using AccECN.  This idea got a critical review from Bob here, so maybe you should check it before you believe me:
https://mailarchive.ietf.org/arch/msg/tsvwg/V5pFpko_JYaFKu68oXqe-nvgmNg/

I didn’t debate it at the time, but I found it uncompelling for a few reasons:
I think that although ack-thinning does reduce the accuracy, unless the trimmed acks are systematically biased, a smoothed NS signal would still converge to the right proportion of markings, so you’d only see minor fluctuation on the hi-fidelity response. The instantaneous error you’ll get from thinned acks isn’t very relevant as long as over time the signal converges to the proportion actually marked by the network.  And in the cases where it does diverge by enough in the unsafe direction, the MD signal would still serve as a solid fail-safe.

To me the problems seem very likely addressable, especially because degenerate cases (e.g. as mentioned in the critique: 50% marking on every other packet but ack-thinning every other ack that makes a 100% or 0% signal out of a 50% one) wouldn’t be stable, because such a pattern would induce a response from the sender that would change the mix of traffic and break the degenerate marking+thinning interaction, and since this is the hi-fidelity path I’d expect usually to see only a relatively quick fluctuation (and again, importantly: backstopped by a reliable fail-safe).

But I’m also not opposed to one ACE codepoint for a latched ECE and another for CWR, with 6 counting slots for the hi-fi signal.  Not that I’ve seen it tested, but offhand it seems like probably a pretty reasonable approach.  So my conclusion is that I’m (tentatively) pretty well convinced that the endpoint feedback signaling is solvable with 2 congestion signals, even under constraints about reliable transport of TCP options.

With that said, I’ll raise my own unrelated point:

3) I think the issue Bob raised about tunnels is the bigger crux.

To me, if we can have only 1 explicit signal and the choice is between the MD signal and the hi-fi signal, I would keep the MD signal because *normal AQM solves 90% of the problem*, and the incremental gains you can get from hi-fi congestion signaling can be realized over the decade it’ll take to get tunnel decapsulators to pass both signals, *provided you don’t break normal AQMs and give people a reason to bleach ECT(1)*.

And anyone who cares enough about latency and also controls the tunnel decapsulator can deploy an upgrade earlier, so it’ll only be the long tail of providers who don’t get around to upgrading the tunnels that will suffer with only the 90% solution for the dualq bottlenecks their tunnels cross.

Many of the tunnel owners are the same entities who’d be rolling out the dualq, so hopefully they’d care enough to also upgrade the tunnels if they’re willing to turn on the dualq.  But it’s true that some tunnels aren’t owned by the same people as the queue owners.  I think that’s a fair point, and that’s the right kind of tradeoff to be debating (or maybe even gathering information on exactly how much of internet traffic we’re talking about that live under these different conditions).

So to me, maybe a better consensus question is: what’s more important, getting the hi-fi signal from new queues, or keeping the MD signal from existing queues?

Because it seems like we can’t have both with the current tunnels out there, and it seems like there’s plenty of tunnels.

And because the response to that question determines which decade-long project is our right next move to eventually get to low latency for all (either deprecating non-FQ classic ECN or upgrading tunnels to export 2 signals, respectively), and either way we should get that ball rolling, IMO.

What I’m really against is the short-cut of rolling out something incompatible with an ongoing rollout of an existing widely implemented technology, over short-term hopes of a rapid deployment for the new thing, with the likely result (as we’ve seen from discussion) of bleaching ECT(1) in at least some networks, with who knows how long a tail it will leave behind.

I’m eager to see the discussion turn from input/output on ECT(1) to “how important are classic shared marking queues”, if that’s actually under question.
</JH>

On Sun, May 10, 2020 at 5:09 PM Holland, Jake <jholland=[email-masked]<mailto:[email-masked]>> wrote:
Hi Mike,

From: "C. M. Heard" <[email-masked]<mailto:[email-masked]>>
> Yes, combinations marked (**) below would have to changed from RFC 6040:
....

that's from here:
https://mailarchive.ietf.org/arch/msg/tsvwg/QudqLu1RTQZCVnS4HNf8jnZ_kIY/<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_msg_tsvwg_QudqLu1RTQZCVnS4HNf8jnZ-5FkIY_&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=TIR1CmYggwBy2HCNBLTpW3pybzL5vXHm2w3hBT4hXr0&s=YZLWdFwAuG265end-PYLEkru5-KxLTgl3TDeIu3ldOM&e=>

(I think the Dave Taht reply he was referencing was this one:
https://mailarchive.ietf.org/arch/msg/tsvwg/2ElPK72IiFg2gHJZ_rLMUKTDfCI/<https://urldefense.proofpoint.com/v2/url?u=https-3A__mailarchive.ietf.org_arch_msg_tsvwg_2ElPK72IiFg2gHJZ-5FrLMUKTDfCI_&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=TIR1CmYggwBy2HCNBLTpW3pybzL5vXHm2w3hBT4hXr0&s=bQnu5-h2PmSQy5CniEvkBvwvMCWRHRBRuw38Oqg7s44&e=> )

Best regards,
Jake

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 19 May 2020 13:58:41 +0100

Martin,

On 18/05/2020 23:11, Martin Duke wrote:
> something that could definitively solve the safety concerns. I'll make 
> two unrelated points:
> 1) If the current L4S proposal is in need of an MD signal, there's 
> always dropping the packet.  Although packet loss is bad, maybe some 
> drops at the end of slow start is the tradeoff we have to make to get 
> low latency. Implementations really concerned about loss can be less 
> aggressive during slow start.
> 2) Clearly, there is no AccECN signaling problem for ECT(1)->ECT(0) 
> for QUIC, and for TCP paths where the option gets through. It this is 
> an issue of the three ACe bits, I think one codepoint in ACE would be 
> sufficient to indicate that a CE mark was received, which IMO would 
> trump whatever other feedback is in that header.. Unless there's some 
> sort of performance cliff in not being able to encode 7 ECT(0) marks, 
> this seems like a non-problem

[BB] I don't know how you get to one codepoint in ACE being enough for 
CE. The whole idea of AccECN is to be robust to ACK loss and thinning. 
If it was as simple as you make out, we could have sorted out this 
problem snip-snap and all be home in time for tea.

As we all try to remove more and more latency, we are all going to find 
that the problems get harder and harder - little details matter a lot. 
Without really solid feedback precision, you lose the consistent latency 
of L4S.

Richard Scheffenegger and I designed a 4-3-1 codepoint scheme for ACE at 
one point (4 for CE, 3 for ECT(1) and one for Not-ECT). See 
https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03#section-3.2.1 
Table 3. However, it was removed because it was "jack of all trades, 
master of none." By trying to cover all the codepoints, it didn't give 
sufficient robustness to any.

Even using all 8 values of the 3-bit counter for feeding back just one 
codepoint is on the edge of what is needed today. I sometime liken the 
ACE part of AccECN to a "stalking horse" - i.e. a vehicle that works for 
today, while also acting as a platform to get the larger AccECN Option 
deployed for the longer term - when the ACE field might have become too 
small for the possible future level of ACK thinning on some paths.

In fact, over the years {1} that we've been trying to get AccECN 
standardized, you will find all sorts of weird and wonderful compressed 
encodings, but in the end, as well as inadequate robustness, arguments 
for simplicity took precedence. Not just because complexity breeds bugs 
(which is not an insignificant consideration for TCP), but also because 
the semantics of ACE also have to be supported by segmentation offload 
hardware.

When, I highlighted the problems with tunnelling that the ECT1->0 scheme 
has, I only hinted at these problems with TCP, because I thought 
incompatibility with tunnelling should be a sufficient argument.

In summary, if you think deploying a change to IP is easy, you've 
probably not absorbed the full breadth of the problem.

Regards

Bob

{1} Since Oct 2005: 
https://tools.ietf.org/html/draft-briscoe-tsvwg-re-ecn-tcp-00#section-4.1

> On Sun, May 10, 2020 at 5:09 PM Holland, Jake 
> <jholland=[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Mike,
>
>     From: "C. M. Heard" <[email-masked] <mailto:[email-masked]>>
>       > as an endpoint we'll be dealing with weird inconsistencies
>     that basically
>       > never fully understood anything beyond "don't lose CE marks"
>     and maybe
>       > "loss is acceptable in confusing cases", if we're lucky.
>
>     > Actually, I would go farther and say that something rather
>     different from the existing AccECN draft would be needed. AccECN
>     provides accurate feedback of the number of CE marks observed.
>     Under the proposed scheme L4S would need to getting accurate
>     feedback of the number of ECT(0) (pre-congestion / some
>     congestion) marks. AccECN would need to be re-worked to provide
>     both that and, in addition, either the existing ECE/CWR handshake
>     or something else that performs the equivalent function. The most
>     obvious solution would be to repurpose NS and one or  more
>     currently reserved flag bits (or use other ideas from RFC 7560 Sec
>     5.2) and leave ECE/CWR unchaged. I note in passing the SCE
>     proposal would have to do something along the same lines (though
>     AFAICT that has not yet been fully fleshed out).
>
>     > If I understand what is being said, this is because ECT(0) would
>     become ambiguous, as it can appear either on an L4S packet with a
>     pre-congestion marking, or a non-L4S packet.  Doesn't the same
>     issue exist with the current L4S proposal for CE-marked packets?
>
>     > Actually, it seems to me that this approach would yield exactly
>     the same congestion signaling capability as using ECT(1) as a 
>     pre-congestion / some congestion mark. All that has been done is
>     to reverse the role of ECT(1) and ECT(0) compared to what the SCE
>     draft and RFC 6040 envisioned. In other words:
>     >
>     >      +-----+-----+
>     >      | ECN FIELD |
>     >      +-----+-----+
>     >         0     0        Not-ECT
>     > Jake, you said that the three issues discussed above -- tunnels,
>     AccECN, and multiple AQMs in the path are "a few of the known
>     tradeoffs." What are the others?
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Black, David" <[email-masked]>
Date: Tue, 19 May 2020 13:19:40 +0000

[still posting as an individual, not WG chair]

Bob,

> [BB] I don't know how you get to one codepoint in ACE being enough for CE. The whole idea of AccECN is to be robust to ACK loss and thinning.
> If it was as simple as you make out, we could have sorted out this problem snip-snap and all be home in time for tea.

I think you may have missed the point.  The topic under discussion is a 2-level marking system that retains the semantics of CE as calling for an RFC3168-style Multiplicative Decrease (MD), and uses ECT(0) [remarked from ECT(1)] as the queue occupancy mark for 1/p congestion control.  With two signals (CE and ECT(0)) arriving at the receiver, there need to be ways to reflect both to the sender.

The MD signal (CE) overrides the queue occupancy signal, as it calls for a much larger backoff.  ACK thinning should be less of a concern for MD because any arrival of CE (or detection of a drop) at the receiver is supposed to cause MD at the sender, and that sender MD reaction is limited to once per RTT.

The underlying reasons why a single codepoint may suffice to signal MD support via the AccECN field are two-fold:

  *   The MD signal (CE) overrides the queue occupancy signal (ECT(0)), as it calls for much more sender backoff, so there’s no point in AccECN for queue occupancy if a CE has been received.
  *   The MD signal does not require the gradations of AccECN – it’s an undifferentiated “Oh sh*t!” signal calling for MD backoff (and reversion to RFC3168 sender behavior).
That would still leave the rest of the AccECN codepoints to reflect the queue occupancy signal in the absence of CE arriving or a drop being detected.

Thanks, --David

From: tsvwg <[email-masked]> On Behalf Of Bob Briscoe
Sent: Tuesday, May 19, 2020 8:59 AM
To: Martin Duke; Holland, Jake
Cc: TSVWG
Subject: Re: [tsvwg] Update to Position Statement on ECT(1)

[EXTERNAL EMAIL]
Martin,

[BB] I don't know how you get to one codepoint in ACE being enough for CE. The whole idea of AccECN is to be robust to ACK loss and thinning. If it was as simple as you make out, we could have sorted out this problem snip-snap and all be home in time for tea.

As we all try to remove more and more latency, we are all going to find that the problems get harder and harder - little details matter a lot. Without really solid feedback precision, you lose the consistent latency of L4S.

Richard Scheffenegger and I designed a 4-3-1 codepoint scheme for ACE at one point (4 for CE, 3 for ECT(1) and one for Not-ECT). See https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03#section-3.2.1 Table 3. However, it was removed because it was "jack of all trades, master of none." By trying to cover all the codepoints, it didn't give sufficient robustness to any.

Even using all 8 values of the 3-bit counter for feeding back just one codepoint is on the edge of what is needed today. I sometime liken the ACE part of AccECN to a "stalking horse" - i.e. a vehicle that works for today, while also acting as a platform to get the larger AccECN Option deployed for the longer term - when the ACE field might have become too small for the possible future level of ACK thinning on some paths.

In fact, over the years {1} that we've been trying to get AccECN standardized, you will find all sorts of weird and wonderful compressed encodings, but in the end, as well as inadequate robustness, arguments for simplicity took precedence. Not just because complexity breeds bugs (which is not an insignificant consideration for TCP), but also because the semantics of ACE also have to be supported by segmentation offload hardware.

When, I highlighted the problems with tunnelling that the ECT1->0 scheme has, I only hinted at these problems with TCP, because I thought incompatibility with tunnelling should be a sufficient argument.

In summary, if you think deploying a change to IP is easy, you've probably not absorbed the full breadth of the problem.

Regards

Bob

{1} Since Oct 2005: https://tools.ietf.org/html/draft-briscoe-tsvwg-re-ecn-tcp-00#section-4.1

....

that's from here:

Best regards,
Jake

--

---

From: Martin Duke <[email-masked]>
Date: Tue, 19 May 2020 10:20:59 -0700

Hi Jake,

Sorry for a not-very-carefully composed email!

On Mon, May 18, 2020 at 7:56 PM Holland, Jake <[email-masked]> wrote:

> Hi Martin,
>
>
>
> Thanks, I think it’s an interesting direction also, and I’m glad to get
> some discussion on it.  Here’s hoping it turns out useful, or at least
> informative.
>
> be less aggressive during slow start.
>
>
>
> <JH>
>
> To me, the problem with loss as the only (viable) MD signal is the
> existing devices that don’t drop until well after exceeding reasonable
> fairness bounds from an unresponsive flow, so for TCP Prague to maintain
> compatibility with classic queues, it would still need a successful classic
> queue detection mechanism.
>

I probably misunderstood. I thought the desired signal was a MD signal in
the low-latency queue. Yes, for 3168 queues you'd have to essentially treat
ECT(1) as Not-ECT. So that may not be a productive direction.

> feedback is in that header.. Unless there's some sort of performance cliff
> If you’re imagining a “latch-on” behavior like ECE that waits for CWR from
> sender before moving along, I think the sender no longer has a CWR if
> they’ve negotiated ACE, so they can’t respond in a way that turns off the
> latch.  But if you’re just using 1 packet with 1 codepoint to indicate “CE
> observed”, you’ll miss the report of CE observed at sender if the ack
> bearing that signal is dropped or thinned or whatever, so I think neither
> of those would work.
>
>
>
> But I guess you could maybe use 1 codepoint for a latched-on ECE and
> another codepoint for CWR, and still have a 6-slot counter for the ECT(0)
> signal (which is superseded by the ECE codepoint while active).
>

Good catch -- soon after I sent this, I thought, "what about CWR". The
problem with using a second codepoint is that the sender signal then
pre-empts the feedback signal -- fine for functionally half-duplex
connections, but not generally. I can see a few ways out of this:

1) Take another TCP header reserved bit so that we can keep CWR and the 3
bits for ACE. (Ugh)
2) Come up with a heuristic to stop sending ECE, obviating CWR.
3) Come up with a heuristic to mix packets with CWR and ACE as needed
depending on counter activity.
Though I'll defer to Bob on the issues of re-engineering ACE -- it sounds
like he's been through all the permutations.

I guess I don't understand the tunneling problem. If we have a mark that
demands MD, that trumps feedback about low-latency fine tuning, does it not?

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 19 May 2020 20:13:27 +0000

Hi Martin,

New responses with <JH2></JH2>

From: Martin Duke <[email-masked]>
Date: Tuesday, May 19, 2020 at 10:21 AM

I probably misunderstood. I thought the desired signal was a MD signal in the low-latency queue. Yes, for 3168 queues you'd have to essentially treat ECT(1) as Not-ECT. So that may not be a productive direction.

<JH2>
As I understand it, this is the current L4S MD signal if the LL queue overflows into the low latency queue and has congestion there.

I agree this works as far as it goes, but the safety problem that seems to need an MD signal is the interaction with existing queue implementations that know nothing about L4S, and have a pre-existing interpretation of ECT(1).

(Even for FQ, the interaction is very poor, but shared queues is where the poor interaction is arguably a major safety concern.  Mostly nobody seems to care much about the occasional hash collision in FQ that would have the same problem, because most FQs are 1024 slots, so after birthday paradox considerations it’s only 1/32 or so of the competing flows.)

Anyway, if there was sufficiently good consensus that these classic queues are irrelevant, then deprecating them responsibly is a potentially viable way forward, but that’s not what any of the current BCPs or standards say today, I think.
</JH2>

Good catch -- soon after I sent this, I thought, "what about CWR". The problem with using a second codepoint is that the sender signal then pre-empts the feedback signal -- fine for functionally half-duplex connections, but not generally. I can see a few ways out of this:

1) Take another TCP header reserved bit so that we can keep CWR and the 3 bits for ACE. (Ugh)
2) Come up with a heuristic to stop sending ECE, obviating CWR.
3) Come up with a heuristic to mix packets with CWR and ACE as needed depending on counter activity.
Though I'll defer to Bob on the issues of re-engineering ACE -- it sounds like he's been through all the permutations.

<JH2>
My conclusion that it seemed workable to me came with the assumption that #3 would be pretty straightforward.

You don’t need very many CWRs, and it might be possible to clean up the specs a bit about when exactly they should be sent (like maybe they’re only appropriate when there’s a MD event, and maybe they shouldn’t be sent for linear reductions during a fine-tuning response, now that we’re contemplating such a thing).

So I was thinking for bidirectional traffic that has the newly suggested ACE signaling (with a codepoint for ECE and CWR, plus 6 count slots), you could just put CWR at the top priority, so the (only occasional) CWR would overrule either a latched ECE that’s still in progress or one of the fine-tuning signal slots, for the occasional packet reporting an MD event.  (The logic from 3168 still holds here: if CWR is on a data packet and the data packet is lost, you’ll still get an MD response, so it doesn’t need to be reliable like ECE, and the counter makes it so skipping a packet doesn’t lose the fine-tuning signal, since it’s no worse than thinning the acks would have been.)

I do worry a bit whether a 6-count is enough to detect rollover well, but like with the ESCE approach for reflecting the fine-tuning signal, losing some accuracy is acceptable if we have a reliable MD signal, so I think it’s probably ok.

I wouldn’t say no to #1 either, even if only to increase the ACE count and still keep a 2-codepoint signal for the non-counting signals.
</JH2>

I guess I don't understand the tunneling problem. If we have a mark that demands MD, that trumps feedback about low-latency fine tuning, does it not?

<JH2>
As tempting as it is to just answer “yes, of course”, I’ll acknowledge it’s not quite so clear-cut.  (And I feel like you might not be alone in not understanding this facet of the problem yet, so thanks for asking about it.  It hasn’t been very easy to follow.)

The issue as I understand it is that tunnels as they stand today can only export the CE signal, period.  (At best.  That’s if they’re RFC6040-compliant.  If we want to carry 2 signals, we’d need to obsolete RFC 6040 with a decapsulation algorithm that carries 2 signals instead, and update a lot of tunnel endpoints, hence a decade-long rollout before most tunnels have it.)

The L4S proponents have argued persistently that it’s not necessary to respond to a classic CE signal with MD, and that the value gained from the fine-tuning signal is key to the L4S value proposition, and therefore that the fine tuning should be carried in CE, regardless of the interactions with classic queues.

And it’s also fair to note that RFC 8311 introduced some significant question as to whether a mark from a classic queue now demands an MD response, so now it’s sort of a judgement call about whether we can get “effective congestion control” without honoring such a signal in the MD way that the classic queues are expecting.

So I think it’s an open question at this point about what constitutes a reasonable outcome when MD is not the response to a classic queue’s CE signal.

Some of that depends on whether you think the classic queue detection can be made successful, since it provides an alternative MD response that’s not a direct response to a CE signal from the classic queue.  (There are currently some differing opinions on how likely this is to work in practice, just in case it wasn’t complicated enough yet.)

I of course have my opinions, but I don’t know of an established consensus here, and to me it seems like the main crux of the debate over safety properties of L4S, at this point.
</JH2>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 20 May 2020 00:18:38 +0100

David,

On 19/05/2020 14:19, Black, David wrote:
> I think you may have missed the point.
>

[BB] I'll get to who is missing the point later...

First,
a) Suspending disbelief regarding ECN marking and tunnels for one moment,;
b) suspending any judgement as to whether the "ECT(1) as an output" boat 
has just sailed;
c) and setting aside terminology problems (a 1/p decrease is just as 
multiplicative as a 1/2 decrease),

> The topic under discussion is a 2-level marking system that retains 
> the semantics of CE as calling for an RFC3168-style Multiplicative 
> Decrease (MD), and uses ECT(0) [remarked from ECT(1)] as the queue 
> occupancy mark for 1/p congestion control.  With two signals (CE and 
> ECT(0)) arriving at the receiver, there need to be ways to reflect 
> both to the sender.
>
> The MD signal (CE) overrides the queue occupancy signal, as it calls 
> for a much larger backoff.
>

[BB]
The idea of 1/p is for the extent of congestion signalling to give a 
/variable/ reduction between 0 and 0.5. Not just a /small/ reduction.

RFC3168 CE mandates a once-per-RTT reduction that has been fixed 
differently in different CCs: 0.5 in Reno, 0.3 in Cubic, 0.15-0.3 in ABE 
Cubic or ABE Reno. None of these numbers is "larger" than the 0.5 that 
1/p will give when there's an "Oh sh*t" moment.

So please be correct and precise and say that a CE response is needed 
for the possibility of encountering a single-queue RFC3168 AQM /during 
transition/.

Beyond transition, a fixed 'back-stop'  response in addition to 1/p 
signalling would be completely redundant.

Protocols should be designed with their deployment progression in mind. 
This is what Jana said - the future is a lot longer than the past or the 
present. Other people prioritize this way of thinking too. When there is 
no right answer, a consensus call is designed to find out how most 
people think.

> ACK thinning should be less of a concern for MD because any arrival of 
> CE (or detection of a drop) at the receiver is supposed to cause MD at 
> the sender, and that sender MD reaction is limited to once per RTT.
>

[BB]
OK, the proposal is one-for-one feedback of CE marks, i.e. completely 
unreliable feedback delivery. That's asking the tcpm WG to overturn the 
requirement that CE feedback must be reliable [RFC7560], and to undo the 
reliable CE feedback that was in RFC3168. Doesn't sound like progress.

Remember the idea of AccECN is a single improved ECN feedback protocol 
for TCP, not a fork with different wire protocols depending on the 
congestion control. So this will be used by RFC3168 senders that only 
recognize CE, or where only CE is induced on the path.

This proposal seems to be based on the totally unproven idea that 
usually/sometimes/maybe at least one CE mark per RTT might get through 
ACK thinning or ACK loss. Why? If there's a congestion situation on the 
back channel, reverse queues will be large, with a potentially large 
fraction of ACKs lost. And thinning is typically more aggressive the 
more ACKs are queued up. It's probable/possible that the window in the 
forward direction will be reducing. Then there would be less ACKs sent 
per RTT in the first place. For RFC3168-style marking, often only a 
small proportion of the forward packets is CE-marked, even during fairly 
congested episodes on the forward path (e.g. 1%, or maybe 5-10% 
worst-case if there's also bad congestion on the forward path).

Surviving_CE_marks/RTT = window * forward_CE_marking_prob * (1 - 
ACK_loss_prob) * (1 - proportion_of_ACKs_thinned)

What makes anyone believe that feedback of at least one CE mark will get 
through per RTT in this scenario? Everything tends towards the opposite 
conclusion. Congestion feedback has to be carefully designed to survive 
congestion (the clue is in the name).

And... if that argument isn't enough (it is, actually), what evidence is 
there that all congestion controls used by real-time applications 
nowadays respond to no more than one congestion signal per RTT? The 
IETF's equation-based CC TFRC does, but most real-time congestion 
controllers respond to the level of loss or ECN probability in 
proprietary ways.

> The underlying reasons why a single codepoint may suffice to signal MD 
> support via the AccECN field are two-fold:
>
>   * The MD signal (CE) overrides the queue occupancy signal (ECT(0)),
>     as it calls for much more sender backoff, so there’s no point in
>     AccECN for queue occupancy if a CE has been received.
>

[BB] See above about being clear on why a second signal is needed - 
backstop is not the reason.

>  *
>
>
>   * The MD signal does not require the gradations of AccECN – it’s an
>     undifferentiated “Oh sh*t!” signal calling for MD backoff (and
>     reversion to RFC3168 sender behavior).
>

[BB] Again, see above - the second signal is purely for transition, so 
no need to try to convince yourself that it's also a useful backstop in 
its own right. Extent-based congestion signalling inherently contains 
its own backstop.

> That would still leave the rest of the AccECN codepoints to reflect 
> the queue occupancy signal in the absence of CE arriving or a drop 
> being detected.
>

[BB] Next, did no-one hear me say that the AccECN ACE field was 
finalized after lengthy consultations with the folks who design and 
build segmentation offload hardware? A more complex encoding of the 
3-bit ACE field than a simple counter is very unlikely to fly.

There's a large system of dependencies here. Every detail can be a 
show-stopper. You are talking to someone who has been trying out all the 
different ways of putting together this jig-saw for years. I am getting 
thoroughly ticked off with being told that I'm the one missing the point.

_________________
And of course, we can't suspend disbelief regarding tunnelling...

The way ECN interacts with protocol layering at the IP layer and below 
has to be laid down over many years, long before you can start to rely 
on it at higher layers.

Many IP-in-IP tunnels and particularly IPSec implementations comply with 
RFC6040.
You (David) helped with those liaisons we did with 3GPP and the IEEE, 
getting them to take note of RFC6040.
They, particularly the 3GPP, put a lot of work into ECN.
Then there was all the evangelizing of RFC6040 I did in loads of intarea 
WGs.
That resulted in all sorts of people going off and fixing their 
implementations.

The guess is that 60-70% of Internet paths involve some form of 
layering, tunnelling, overlay, etc. At the moment, L4S just works with 
any of the whole history of ECN tunnels: RFC3168, RFC4301 and RFC6040.

This ECT1->0 proposal would make L4S not work at all with /any/ existing 
tunnels or layerings. Instead, it proposes to obsolete and reverse 
RFC6040 (standards track) {1} and force L4S to /depend/ on deployment of 
this new tunnel behaviour before L4S will work at all (even tho it 
doesn't currently depend on RFC6040).

There are a hundred or so companies that have said they are planning to 
deploy L4S, many with products imminent, but hanging on this IETF 
decision. Under this proposal, they will then be expected to wait a few 
years (probably more like a decade) until these tunnels are designed, 
specified, implemented and deployed. Until then, L4S just doesn't work 
at all over 60-70% of the Internet.

This isn't going to happen. This is fantasy. Can you not see how this 
will look? Actually, how it does look? It looks like you are trying to 
give the IETF a reputation for shooting itself in both feet, then laying 
land mines and blowing off both legs.

We need to focus on other ways of mitigating any problems with existing 
single-queue RFC 3168 AQMs.

Bob

{1} Not to mention reversing other standards track RFCs, like RFC6660.

> Thanks, --David
>
> *From:*tsvwg <[email-masked]> *On Behalf Of *Bob Briscoe
> *Sent:* Tuesday, May 19, 2020 8:59 AM
> *To:* Martin Duke; Holland, Jake
> *Cc:* TSVWG
> *Subject:* Re: [tsvwg] Update to Position Statement on ECT(1)
>
> [EXTERNAL EMAIL]
>
> Martin,
>
>     I'm intrigued by this discussion of the ECT(1)->ECT(0) proposal,
>     as something that could definitively solve the safety concerns.
>     I'll make two unrelated points:
>
>     always dropping the packet.  Although packet loss is bad, maybe
>     some drops at the end of slow start is the tradeoff we have to
>     make to get low latency. Implementations really concerned about
>     loss can be less aggressive during slow start.
>
>     2) Clearly, there is no AccECN signaling problem for
>     ECT(1)->ECT(0) for QUIC, and for TCP paths where the option gets
>     through. It this is an issue of the three ACe bits, I think one
>     codepoint in ACE would be sufficient to indicate that a CE mark
>     was received, which IMO would trump whatever other feedback is in
>     that header.. Unless there's some sort of performance cliff in not
>     being able to encode 7 ECT(0) marks, this seems like a non-problem
>
>
> As we all try to remove more and more latency, we are all going to 
> find that the problems get harder and harder - little details matter a 
> lot. Without really solid feedback precision, you lose the consistent 
> latency of L4S.
>
> Richard Scheffenegger and I designed a 4-3-1 codepoint scheme for ACE 
> at one point (4 for CE, 3 for ECT(1) and one for Not-ECT). See 
> ACE part of AccECN to a "stalking horse" - i.e. a vehicle that works 
> for today, while also acting as a platform to get the larger AccECN 
> Option deployed for the longer term - when the ACE field might have 
> become too small for the possible future level of ACK thinning on some 
> paths.
>
> standardized, you will find all sorts of weird and wonderful 
> compressed encodings, but in the end, as well as inadequate 
> robustness, arguments for simplicity took precedence. Not just because 
> complexity breeds bugs (which is not an insignificant consideration 
> for TCP), but also because the semantics of ACE also have to be 
> supported by segmentation offload hardware.
>
> When, I highlighted the problems with tunnelling that the ECT1->0 
> scheme has, I only hinted at these problems with TCP, because I 
> thought incompatibility with tunnelling should be a sufficient argument.
>
>           > never fully understood anything beyond "don't lose CE
>         marks" and maybe
>         different from the existing AccECN draft would be needed.
>         AccECN provides accurate feedback of the number of CE marks
>         observed. Under the proposed scheme L4S would need to getting
>         accurate feedback of the number of ECT(0)  (pre-congestion /
>         some congestion) marks. AccECN would need to be re-worked to
>         provide both that and, in addition, either the existing
>         ECE/CWR handshake or something else that performs the
>         equivalent function. The most obvious solution would be to
>         repurpose NS and one or more currently reserved flag bits (or
>         use other ideas from RFC 7560 Sec 5.2) and leave ECE/CWR
>         unchaged. I note in passing the SCE proposal would have to do
>         something along the same lines (though AFAICT that has not yet
>         been fully fleshed out).
>
>         > If I understand what is being said, this is because ECT(0)
>         would become ambiguous, as it can appear either on an L4S
>         packet with a pre-congestion marking, or a non-L4S packet. 
>         Doesn't the same issue exist with the current L4S proposal for
>         CE-marked packets?
>
>         > Actually, it seems to me that this approach would yield
>         exactly the same congestion signaling capability as using
>         ECT(1) as a  pre-congestion / some congestion mark. All that
>         has been done is to reverse the role of ECT(1) and ECT(0)
>         compared to what the SCE draft and RFC 6040 envisioned. In
>         other words:
>         >
>         >      +-----+-----+
>         >      | ECN FIELD |
>         >      +-----+-----+
>         >         0     0        Not-ECT
>         > Jake, you said that the three issues discussed above --
>         tunnels, AccECN, and multiple AQMs in the path are "a few of
>         the known tradeoffs." What are the others?
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: Martin Duke <[email-masked]>
Date: Tue, 19 May 2020 17:51:09 -0700

Hi Jake,

First, thanks to Bob for the RFC 6040 reference. That cleared my confusion
about the tunnel problem up right away.

Jake, I like your frame of the interesting trade-off between losing the L4S
signal and losing the 3168 signal. However, let me add yet another concern
about ECT1->0: by design, packets so marked may arrive *much* later than
neighbors not marked, which means they are likely to be declared lost
anyway. Do you see this as a problem, or post bottleneck is the latency
difference likely small?

Martin

On Tue, May 19, 2020, 13:13 Holland, Jake <[email-masked]> wrote:

> *From: *Martin Duke <[email-masked]>
> *Date: *Tuesday, May 19, 2020 at 10:21 AM
>
> As I understand it, this is the current L4S MD signal if the LL queue
> overflows into the low latency queue and has congestion there.
>
>
>
> I agree this works as far as it goes, but the safety problem that seems to
> need an MD signal is the interaction with existing queue implementations
> that know nothing about L4S, and have a pre-existing interpretation of
> ECT(1).
>
>
>
> (Even for FQ, the interaction is very poor, but shared queues is where the
> poor interaction is arguably a major safety concern.  Mostly nobody seems
> to care much about the occasional hash collision in FQ that would have the
> same problem, because most FQs are 1024 slots, so after birthday paradox
> considerations it’s only 1/32 or so of the competing flows.)
>
>
>
> Anyway, if there was sufficiently good consensus that these classic queues
> are irrelevant, then deprecating them responsibly is a potentially viable
> way forward, but that’s not what any of the current BCPs or standards say
> today, I think.
>
> </JH2>
>
>
>
>
>
> You don’t need very many CWRs, and it might be possible to clean up the
> specs a bit about when exactly they should be sent (like maybe they’re only
> appropriate when there’s a MD event, and maybe they shouldn’t be sent for
> linear reductions during a fine-tuning response, now that we’re
> contemplating such a thing).
>
>
>
> So I was thinking for bidirectional traffic that has the newly suggested
> ACE signaling (with a codepoint for ECE and CWR, plus 6 count slots), you
> could just put CWR at the top priority, so the (only occasional) CWR would
> overrule either a latched ECE that’s still in progress or one of the
> fine-tuning signal slots, for the occasional packet reporting an MD event.
> (The logic from 3168 still holds here: if CWR is on a data packet and the
> data packet is lost, you’ll still get an MD response, so it doesn’t need to
> be reliable like ECE, and the counter makes it so skipping a packet doesn’t
> lose the fine-tuning signal, since it’s no worse than thinning the acks
> would have been.)
>
>
>
> I do worry a bit whether a 6-count is enough to detect rollover well, but
> like with the ESCE approach for reflecting the fine-tuning signal, losing
> some accuracy is acceptable if we have a reliable MD signal, so I think
> it’s probably ok.
>
>
>
> I wouldn’t say no to #1 either, even if only to increase the ACE count and
> still keep a 2-codepoint signal for the non-counting signals.
>
> </JH2>
>
>
>
>
>
> As tempting as it is to just answer “yes, of course”, I’ll acknowledge
> it’s not quite so clear-cut.  (And I feel like you might not be alone in
> not understanding this facet of the problem yet, so thanks for asking about
> it.  It hasn’t been very easy to follow.)
>
>
>
> The issue as I understand it is that tunnels as they stand today can only
> export the CE signal, period.  (At best.  That’s if they’re
> RFC6040-compliant.  If we want to carry 2 signals, we’d need to obsolete
> RFC 6040 with a decapsulation algorithm that carries 2 signals instead, and
> update a lot of tunnel endpoints, hence a decade-long rollout before most
> tunnels have it.)
>
>
>
> The L4S proponents have argued persistently that it’s not necessary to
> respond to a classic CE signal with MD, and that the value gained from the
> fine-tuning signal is key to the L4S value proposition, and therefore that
> the fine tuning should be carried in CE, regardless of the interactions
> with classic queues.
>
>
>
> And it’s also fair to note that RFC 8311 introduced some significant
> question as to whether a mark from a classic queue now demands an MD
> response, so now it’s sort of a judgement call about whether we can get
> “effective congestion control” without honoring such a signal in the MD way
> that the classic queues are expecting.
>
>
>
> So I think it’s an open question at this point about what constitutes a
> reasonable outcome when MD is not the response to a classic queue’s CE
> signal.
>
>
>
> Some of that depends on whether you think the classic queue detection can
> be made successful, since it provides an alternative MD response that’s not
> a direct response to a CE signal from the classic queue.  (There are
> currently some differing opinions on how likely this is to work in
> practice, just in case it wasn’t complicated enough yet.)
>
>
>
> I of course have my opinions, but I don’t know of an established consensus
> here, and to me it seems like the main crux of the debate over safety
> properties of L4S, at this point.
>
> </JH2>
>
>
>

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 20 May 2020 01:32:14 +0000

Hi Martin,

> Jake, I like your frame of the interesting trade-off between losing the L4S signal and losing the 3168 signal. However, let me add yet another concern about ECT1->0: by design, packets so marked may arrive *much* later than neighbors not marked, which means they are likely to be declared lost anyway. Do you see this as a problem, or post bottleneck is the latency difference likely small?

My take is that this basically happens only in a dualq (leaving aside other causes of reordering), and the classic portion of a dualq is not a deep dumb fifo, but rather a ordinary PIE AQM with a good delay response like normal classic PIE AQMs.  So the time difference that’s added by a single such queue is what you’d expect from a normal classic PIE AQM, if you have multiple loaded links (which by the way we’ve seen argued is quite uncommon, at least when L4S proponents are supporting the notion that mis-classifying classic CE-marked packets isn’t a problem).

According to the same charts, such a PIE queue tends to vary from up to what looks like about 5ms at near the median to about 50ms at the 5 9’s level:
https://datatracker.ietf.org/meeting/105/materials/slides-105-tsvwg-sessa-5-l4s-presentation-00.pdf#page=2

This time difference is well below RTO, but could easily hit a fast rexmit from dupacks if the L4S endpoint isn’t using RACK, or if it is using RACK it could reasonably at the higher end fall outside the RACK reordering window if it’s a short enough path, since the reordering window at min_rtt/4, according to https://tools.ietf.org/html/draft-ietf-tcpm-rack-08#section-8.3.  (And of course this can cascade if there are multiple queues, so at, say, 10-15 such loaded bottlenecks when the first one gets a high-fi mark, you might be able to make it start to bump into RTO.)

However, IIRC previous review found that a fast retransmit in a modern linux would get retroactively fixed up to be a non-loss signal when the late ack arrives, though you could end up with some extra traffic generated when it happens.

So to me the reordering issue doesn’t look offhand like a likely stopper for this, though it’s certainly awkward when there’s multiple loaded paths in sequence and this is a clear downside for this signaling strategy, regardless of whether people agree that the MD signal is the more important of the 2 signals to keep in tunneled paths.  The similar worry over classic CE reordering got a lot of discussion IIRC, but I think the resolution concluded that it wasn’t a big problem.  However, I think this would be somewhat worse, and could also get some similar discussion, even if this otherwise looks like a worthwhile path.

Caveat emptor: this is all off-the-cuff reasoning, I haven’t tested any of this of course.  Just laying out the line of reasoning as I understand it, but if anybody ever tests this, it could reasonably show out problems I haven’t thought of.  I do think it would be potentially quite a bit of reordering sometimes in paths with multiple dualqs.

Best,
Jake

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 20 May 2020 19:59:06 +0100

Jake, Martin,

On 20/05/2020 02:32, Holland, Jake wrote:
>
> Hi Martin,
>
> > Jake, I like your frame of the interesting trade-off between losing 
> the L4S signal and losing the 3168 signal. However, let me add yet 
> another concern about ECT1->0: by design, packets so marked may arrive 
> *much* later than neighbors not marked, which means they are likely to 
> be declared lost anyway. Do you see this as a problem, or post 
> bottleneck is the latency difference likely small?
>
> My take is that this basically happens only in a dualq (leaving aside 
> other causes of reordering), and the classic portion of a dualq is not 
> a deep dumb fifo, but rather a ordinary PIE AQM with a good delay 
> response like normal classic PIE AQMs.  So the time difference that’s 
> added by a single such queue is what you’d expect from a normal 
> classic PIE AQM, if you have multiple loaded links (which by the way 
> we’ve seen argued is quite uncommon, at least when L4S proponents are 
> supporting the notion that mis-classifying classic CE-marked packets 
> isn’t a problem).
>
> According to the same charts, such a PIE queue tends to vary from up 
> to what looks like about 5ms at near the median to about 50ms at the 5 
> 9’s level:
>
> This time difference is well below RTO, but could easily hit a fast 
> rexmit from dupacks if the L4S endpoint isn’t using RACK, or if it is 
> using RACK it could reasonably at the higher end fall outside the RACK 
> reordering window if it’s a short enough path, since the reordering 
> window at min_rtt/4, according to 
> https://tools.ietf.org/html/draft-ietf-tcpm-rack-08#section-8.3. (And 
> of course this can cascade if there are multiple queues, so at, say, 
> 10-15 such loaded bottlenecks when the first one gets a high-fi mark, 
> you might be able to make it start to bump into RTO.)
>
> However, IIRC previous review found that a fast retransmit in a modern 
> linux would get retroactively fixed up to be a non-loss signal when 
> the late ack arrives, though you could end up with some extra traffic 
> generated when it happens.
>
> So to me the reordering issue doesn’t look offhand like a likely 
> stopper for this, though it’s certainly awkward when there’s multiple 
> loaded paths in sequence and this is a clear downside for this 
> signaling strategy, regardless of whether people agree that the MD 
> signal is the more important of the 2 signals to keep in tunneled 
> paths. The similar worry over classic CE reordering got a lot of 
> discussion IIRC, but I think the resolution concluded that it wasn’t a 
> big problem.  However, I think this would be somewhat worse, and could 
> also get some similar discussion, even if this otherwise looks like a 
> worthwhile path.
>

It's worth revisiting
"Risk of reordering Classic CE packets" in the appendix here: 
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#appendix-B.1
It explains the likelihood of harm if a packet that was originally sent 
as ECT0 from a 3168 sender gets marked CE at a first bottleneck, then 
gets classified into the L queue instead of the C queue at a subsequent 
dualQ coupled AQM.

I've quoted it below (indented), then added in commentary on how it 
would change for ECT1->0 (not indented).

The main difference is that ECT0->CE causes early reordering, whereas 
ECT1->0 causes late reordering. We designed L4S this way round because 
early reordering requires other ducks to all sit in a row before it 
causes any harm, whereas late reordering would generally cause harm at 
every marking.

    1.  It is quite unusual to experience queuing at more than one
           bottleneck on the same path (the available capacities have to
           be identical).

Still true.

       2.  In only a subset of these unusual cases would the first
           bottleneck support Classic ECN marking while the second
           supported L4S ECN marking, which would be the only scenario
           where some ECT(0) packets could be CE marked by an AQM
           supporting Classic ECN then the remainder experienced further
           delay through the Classic side of a subsequent L4S DualQ AQM.

For ECT0->CE, this problem exists only during transition from RFC3168 to 
L4S, but goes away if the L4S experiment succeeds in superseding 3168.
For ECT1->0, this problem occurs when there is more than one L4S 
bottleneck on a path (as opposed to a 3168 bottleneck followed by an L4S 
bottleneck). So it persists even after 3168 traffic on the Internet has 
reduced to near zero.

As 3168 sunsets, there would rarely be traffic in the C queue. However, 
packets marked ECT1->0 at a first AQM, would get held up in the second C 
queue while those around them were scheduled in front of them (assuming 
the rest of the flow and potentially other traffic in the L queue was 
keeping it busy).

* With a WRR scheduler weighted as we recommend, these ECT0 packets 
would pop out usually 15 packets late.
* With a Time-shifted FIFO, they would wait for the time shift - we 
recommend 4*target in the dualQ draft, where target=15ms, i.e. 60ms.

       3.  Even then, when a few packets are delivered early, it takes
           very unusual conditions to cause a spurious retransmission, in
           contrast to when some packets are delivered late.  The first
           bottleneck has to apply CE-marks to at least N contiguous
           packets and the second bottleneck has to inject an
           uninterrupted sequence of at least N of these packets between
           two packets earlier in the stream (where N is the reordering
           window that the transport protocol allows before it considers
           a packet is lost).

* ECT0->CE promotes C packets to L, causing early reordering, which 
leads to no harm except in the above extremely unlikely condition of N 
contiguous CE-marked packets. Also, the marking has to have been applied 
by a classic AQM that uses a low marking probability.
* For ECT1->0, the reordered packets are later, not earlier. So, if the 
reordering exceeds the reordering window, each delayed packet 
immediately causes a spurious retransmission. Also, the upstream marking 
would be applied by an L4S AQM, that often applies a higher marking 
probability, including frequent long runs of 100% marking.

Where more than one packet in a row was wrongly classified into the C 
queue, the delays for the later packets would accumulate. For instance, 
a WRR scheduler would release one C packet for every 15 L packets, even 
if the C packets were contiguous. In the other direction, the more L 
packets wrongly classifed into C, the more the L queue would have time 
to drain and leave a gap for a C packet to be released without waiting 
to be scheduled explicitly.

              For example consider N=3, and consider the sequence of
              packets 100, 101, 102, 103,... and imagine that packets
              150,151,152 from later in the flow are injected as follows:
              100, 150, 151, 101, 152, 102, 103...  If this were late
              reordering, even one packet arriving 50 out of sequence
              would trigger a spurious retransmission, but there is no
              spurious retransmission here, with early reordering,
              because packet 101 moves the cumulative ACK counter forward
              before 3 packets have arrived out of order.  Later, when
              packets 148, 149, 153... arrive, even though there is a
              3-packet hole, there will be no problem, because the
              packets to fill the hole are already in the receive buffer.

       4.  Even with the current TCP recommendation of N=3 [RFC5681]
           spurious retransmissions will be unlikely for all the above
           reasons.  As RACK [I-D.ietf-tcpm-rack] is becoming widely
           deployed, it tends to adapt its reordering window to a larger
           value of N, which will make the chance of a contiguous
           sequence of N early arrivals vanishingly small.

ECT1->0 causes harm whether or not the reordered packets are contiguous, 
so this factor does not reduce the chance of harm.

       5.  Even a run of 2 CE marks within a Classic ECN flow is
           unlikely, given FQ-CoDel is the only known widely deployed AQM
           that supports Classic ECN marking and it takes great care to
           separate out flows and to space any markings evenly along each
           flow.

Ditto.

       It is extremely unlikely that the above set of 5 eventualities
       that are each unusual in themselves would all happen
       simultaneously.  But, even if they did, the consequences would
       hardly be dire: the odd spurious fast retransmission. Admittedly
       TCP (and similar transports) reduce their congestion window when
       they deem there has been a loss, but even this can be recovered
       once the sender detects that the retransmission was spurious.

In summary,
* There is only one unlikely condition (no.1) for spurious 
retransmissions as a result of ECT1->0.
* Whereas ECT0->CE only causes spurious retransmissions if all 5 
unlikely or extremely unlikely conditions occur together.

I didn't quote a figure for condition #1, 'cos the only paper I found 
that could quantify it, describes how to get a figure, but doesn't 
really have any sound experimental results.

Deng, L. & Kuzmanovic, A., "Pong: Diagnosing Spatio-temporal Internet 
Congestion Properties," SIGMETRICS Perform. Eval. Rev. 35(1):381--382 
ACM (June 2007)

Having engaged with this argument, I do want to emphasize that the 
tunnel argument causes ECT1->0 not to work at all over 60-70% of the 
Internet for many years.

I find it weird that this thread is trying to count how many angels can 
dance on the point of a pin, when there's the tunnelling elephant 
standing in the room, squashing all the angels and bending the pin.

Bob

> Caveat emptor: this is all off-the-cuff reasoning, I haven’t tested 
> any of this of course.  Just laying out the line of reasoning as I 
> understand it, but if anybody ever tests this, it could reasonably 
> show out problems I haven’t thought of.  I do think it would be 
> potentially quite a bit of reordering sometimes in paths with multiple 
> dualqs.
>
> Best,
>
> Jake
>

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 21 May 2020 12:50:34 +0000

Hi Bob,

For brevity and clarity, I've trimmed and reordered quotes to highlight
the relevant bits I'm responding to, leaving aside discussion on some
of the lower level details you raised to focus on a higher level point.
But for easier reference in case there's questions of context, the full
preceding message is here:
https://mailarchive.ietf.org/arch/msg/tsvwg/-uFazVf2tupkK011Y5ps83ENCvc/

> From: Bob Briscoe <[email-masked]>
> Date: Tuesday, May 19, 2020 at 4:18 PM
...
> land mines and blowing off both legs.

This sounds to me like “whoops, we have made a terrible mistake by
defining the tunnel semantics in such a way that only one wire signal
(CE) can be transported, and now we realized we would like to provide a
signal with a different meaning, and unfortunately are painted into a
corner, since CE’s meaning is defined and used in widely distributed
implementations that are currently engaged in a long-overdue rollout”.

If there is a "fantasy" here, it's this notion that we'll quickly just
breeze through a transition period before classic ECN is phased out, and
so the interoperability problems don't need a serious and long-term
solution, unless I'm misunderstanding the meaning of the "transition"
you've referred to with remarks like these:

  > its own right. Extent-based congestion signalling inherently contains 

What seems to be suggested here sounds like it requires a flag day, unless
the interoperability problems we've seen, which come as a direct consequence
of the proposed major change in the meaning of CE, have a robust solution.

> single-queue RFC 3168 AQMs.

Perhaps the right choice is to deprecate classic ECN in shared queues and
have the flag day.  Put that to a consensus call, and if it's really such
a small footprint of people who would be affected, maybe it will fly.

But I do not think we can bless the deployment of something incompatible
without taking this step, and right now it looks like the L4S CE signal and
its associated response is fundamentally incompatible with the classic CE
signal, because of the effect it has on the classic queue.  (And while it's
great that FQ comes with a built-in partial mitigation for the non-compliant
traffic, the core problem here is the signaling incompatibility.)

To me it seems irresponsible to disregard the decades of work that is
now, finally, beginning to come to fruition from people turning on the
widely available and already deployed implementations that only need
a config update to enable, just in time to tell everybody they should
stop doing that and upgrade their hardware and endpoints to this new
idea.

I sincerely wish this problem had gotten more focus early enough for
the companies that have imminent products to realize this proposal
would have serious interoperability issues that now seem intractable.
But that doesn't make it OK to ship this stuff without either organizing
the flag day or finding a robust solution.

Nonetheless, there is some very promising work here, and some real value
looks like it can be salvaged, with time and effort.  That's what's
underpinning my efforts to discuss ways to get the new signal exported
in an incrementally deployable way that wouldn't suffer the interop
problems.

Upgrading the tunneling semantics to provide multiple signals could be
a step forward that could help solve this problem, and might make this
kind of thing possible to get to eventually without needing the flag
day.  It's not fast, but as you point out, the future is much bigger
than the present.

But attempting to wipe out the existing meaning of CE by fiat is a
disaster in the making.  It would wipe out vastly more value than the
unfortunate delay to the immediate plans of the early L4S adopters,
because it seems likely to result in bleaching of the classifier
codepoint that might never go away.

Best regards,
Jake

---

From: "C. M. Heard" <[email-masked]>
Date: Thu, 21 May 2020 20:06:46 -0700

On Thu, May 21, 2020 at 5:51 AM Holland, Jake wrote:
> > There are a hundred or so companies that have said they are planning
> > to deploy L4S, many with products imminent, but hanging on this IETF
> > decision. Under this proposal, they will then be expected to wait a
> > few years (probably more like a decade) until these tunnels are
> > designed, specified, implemented and deployed. Until then, L4S just
> > doesn't work at all over 60-70% of the Internet.
> >
> > This isn't going to happen. This is fantasy. Can you not see how
> > this will look? Actually, how it does look? It looks like you are
> > trying to give the IETF a reputation for shooting itself in both
> > feet, then laying land mines and blowing off both legs.
>
> (CE) can be transported [...]"

While I broadly agree with Jake's points, I don't think this last part
is quite right. Some of the trimmed text provided what I understand to
be the principal objection to the ECT1->0 proposal:

> > layering, tunnelling, overlay, etc. At the moment, L4S just works
> > with any of the whole history of ECN tunnels: RFC3168, RFC4301 and
> > RFC6040.
> >
> > This ECT1->0 proposal would make L4S not work at all with /any/
> > existing tunnels or layerings. Instead, it proposes to obsolete and
> > reverse RFC6040 (standards track) {1} and force L4S to /depend/ on
> > deployment of this new tunnel behaviour before L4S will work at all
> > (even tho it doesn't currently depend on RFC6040).
> > [...]

RFC 6040 adopted the rule that ECT(1) overrides ECT(1) at tunnel egress
because ECT(1) was used at the threshold mark in the pre-congestion
notification work leading up to RFC 6660. So indeed there is standards
track work that would have to be deprecated in order for the ECT1->0
proposal to move forward. The choice of ECT(1) rather than ECT(0) as
the PCN threshold mark (and the resulting tunnel egress behavior in
RFC 6040) was a natural choice at the time and is perfectly compatible
with SCE (without the bits reversed). It was in hindsight a suboptimal
choice, but that was not in any was the fault of the L4S community.

With that said ...

> breeze through a transition period before classic ECN is phased out,
> and so the interoperability problems don't need a serious and
> long-term solution, unless I'm misunderstanding the meaning of the
> "transition" you've referred to with remarks like these:
>
>   > So please be correct and precise and say that a CE response is
>   > needed for the possibility of encountering a single-queue RFC3168
>   > AQM /during transition/.
>   >
>   > Beyond transition, a fixed 'back-stop'  response in addition to
>   > 1/p signalling would be completely redundant.
>
>   > [BB] Again, see above - the second signal is purely for
>   > transition, so no need to try to convince yourself that it's
>   > also a useful backstop in its own right. Extent-based congestion
>   > signalling inherently contains its own backstop.

These remarks echo the theme I've heard in the comments from many of
the L4S advocates who spoke at the interim meeting on 2020 April 27
and seem to me to rest on a firm belief that L4S signalling offers
such a dramatic over classic RFC 3168 that it will sweep the latter
aside. I'm not convinced that this is in fact true, but if it is ...

> Perhaps the right choice is to deprecate classic ECN in shared queues
> and have the flag day.  Put that to a consensus call, and if it's
> really such a small footprint of people who would be affected, maybe
> it will fly.

+1

I see that the consensus call is to proceed with ECT(1) as an input and
develop L4S. I hope that classic AQM detection really can be made to
work as lot better than I now believe.

Mike Heard

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sun, 24 May 2020 08:50:19 +0200

David, Jake,

Just to be completely clear here:

The proposed signal is, that a receive can latch one codepoint,
indicating CE was received (currently ECE). And that latch is
maintained, until the sender - successfully - conveyes, that it did in
fact react to the CE signal (currently CWR)...

So, any signal for fractional queue occupancy during this window can no
longer signaled in the TCP header, only in the AccECN options.

And if the CWR-like codepoint is missed, ignored (like a least one stack
currently does at times), dropped, another window of fractional feedback
is gone (in the TCP header) while another Multiplicative Decrease
reaction has to happen.

Maybe I missed this, but under the SCE scheme, I was assuming the 1/p
signal was ECT(1), not ECT(0). Or have we all agreed already, that
ECT(1) is the "new" network input signal (both SCE and L4S), but an AQM
may re-mark it into ECT(0) or CE?

I bring this up because I apparently missed the context around "and uses
ECT(0) [remarked from ECT(1)]".

Regards,
    Richard

Am 19.05.2020 um 15:19 schrieb Black, David:
> I think you may have missed the point.  The topic under discussion is a
> 2-level marking system that retains the semantics of CE as calling for
> an RFC3168-style Multiplicative Decrease (MD), and uses ECT(0) [remarked
> from ECT(1)] as the queue occupancy mark for 1/p congestion control.
> With two signals (CE and ECT(0)) arriving at the receiver, there need to
> be ways to reflect both to the sender.
>
> The MD signal (CE) overrides the queue occupancy signal, as it calls for
> a much larger backoff.  ACK thinning should be less of a concern for MD
> because any arrival of CE (or detection of a drop) at the receiver is
> supposed to cause MD at the sender, and that sender MD reaction is
> limited to once per RTT.
>
>   * The MD signal (CE) overrides the queue occupancy signal (ECT(0)), as
>     it calls for much more sender backoff, so there’s no point in AccECN
>     for queue occupancy if a CE has been received.
> That would still leave the rest of the AccECN codepoints to reflect the
> queue occupancy signal in the absence of CE arriving or a drop being
> detected.
>
> Thanks, --David
>
>     something that could definitively solve the safety concerns. I'll
>     make two unrelated points:
>
>     drops at the end of slow start is the tradeoff we have to make to
>     get low latency. Implementations really concerned about loss can be
>     less aggressive during slow start.
>
>     for QUIC, and for TCP paths where the option gets through. It this
>     is an issue of the three ACe bits, I think one codepoint in ACE
>     would be sufficient to indicate that a CE mark was received, which
>     IMO would trump whatever other feedback is in that header.. Unless
>     there's some sort of performance cliff in not being able to encode 7
>     ECT(0) marks, this seems like a non-problem
>
>
>         both that and, in addition, either the existing ECE/CWR
>         handshake or something else that performs the equivalent
>         function. The most obvious solution would be to repurpose NS and
>         one or  more currently reserved flag bits (or use other ideas
>         from RFC 7560 Sec 5.2) and leave ECE/CWR unchaged. I note in
>         passing the SCE proposal would have to do something along the
>         same lines (though AFAICT that has not yet been fully fleshed out).
>
>         would become ambiguous, as it can appear either on an L4S packet
>         with a pre-congestion marking, or a non-L4S packet.  Doesn't the
>         same issue exist with the current L4S proposal for CE-marked
>         packets?
>
>         exactly the same congestion signaling capability as using ECT(1)
>         as a  pre-congestion / some congestion mark. All that has been
>         done is to reverse the role of ECT(1) and ECT(0) compared to
>         what the SCE draft and RFC 6040 envisioned. In other words:
>          >
>          >      +-----+-----+
>          >      | ECN FIELD |
>          >      +-----+-----+
>          >         0     0        Not-ECT
>         tunnels, AccECN, and multiple AQMs in the path are "a few of the
>         known tradeoffs." What are the others?
>

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 24 May 2020 13:54:38 +0300

> On 24 May, 2020, at 9:50 am, Scheffenegger, Richard <[email-masked]> wrote:
> 

Jake's proposal reverses the sense of ECT(0) and ECT(1) relative to SCE.  His aim is to give the backwards compatibility of SCE whilst retaining the identifying power of setting ECT(1) at origin.  You may wish to re-read older posts from him to get more detail.

 - Jonathan Morton

---

From: "C. M. Heard" <[email-masked]>
Date: Sun, 24 May 2020 06:17:57 -0700

> RFC 6040 adopted the rule that ECT(1) overrides ECT(1) at tunnel egress

That should have said " ECT(1) overrides ECT(0) at tunnel egress"

Mike Heard

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 28 May 2020 04:46:28 +0000

Hi Bob,

Again, I'm trimming down to the summary in hopes of getting
better clarity, but for reference the full message I'm replying
to is here:
https://mailarchive.ietf.org/arch/msg/tsvwg/h6DPXNbPR9kXXzsZTuD9x7_Bs88/

I just wanted to respond to a few points.

> From: Bob Briscoe <[email-masked]>
> Date: Wednesday, May 20, 2020 at 11:59 AM
...
> unlikely or extremely unlikely conditions occur together.

I roughly agree with the analysis you gave; I do think you'd get
more spurious retransmissions in the case of multiple loaded
dualqs in series using the 2-signal scheme that Martin and I were
discussing than using the scheme in l4s-id.  (And thanks for
the references and the deeper dive, and the engagement with this
discussion.)

But the thing to compare is the overall consequences between the
2 schemes, of which reordering is one part.  (Not the direct
comparison of the reordering differences in a vacuum.)

The 2-signal scheme under discussion provides one major benefit,
namely that it would not redefine the meaning of a ECT(1)->CE
transition, and thus would not need the flag day I argued the
current L4S spec would need in my other recent message here:
https://mailarchive.ietf.org/arch/msg/tsvwg/THuFmguARLFQHWXG_VOyt0Wj5R8/

> tunnel argument causes ECT1->0 not to work at all over 60-70% of the 

I also agree the tunnel problem is a bigger downside of the proposed
2-signal scheme than the reordering issues.

You convinced me tunnels are a significant and serious issue, which
is in large part the driver on this discussion, and why I framed it
as possibly coming down to a choice between the 1/p signal and the
1/sqrt(p) signal, if we can't have both safely out of a tunnel.

However, organizing a flag day in order to enable an experiment that's
incompatible with a standard as widely implemented as RFC 3168 is a
heavy lift, and that proposal might not achieve the kind of consensus
that would be necessary to move that forward.

If no solution can be found that makes L4S's use of CE compatible with
RFC 3168 queues sharing with classic traffic, and no flag day can be
organized to responsibly deprecate the existing meaning of the
ECT(1)->CE transition, then the question becomes how to do something
worthwhile.  Even if that means starting with the 30-40% of the
internet that *can* be addressed in the near term without breaking
the one signal (CE) that can currently be passed through a tunnel
egress (with an eventual path forward over time if it proves useful).

This is the context where I thought our examination of the reordering
consequences made sense, in terms of thinking through the tradeoffs for
a proposal that does not redefine CE.

I'm not sure whether you're saying it's not worth doing anything at all
if the L4S 1/p redefinition of CE can't be deployed, but if you are I'd
tend to disagree.  A 1/p congestion signal seems worthwhile to try to
make happen and to use in the places where it can be used, even if those
are limited at first.  I just don't see it as clear-cut that the CE
semantic redefinition is going to be able to move forward, given the
poor interaction with existing implementations and the fundamentally
fragile heuristics in the proposed classic queue detection solution
(not to mention the poor behavior under testing that the SCE team and
your own collaborators have observed).

As I think Gorry put it, we're faced with a shortage of bits here, so
we might have to take a choice that's not as globally optimal as it
could be if we could change deployed systems whenever we wanted.  A
flag day is one path forward if there's a sufficiently broad consensus,
but I'm not convinced it'll be doable.

And I'd tend to be against it today, since I think we should keep
expanding on the deployment of the existing queues.  But I'm not
opposed to a standards track spec that says to treat ECT(1) as NECT
instead of as ECT(0), I think that's a fine idea that could open up
the possibility of using ECT(1) the way you want within a decade or so,
as support rolls out for avoiding the ambiguous signaling.  So it's
another reasonable path forward that might make L4S-style signaling
possible to set up for success down the road.

But again, it doesn't solve the near term problem of companies that
would like to get something rolling out sooner, but have so far been
working with a signal that's an incompatible use of an existing signal.

So to me it still seems worth considering a 2-signal approach first,
since it works right away for 30-40% of paths, doesn't break regular
AQM and CE (which, I must emphasize again, is already widely available
and finally in use at endpoints, and does most of the latency job
compared with drop-tail, which is the proper point of comparison for
the internet paths that are suffering from queuing delay today), and
could move forward with a longer term incremental improvement from
there.

Short of a magical bitless packet classifier, I'm not sure what other
options can help achieve anything that improves the congestion
signaling fidelity at internet scope in the near term.

Best regards,
Jake

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 28 May 2020 04:52:26 +0000

Hi Richard,

> longer signaled in the TCP header, only in the AccECN options.

Correct, that's my understanding.  While the ECE signal is alive there
would be no ACE signal containing a count for the lower level of
congestion marks.

> is gone (in the TCP header) while another Multiplicative Decrease

As RFC 3168 explains, if there is a loss of a data packet containing CWR,
there should be a MD response anyway.

However, yes, if the receive stack ignores CWR (or if it's carried in a
pure ack packet that gets ack-thinned) there would be a poor response, and
I think this approach does not itself do anything to fix the failure mode
you reported observing a few months back.  So I think that would still
need fixing in probably the same way that the handling of an RFC 3168 CWR
needs.

> ECT(0) [remarked from ECT(1)]".

Right, as Jonathan said, the proposal under discussion was to reverse the
ECT(1) and ECT(0) from SCE, but to provide 2 signals from the network in
essentially the same way that SCE proposed (so endpoints that are capable
would send ECT(1), and a 1/p high-fidelity signal would be issued by a
dualq or others by changing ECT(1) to ECT(0), rather than by changing
ECT(1) to CE as an L4S dualq does).

My goal here was to consider an approach that doesn't have to change the
meaning of CE in a way that produces a response incompatible with existing
AQMs, that still can provide a high-fidelity congestion signal.  Since
swapping the codepoints from SCE also provides a good classifier at the
first AQM, as many people pointed out is a key requirement for a dualq to
function, it seemed worth considering.

The initial outline on that proposal was here:
https://mailarchive.ietf.org/arch/msg/tsvwg/vW8eEt7V58SWPoLIJ5dklCG4WqM/

My initial response to Martin's follow-up suggestion to reserve 1 codepoint
from ACE for ECE was here, following Martin's "2) Clearly" point:
https://mailarchive.ietf.org/arch/msg/tsvwg/uplXgsXm4d7oieQGkkGU6fG0CaE/

(There has been some subsequent discussion as well.)

HTH, and sorry for the confusing web of threads.

Best,
Jake

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 28 May 2020 05:35:09 +0000

Hi Mike,

> be the principal objection to the ECT1->0 proposal:

Thanks for the correction, Mike.  I agree having now looked closer that
in Section 4.2 of RFC 6040 an ECT(0)->ECT(1) transition in the outer
header is preserved at tunnel egress, but not an ECT(1)->ECT(0) signal.
https://tools.ietf.org/html/rfc6040#section-4.2

I guess with hindsight it's not clear to me why the outer signal
shouldn't have overwritten either one of those transitions for better
future compatibility with signals that hadn't yet been defined, but I
can see how in the context of PCN it would have been an unclear
decision, and introduces some risk of undoing a pre-congestion mark
in the inner tunnel's PCN domain, which has to be considered a downside.

> proposal to move forward.

Agreed, I didn't mean to try to sweep that under the rug, and sorry if
that's how it came out.  Thanks for highlighting the point.

However, although I could be mistaken here, I don't think these points
would pose the same risks to effective congestion control that the
deprecation of the proposed RFC 3168 ECT(1)->CE transition (or more
precisely, the associated response of an L4S-compatible scalable flow
going through a classic queue and responding as if it's dualq marking;
it's not just TCP Prague, I think it's inherent to the generalized 1/p
response of the expected scalable congestion controllers described in
section 2.1 of aqm-dualq-coupled-11.)

> The choice of ECT(1) rather than ECT(0) as

Agreed, I didn't mean to imply any fault here.

Anyway, thanks for the overall support on the broader points raised,
it's nice to hear that someone thinks I'm making some sense on some
parts of this thread.  I'm not always completely sure from this
side... :)

Best regards,
Jake

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 May 2020 09:57:13 +0200

Hi Jake, all,

I have a question regarding tunneling, that IMHO is left unanswered so far. Why is the rule for ECN and tunneling not simply: copy the inner to outer on encapsulation and the outer to inner on decapsulation. Period. Sure this will catch a few corner cases, but arguably all those are currently faulty anyway. This gets rid of all code point precedence games in the current RFC's, has a simple predictable behavior, that might eve allow to diagnose mis-configured tunnel entry/exit points. 
	Yes, this might wash away ECN code points if the tunnel is explicitly configured to re-map the ECN bitfield to 00 on entry (or exit), but that is life*... (as far as I can tell re-map to all zero is against current RFC recommendations already). Keeping things simple, predictable and easy to implement (like unconditionally copy inner2outer on entry, outer2inner on exit) seems IMHO better than trying to put too much smarts into the network....

Best Regards
	Sebastian

*) Assuming that Bob's assertion that CE travels tunnels mostly intact already, this leaves a only faulty encapsulations as choking points for ECT([0|1]), I would argue that localizing and fixing the offending nodes is a more productive way forward than trying to come up with sophisticated rules for clever ECN transformations on en- and decapsulation that aim to fix up faulty behavior post-hoc, no?
The beauty is that this simple rule will work with all schemes, as long as they do not insist upon redefining the meaning of an existing signal, so draft L4S, would not really work well with this (assuming an rfc3168 AQM on the tunneled path), while Jake's modified L4S and SCE would gracefully deal with that situation.

> On May 28, 2020, at 07:35, Holland, Jake <jholland=[email-masked]> wrote:
> 
> Hi Mike,
>

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 28 May 2020 08:21:20 +0000

Hi Sebastian,

On 5/28/20, 12:57 AM, "Sebastian Moeller" <[email-masked]> wrote:
> I have a question regarding tunneling, that IMHO is left unanswered
> so far. Why is the rule for ECN and tunneling not simply: copy the
> inner to outer on encapsulation and the outer to inner on
> decapsulation. Period.

Please see RFC 6040.  Section 7 gives the design principles,
and Section 4 describes the current standard rules.  The Intro
also reviews some of the other considerations that went into it.

Of course there are some important considerations toward ensuring
that CE is not erased on the inner or outer packet, regardless
of whatever network condition led to that occurring, if it
occurs, which seemed to get a bit more attention than the ECT(1)
overwriting.  But as far as I can tell, the overall reasoning is
more or less laid out there.

If you're looking for a deeper "why", I guess there's also the
comments in the writeup and final reviews:
https://datatracker.ietf.org/doc/rfc6040/writeup/
https://datatracker.ietf.org/doc/rfc6040/ballot/

Probably also there's some discussion in the list archives around
that time if you care to search for it.

But it seems the consensus at the time landed on a copy of outer
to inner for ECT(1) overwriting ECT(0) on decapsulation, but not
for ECT(0) overwriting ECT(1) (which I think is the only change I'd
want in order to enable the 2-signal proposal, to the extent it
seems possibly worth doing).  To me it's an understandable decision,
especially considering RFC 6660, which seems essentially "SCE but
only applicable inside DSCP", and would prefer not to lose that pre-
congestion signal if something weird happens while it's tunneled.

As an aside, I think it's a fair claim that Bob made, that he's the
one who has been trying to fit this jigsaw together for over a decade.
Not to endorse his conclusion that the best choice today is to do
something that seems to break regular ECN, but between his authorship
of those specs and his long list of other publications on the topic,
it's a good bet that Bob has given the whole space a larger quantity
of thought than anyone else alive, which is certainly worth something.

Best regards,
Jake

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 May 2020 12:54:45 +0200

Hi Jake,

thanks for the information. Indeed a case of the perfect being an enemy of the good.
From rfc3168:
"The presence of a copy of the ECN field in the inner header of an IP
   tunnel mode packet provides an opportunity for detection of
   unauthorized modifications to the ECN field in the outer header."

This in a nutshell was an original sin, of being clever when KISS could have carried the day. Now, at the time that was probably a reasonable choice, but I wonder whether that design decision is not worth revisiting?

I would humbly ask, whether it is possible to get rid of that requirement and get back to the simple strict copy regime and leave the tampering detection to the endpoints? As the current L4S tunneling discussion indicates, that intent of being cautious back then now is a burden.

I note that according to https://tools.ietf.org/html/rfc4301 tunnel entry already does the simple copy inner to outer, it is really ony the decapsulation side that needs special care...

> On May 28, 2020, at 10:21, Holland, Jake <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] Thanks.

> more or less laid out there.

	[SM] So what puzzles me is that these rules all give tunneled traffic additional ECN sanity checking that non-encapsulated packets traveling over the same path and hence the same AQMs do not enjoy?

> https://datatracker.ietf.org/doc/rfc6040/ballot/

	[SM] Thanks a lot, short but helpful.

> congestion signal if something weird happens while it's tunneled.

	[SM] But again that same pre-congestion signal would be lost for non tunneled packets encountering the same path, no? The only additional step would e the tunnel decap, but we need that to work reliable anyway, as otherwise nothing is guaranteed anymore?

> one who has been trying to fit this jigsaw together for over a decade.

	[SM] Sure, that is quite some work and time invested in all of this.

> of thought than anyone else alive, which is certainly worth something.

	[SM] Sure, the sign that I propose simplistic solutions, is probably a cause of Kruger-Dunning and nit fully appreciation the intricacies of the problem space and what kind of unfortunate behavior already exists in the wild and needs to be dealt with.

	That said, "working best" with the current breed of tunnels is not the only sane criterion, especially if we are talking about longer term evolution of the network.

Best Regards & Thanks
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 28 May 2020 12:17:40 +0100

Please relate ECN tunnel discussions to the relevant work items. Is this 
related to draft-ietf-tsvwg-rfc6040update-shim or the 
draft-ietf-tsvwg-ecn-encap-guidelines or to draft-ietf-intarea-tunnels?

Gorry

-- 
G. Fairhurst, School of Engineering

---

From: "C. M. Heard" <[email-masked]>
Date: Fri, 29 May 2020 21:27:59 -0700

On Thu, May 28, 2020 at 1:21 AM "Holland, Jake" wrote:

> On 5/28/20, 12:57 AM, "Sebastian Moeller" wrote:

Consider applying Sebastian's suggestion to ECT(0) and ECT(1) only.

Only the combination marked (**) below would be changed from RFC 6040:

            |  ECT(0) |  ECT(0) | ECT(0)     | ECT(1) (++)|     CE     |

That combination was marked with (!) in RFC 6040 because no existing
standard relied upon it.

[I]t seems the consensus at the time landed on a copy of outer
> congestion signal if something weird happens while it's tunneled.

But RFC 6660 actually relies only on the combination (++) which would be
unchanged.

> draft-ietf-tsvwg-ecn-encap-guidelines or to draft-ietf-intarea-tunnels?

The discussion strayed a bit but the topic of this thread is the
ECT(1)->ECT(0) alternative for safely accomplishing the goals of L4S. This
message concerns a minimal change to RFC 6040 needed to make ECT(1)->ECT(0)
work.

AFAICT that change would break no existing standard. Also AFAICT the only
effect on draft-ietf-intarea-tunnels would be a need to refer to a 6040bis
instead of RFC 6040. In draft-ietf-tsvwg-ecn-encap-guidelines the change
would necessitate an update to the rules in Section 4.4, Decapsulation
Guidelines, specifically rules 1 and 4, and in addition numerous references
to 6040 would need to be changed. I have not reviewed all 41 occurrences,
but a spot check indicates that some of the text surrounding those
references would need updating as well. The effect on
draft-ietf-tsvwg-rfc6040update-shim, to the extent that it was not rolled
into a 6040bis, would be to update the discussion of fragment reassembly,
probably to recommend preserving the pre-reassembly and post-reassembly
fractions of packets with ECT(0), ECT(1), and CE and to update the
discussion of ECN handling in the various protocols under IETF change
control.

There are lot of other changes to ongoing work items needed to make the
ECT(1)->ECT(0) alternative work, most notably (AFAICT) a reworking of
AccECN. That does not mean all of the existing work goes down the drain,
but the changes are not trivial.

And as Bob Briscoe has vocally noted, the adoption tail would grow very
long (though as he also notes, the future is longer than the present or
past),

Chairs: if all this is considered off-topic for the TSVWG list, given the
decision to move forward with the L4S work, just say the word, and I'll
happily comply. But, as David Black said in the "dispute" thread, "I am
confident that TSVWG *does not* currently have rough consensus that the L4S
experiment is safe to perform on the Internet." So there may be some value
in further exploration of the ECT(1)->ECT(0) alternative. If that needs to
go elsewhere, so be it.

Thanks & regards,

Mike Heard