Subject: [tsvwg] Results of consensus call on ECT(1) usage

From: Wesley Eddy <[email-masked]>
Date: Thu, 21 May 2020 22:32:45 -0400

**

*Thank you to all of the participants that provided feedback to our 
consensus call on ECT(1) usage.  After discussing the contributions 
between the chairs and our ADs, here are the conclusions we’ve drawn.*

*

Result of consensus call: TSVWG has interest in working on a low-latency 
service using ECT(1) as an input (e.g. L4S) in preference to working on 
enabling high-fidelity congestion control using ECT(1) as an network 
output signal (e.g. SCE). The chairs see strong support for this 
direction, including significant interest from network operators and 
vendors.

Specific concerns that still need to be worked on (not an exhaustive list):

 1.

    The WG needs to consider operational issues in L4S deployment and
    provide guidance to operators of networks that participate in the
    L4S experiment and operators of networks that do not.

 2.

    The WG must ensure that L4S does not add significant new incentives
    for network users or operators to lie or cheat in packet markings,
    beyond marking incentives for current ECN functionality.

Path forward:

  *

    The objective is to advance the L4S drafts to WGLC after revising
    them to resolve technical issues.

  *

    A successful WGLC on the drafts will require TSVWG rough consensus
    that the L4S experiment is safe to perform on the Internet.

  *

    Based on the discussion to date, the chairs recommend that a draft
    describing TCP Prague be produced, to provide an example of endpoint
    congestion control that is suitable for use with L4S.

*

---

From: Paul Vixie <[email-masked]>
Date: Fri, 22 May 2020 03:03:03 +0000

On Friday, 22 May 2020 02:32:45 UTC Wesley Eddy wrote:
> ...

the relevant section of RFC 2026 is as follows:

> 6.5.1 Working Group Disputes
> 
>    An individual (whether a participant in the relevant Working Group or
>    not) may disagree with a Working Group recommendation based on his or
>    her belief that either (a) his or her own views have not been
>    adequately considered by the Working Group, or (b) the Working Group
>    has made an incorrect technical choice which places the quality
>    and/or integrity of the Working Group's product(s) in significant
>    jeopardy.  The first issue is a difficulty with Working Group
>    process;  the latter is an assertion of technical error.  These two
>    types of disagreement are quite different, but both are handled by
>    the same process of review.

on a personal note, my own expressed position was, that if something had to be 
done it should be an input not an output. so, the chairs' declared result is 
compatible with what i said. therefore i'm not claiming that my own views were 
not adequately considered (a, above). rather, i'd concerned about (b) above.

this choice was not made based on technical merit, by the chair's own 
description (quoted above). both the rough consensus and the running code 
looked to me more like choice 3 ("more testing"). what the operators and 
vendors who can afford to attend these meetings are interested in, may differ 
from the community's long term interests.

>    A person who disagrees with a Working Group recommendation shall
>    always first discuss the matter with the Working Group's chair(s),
>    who may involve other members of the Working Group (or the Working
>    Group as a whole) in the discussion.

we are "here." i hope the chairs can explain why choice 3, more testing, was 
not the rough consensus they think they witnessed, and also, how the time 
value of a choice of 1 or 2 "now" is higher than the merit value of getting to 
an evidence-based consensus "later".

>    If the disagreement cannot be resolved in this way, any of the
>    parties involved may bring it to the attention of the Area
>    Director(s) for the area in which the Working Group is chartered.
>    The Area Director(s) shall attempt to resolve the dispute.
>    
>    If the disagreement cannot be resolved by the Area Director(s) any of
>    the parties involved may then appeal to the IESG as a whole.  The
>    IESG shall then review the situation and attempt to resolve it in a
>    manner of its own choosing.
>    
>    If the disagreement is not resolved to the satisfaction of the
>    parties at the IESG level, any of the parties involved may appeal the
>    decision to the IAB.  The IAB shall then review the situation and
>    attempt to resolve it in a manner of its own choosing.

i think a challenge was inevitable no matter what outcome was designated as 
the consensus, and that this probably is a matter to be settled by IETF as a 
body rather than by this working group in isolation.

-- 
Paul

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 May 2020 22:18:04 -0700

> On May 21, 2020, at 8:03 PM, Paul Vixie <[email-masked]> wrote:
> 
> done it should be an input not an output.

I wonder how many positions are based on that premise.

We were never given a “none of the above” choice.

Joe

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Fri, 22 May 2020 06:35:14 -0700 (PDT)

> We were never given a ?none of the above? choice.

Which was actually itemized by the IETF Chair Alisha Cooper at the
very end of the Interim Meeting, I was rather shocked to not see
it as an option, and hench my vote was made with a comment that
I had been "boxed into a choice."

-- 
Rod Grimes                                                 [email-masked]

---

From: Lars Eggert <[email-masked]>
Date: Fri, 22 May 2020 17:25:21 +0300

Hi,

> I had been "boxed into a choice."

you can always refrain from responding to the poll?

Lars

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Fri, 22 May 2020 07:50:03 -0700 (PDT)

> you can always refrain from responding to the poll?

I can not even believe I am reading that statement.

So in effect squelching any voice I would have in the decision process?  

Had it been a simple poll with little effect on the future I might
of taken that route, but this is a major concensus call with high
contention.

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 22 May 2020 16:15:41 -0400

Hello, this is my own attempt to answer your questions/concerns, not 
necessarily matching what other co-chairs or our ADs would say, though 
we did coordinate on evaluating the consensus call, and the original 
statement of results.

Just to make sure it's clear, this was not a WGLC on any of the drafts.  
It was about finding direction on one particular issue that has been an 
active part of the discussion for around a year ("what to do with 
ECT(1)?"), and for which there is a broad desire to move forward somehow 
on.  We tried to be clear that there is still work to be done in this 
selected direction for the WG to make sure the technology is safely 
deployable, before we do a WGLC.  Pruning the other possible directions 
helps to focus on that work, and getting to the WGLC will be based on 
technical merit.

Specifically on your question of why choice 3 ("more testing") was not 
the rough consensus on how to use ECT(1), I don't think the responses 
indicated that we had enough energy or bodies to work in this direction, 
though there were some helpful points that people with choice 3 response 
made about what we should be looking for while proceeding in the choice 
1 direction, that I think we are all taking into account.

Joe and others have asked about a "do nothing" (or "none of the above") 
choice that was missing from the poll.  If there were inconclusive 
results to the poll, I suppose that would have been a default state we 
would fall into.   This would basically be maintaining the status quo 
with regard to ECT(1) usage, and have both L4S and SCE needing to find 
alternative ways to function. The inputs we received show that this 
would not have been a favorable option to the group.

I hope this is all more clear and addresses your questions/concerns, and 
sorry if it wasn't evident in the original response we chairs had 
coordinated on.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 22 May 2020 21:47:25 +0100

Paul, Chairs

On 22/05/2020 04:03, Paul Vixie wrote:
> not the rough consensus they think they witnessed,

[BB] Can I suggest that the chairs ask those asking for more testing to 
write a review of all the tests that have been done so far (by 
proponents and opposition), and identify the specific further tests that 
they believe are necessary. Then the chairs or the WG or whoever is 
judging this dispute can decide whether the tests they are asking for 
are critical prerequisites for assignment of an experimental codepoint 
in the IP header.

Reasoning: There have been a number of cases where people have 
complained in general terms about testing, and subsequently it became 
apparent that they had not read the test reports - some of which had 
even been verified independently and published by third parties.

> and also, how the time

[BB] I notice that you (Paul) said you'd been following this since the 
first interim. You may not be aware that this work was brought to the 
IRTF in March 2015, then brought to the IETF in July 2015 along with 
thousands of test results. A very large BoF was held in Jul 2016 that 
led to L4S drafting work being started in tsvwg and tcpm. The debate 
over the pros and cons of different codepoints was completed around Nov 
2017, when ECT(1) was settled on and that debate was written up in the 
ecn-l4s-id draft. In March 2019, L4S was approaching WGLC, when the SCE 
proposal for a different codepoint arrangement was first published, and 
proponents of both approaches have published large volumes of test 
results in the 14 months since then.

So those taking part in the recent consensus call had a wealth of test 
data on which to base their decision, from both proponents and opponents 
of the outcome that the chairs have just announced.

In the past, I have been in a similar position, when I have discovered 
that a major decision was imminent that would close off areas that were 
important to me. The closest example was when draft-ietf-tsvwg-ecn 
[RFC3168-to-be] was approaching WGLC. I took it upon myself to 
thoroughly read the draft and all its references, to catch up on all the 
mail archives, and to write a draft articulating our concerns (with Jon 
Crowcroft at that time). As I put our concerns, I could feel the time 
pressure of everyone involved breathing down my neck. By putting in the 
effort to put our concerns concisely and rapidly, the WG got them 
resolved quickly - Jon & I had to concede some aspects in the interest 
of keeping progress moving. As a result, RFC3168 went through WGLC soon 
after and was published only 7 months after we first intervened.

So, yes, there is a huge degree of time pressure now. The closing of 
this consensus call allows us to get moving in anger. You will have seen 
the number of companies that plan to deploy. No-one has bottomless 
project funding. I (and hopefully others) will try to point you at the 
resources that are relevant for your concerns. But I hope you will 
understand that, at the closing stages of the debate, we can't be 
expected to retype all the discussion each time a new person joins.

Cheers

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Joseph Touch <[email-masked]>
Date: Fri, 22 May 2020 14:11:48 -0700

> On May 22, 2020, at 1:15 PM, Wesley Eddy <[email-masked]> wrote:
> 
> Joe and others have asked about a "do nothing" (or "none of the above") choice that was missing from the poll.  If there were inconclusive results to the poll, I suppose that would have been a default state we would fall into.   This would basically be maintaining the status quo with regard to ECT(1) usage, and have both L4S and SCE needing to find alternative ways to function. The inputs we received show that this would not have been a favorable option to the group.

You asked whether we want to eat chicken or fish. You didn’t give us a choice of not dining.

So sure, the conclusion was one of the pre-determined paths. All you know right now is that there is more interest in one than the other.

An alternate conclusion would be that we don’t NEED to move forward with either of these at this time. I guess if the goal is to align effort in the WG, that’s fine.

But we have not seen a question that suggests either approach overcomes the “first, do nothing” approach - which frankly ought to be our first answer, in this WG.

Joe

---

From: "Black, David" <[email-masked]>
Date: Fri, 22 May 2020 22:37:26 +0000

Comments inline with the same “not necessarily matching what other co-chairs or our ADs would say” caution as Wes’s note ... Thanks, --David

From: tsvwg <[email-masked]> On Behalf Of Joseph Touch
Sent: Friday, May 22, 2020 5:12 PM
To: Wes Eddy
Cc: [email-masked]
Subject: Re: [tsvwg] dispute (Re: Results of consensus call on ECT(1) usage)

[EXTERNAL EMAIL]

On May 22, 2020, at 1:15 PM, Wesley Eddy <[email-masked]<mailto:[email-masked]>> wrote:

So sure, the conclusion was one of the pre-determined paths. All you know right now is that there is more interest in one than the other.
[David>] More is also known about the range and nature of other views.  By my count, although there were 3 choices of responses, I saw at least 5 distinct views expressed, including do nothing now.

An alternate conclusion would be that we don’t NEED to move forward with either of these at this time. I guess if the goal is to align effort in the WG, that’s fine.
[David>] Indeed, the latter (align effort in the WG) was the primary goal.  I view the decision here as a working group management decision about what the WG is to work on.

But we have not seen a question that suggests either approach overcomes the “first, do nothing” approach - which frankly ought to be our first answer, in this WG.
[David>] There will be opportunities to express that view in the future, including Working Group Last Call on the L4S drafts (assuming that we get to that stage).

Joe

---

From: Paul Vixie <[email-masked]>
Date: Sat, 23 May 2020 02:31:48 +0000

> you can always refrain from responding to the poll?

no. after the interrim meeting and the ietf chair's explicit comments, the 
right response was not silence, but a protest. i didn't realize this or i 
would have done so earlier.

the case for "something has got to be done, now" has not been made, and so the 
assertion that "both inbound and outbound are things that could be done, so we 
must pick one of them" is a nonsequitur.

-- 
Paul

---

From: Joseph Touch <[email-masked]>
Date: Fri, 22 May 2020 21:08:21 -0700

> On May 22, 2020, at 3:37 PM, Black, David <[email-masked]> wrote:
> 

In theory.

In practice, “whether to do something or not” at best (and rarely) happens at WG adoption. After that, the deliverable isn’t “decide whether to proceed” but “deliver X”.

LC is typically only “speak now or forever hold your peace” for updates, never really “should we proceed”. 

—

Note that I appreciate that the IETF wants to let a thousand flowers bloom, encouraging a “denial of service” attack on WG resources that is far too easy to game. But in this case, we’re talking about nailing in stone (in a sense) a decision about a VERY limited resource (TCP header bits).

First, do no harm isn’t the same as ‘which harm would you each prefer”.

Joe

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 23 May 2020 11:28:16 +0200

Bob, chairs, list,

Top post, because this is not a on-to-ne response to Bob points, but to the sentiment of his argument about testing, as I understood it; since this was triggered by Bob' response I opt to keep it in thread instead of opening a new thread.

I want to take the opportunity to register my objections to Bob's implicit claim below, that L4S has seen sufficient  and diverse enough testing, and that legacy testing  results with DCTCP fro the L4S's effort early days is simply transferable to TCP Prague. 
	Also I want to note that quantity of testing is a imprecise (bad) proxy for quality of testing. In a Popperian sense, I got the impression, that L4S went out of its way to document where white swans can be found instead of spending time and effort trying to find non-white swans, as a result the hypothesis all swans are white (or precisely the proposed L4S design works robustly and reliably) is still insufficiently tested.

1) Quantity: 
	I agree that the L4S team has presented an impressive array of testing results, so I will concede the sheer number. 

2) Diversity:
	Here the effort is considerably less impressive, it took the SCE's team effort to present data for the natural cases of asymmetric links and bi-directional saturation. For a technique primarily intended to the ISP/end-user edge these two are not extreme corner-cases but rather the bread and butter any AQM/QoS solution needs to deal with acceptably, period.
	I find it immensely troubling that Bob, in one earlier e-mail conceded that L4S would probably not work well with bi-directional saturation, and never ever bothered to make that a testing priority. This is the looking for more white swans approach I allude to above, but seemingly by conscious choice and not by simply overlooking the critical cases.
	I am also puzzled that the WG here seems determined to push L4S/ETC(1) through even without sufficient evidence that all the pie-in-the-sky promises actually are kept. All tire-kicking so far implies that L4S over-promises and under-delivers once one looks beyond the "normal" conditions which admittedly have been tested in considerable quantity, but essentially testing the same condition that L4S deals with well multiple times does not increase the relevant depth of testing, IMHO.

3) Legacy testing:
	The amount of major bugs, that the SCE team found in TCP Prague when doing comparison testing reveals that TCP Prague development does not seem to use regular regression testing. Based on that fact and the fact the TCP Prague by now has diverged from DCTCP in a noticeable fashion. I question whether the old DCTCP testing results have more that historic value by now, as it seems far from clear that TCP Prague will behave similarly, either because of evolutionary divergence or because of newly introduced but yet unfixed bugs. If the L4S team wants to have this testing taken seriously I encourage them to repeat these tests with a current TCP Prague version. Otherwise these tests carry little vsalue for predicting how current L4S implementation will behave.

Best Regards
	Sebastian

P.S.: I realize that engineering is the "art of the possible" and that often perfect is a worse goal than good enough. But my objections against the current L4S implementation & design are that they simply are not :good enough" (yet?), and that I consider it to be an odd situation that the WG seemingly steers down a road of spending the last ECN codepoint on a sheer promise alone. Then again I predicted that outcome immediately after the call for consensus was made to the lists, as the chairs might remember.

> On May 22, 2020, at 22:47, Bob Briscoe <[email-masked]> wrote:
> 
> Paul, Chairs
> 
> [BB] Can I suggest that the chairs ask those asking for more testing to write a review of all the tests that have been done so far (by proponents and opposition), and identify the specific further tests that they believe are necessary. Then the chairs or the WG or whoever is judging this dispute can decide whether the tests they are asking for are critical prerequisites for assignment of an experimental codepoint in the IP header.
> 
> Reasoning: There have been a number of cases where people have complained in general terms about testing, and subsequently it became apparent that they had not read the test reports - some of which had even been verified independently and published by third parties.
> 
>> and also, how the time
> [BB] I notice that you (Paul) said you'd been following this since the first interim. You may not be aware that this work was brought to the IRTF in March 2015, then brought to the IETF in July 2015 along with thousands of test results. A very large BoF was held in Jul 2016 that led to L4S drafting work being started in tsvwg and tcpm. The debate over the pros and cons of different codepoints was completed around Nov 2017, when ECT(1) was settled on and that debate was written up in the ecn-l4s-id draft. In March 2019, L4S was approaching WGLC, when the SCE proposal for a different codepoint arrangement was first published, and proponents of both approaches have published large volumes of test results in the 14 months since then.
> 
> So those taking part in the recent consensus call had a wealth of test data on which to base their decision, from both proponents and opponents of the outcome that the chairs have just announced.
> 
> 
> In the past, I have been in a similar position, when I have discovered that a major decision was imminent that would close off areas that were important to me. The closest example was when draft-ietf-tsvwg-ecn [RFC3168-to-be] was approaching WGLC. I took it upon myself to thoroughly read the draft and all its references, to catch up on all the mail archives, and to write a draft articulating our concerns (with Jon Crowcroft at that time). As I put our concerns, I could feel the time pressure of everyone involved breathing down my neck. By putting in the effort to put our concerns concisely and rapidly, the WG got them resolved quickly - Jon & I had to concede some aspects in the interest of keeping progress moving. As a result, RFC3168 went through WGLC soon after and was published only 7 months after we first intervened.
> 
> So, yes, there is a huge degree of time pressure now. The closing of this consensus call allows us to get moving in anger. You will have seen the number of companies that plan to deploy. No-one has bottomless project funding. I (and hopefully others) will try to point you at the resources that are relevant for your concerns. But I hope you will understand that, at the closing stages of the debate, we can't be expected to retype all the discussion each time a new person joins.
> 
> Cheers
> 
> 
> 
> Bob
> 
> 
>

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 23 May 2020 12:12:05 +0200

Hallo Lars,

> On May 22, 2020, at 16:25, Lars Eggert <[email-masked]> wrote:
> 
> Hi,
> 

[SM] Sue that is an option. If L4S would have been designed without side-effects for non-L4S traffic, I would happily ignored both the discussion and the poll. The fact of the matter is, that IMHO L4S will not achieve its stated goals (because its goals are considerably too ambitious, and its developers, ever the optimists, never bothered to actually test their design under even mildly adversarial conditions, what can go wrong?). 
	To add insult to injury, L4S will not only not work as promised, but it will also introduce a nice and juicy new DOS target (the LL-queue can be disrupted with simple bursts of traffic, no sustained overload necessary, try catching that with "circuit-breakers" and similar heuristics).
	As stated before, all this for essentially potentially ~1 ms of latency-under-load-increase instead of 5-10ms (as compared to the current state of the art, even still using 1/sqrt(p) CC signaling), so rather modest gains for rather unpleasant side-effects. 
	In short, a decision important enough to participate in even after the WG restricted the choices to a sub-optimal selection, since the outcome of the WG decision might affect all internet users in the future. 
	If I would not know better, I would feel that the poll got the outcome it was designed to yield; but we knew that before, there are companies that already invested in L4S and hence there are already vested interests in it proceeding along the standardization path. And to be explicit I see nothing nefarious in that, but it is not the best conditions under which to ask an engineering question about how to use ECT(1) in the future.
	
Interestingly almost nobody of those voicing their support for ECT(1) as input now, had contributed in the past when we discussed the short-comings of L4S and nobody volunteered empiric data supporting the L4S hypothesis that the criticism was overblown.

Best Regards
	Sebastian

---

From: Paul Vixie <[email-masked]>
Date: Mon, 25 May 2020 00:19:54 +0000

> sorry if it wasn't evident in the original response we chairs had

it is clear, but it is your unofficial (no-hats) position, and also, it does 
not address my concerns. what's our next step, per RFC 2026?

-- 
Paul

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 25 May 2020 11:15:41 -0400

> not address my concerns. what's our next step, per RFC 2026?

It's my official position; I only said it wasn't coordinated with other 
co-chairs and ADs (because I wanted to provide a more timely response 
than coordination among 5 people would allow).

I'm happy to try to answer reasonable follow-up questions, of course.  
Right now, I'm curious why you don't think your concerns were addressed, 
since you asked the chairs to explain two things (why choice 3 wasn't 
the outcome, and why a timely choice as important), both of which I've 
tried to explain.  If there was a part of that explanation you want to 
follow up on, please do!

---

From: "Black, David" <[email-masked]>
Date: Tue, 26 May 2020 01:22:48 +0000

Paul,

Going back to the original expression of concerns, and writing as a WG chair:

> ... i hope the chairs can explain why choice 3, more testing, was not the rough consensus they think they witnessed,
> and also, how the time value of a choice of 1 or 2 "now" is higher than the merit value of getting to an evidence-based consensus "later".

In reverse order, the goal is to get to an evidence-based consensus "later", e.g., as indicated by the following text from the announcement email:

	•	A successful WGLC on the drafts will require TSVWG rough consensus that the L4S experiment is safe to perform on the Internet.

I am confident that TSVWG does not currently have rough consensus that the L4S experiment is safe to perform on the Internet.  RFC 7282 is relevant background reading on rough consensus.

The only way that I could see choice 3 (more testing) being viewed as the outcome of the consensus call would be to assume that almost everyone who expressed support of choice 1 (ECT(1) as input) would also support choice 3.  Not only was that not the structure of the choices, but I am also confident that such an assumption would be incorrect.  Please keep in mind that choice 3 was not "more testing" in general - choice 3 specifically asked for specific tests:

	3.  	There is a specific test or tests I need to see before making a decision about ECT(1). Please be specific about the tests in your response.

The time value in choosing a direction now is that the WG has spent most of the past year not getting much done in this area to the consternation (and more) of many of the people involved on all sides.  From a WG management perspective, it would be irresponsible of the WG chairs to allow this situation to continue for another year.

Thanks, --David (as a TSVWG chair)

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Wesley Eddy
> Sent: Monday, May 25, 2020 11:16 AM
> To: Paul Vixie; [email-masked]
> >> necessarily matching what other co-chairs or our ADs would say,
> >> though we did coordinate on evaluating the consensus call, and the
> >> original statement of results.
> >>
> >> ...
> >>
> >> I hope this is all more clear and addresses your questions/concerns,
> >> and sorry if it wasn't evident in the original response we chairs had
> >> coordinated on.
> > it is clear, but it is your unofficial (no-hats) position, and also,
> > it does not address my concerns. what's our next step, per RFC 2026?
> 
> 
> It's my official position; I only said it wasn't coordinated with other co-chairs and
> ADs (because I wanted to provide a more timely response than coordination among
> 5 people would allow).
> 
> I'm happy to try to answer reasonable follow-up questions, of course. Right now, I'm
> curious why you don't think your concerns were addressed, since you asked the
> chairs to explain two things (why choice 3 wasn't the outcome, and why a timely
> choice as important), both of which I've tried to explain.  If there was a part of that
> explanation you want to follow up on, please do!
>