Subject: Re: [tsvwg] plan for L4S issue #29

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Tue, 15 Sep 2020 12:43:09 +0000

Hi Wes, Pete,

I think to make progress on avoiding both false negatives and false positives, a good view on the conditions that cause problems is needed. So we better have the means to detect the real life impact of Classic-ECN-FIFO deployments. This means we need to be able to switch off Classic ECN detection (under controlled or even known conditions).

Another point is that it would be useful also to have all control variables of the existing implementation configurable for everyone willing to further experiment (without necessarily needing to change code). As I understood, the right tuning of these can bring a lot of further improvement opportunities. Also depending on a typical deployment, these parameters could be tuned for that specific targeted case.

So the resolution of this issue is exactly to facilitate further improving the detection algorithm (preferably via tuning), and being able to disable it when conditions are controlled or safe to avoid these false negatives.

I think these are topics that can be covered by the Operational Guidelines draft.

Regards,
Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Pete Heist
Sent: Friday, July 31, 2020 8:53 PM
To: Wesley Eddy <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] plan for L4S issue #29

Hi Wesley,

One thing I noticed during testing was that the current implementation of TCP Prague in Linux allows disabling bottleneck detection through the prague_ecn_fallback kernel module parameter (https://github.com/L4STeam/linux/blob/0e7cf8acb318873c3f61084453f8da15b2e398be/net/ipv4/tcp_prague.c, line 158). I don’t know if that was left in only for testing.

In section 6.3.3 of l4s-arch, there is discussion around classic bottleneck detection. Since I don’t see an explicit MUST that it remain enabled (although I do see the text “an L4S sender will have to fall back to…”), it’s not completely clear to me if it’s actually required to be implemented and permanently enabled in all implementations. If it is, I suppose the implementation should reflect that also.

While I feel it best that detection identifies both types of queues accurately, if bottleneck detection were both an explicit MUST in the text *and* not possible to disable in any implementation, I think that would make the misidentification of L4S queues as classic ECN queues less of a safety concern, since it would be impossible to turn off. It would remain an issue for the architecture overall though.

Hope that helps...

Pete

> On Jul 31, 2020, at 5:41 PM, Wesley Eddy <[email-masked]> wrote:
> 
> Hello, ticket #29 for the L4S documents is about classic bottleneck detection misidentifying L4S queues as classic ECN queues.
> 
> https://trac.ietf.org/trac/tsvwg/ticket/29
> 
> In contrast to other issues, it doesn't seem like this should block a WGLC on the L4S drafts.
> 
> 	• It is specific to classic bottleneck detection algorithm, which is planned to be worked on in the Prague ICCRG draft.
> 	• The result is sometimes failing to achieve the best possible L4S behavior, but doesn't seem to be an Internet safety issue.  This resulting in people turning off classic bottleneck detection would be a different issue, and something maybe the operator guidelines would address.
> 	• It seems like it can be worked on further in the course of L4S experimentation, without negative effects to others.
> So, I believe we should track this work in the ICCRG, and close the ticket here.  Please let me know in the next week if I've misunderstood any aspect of this and it should remain open.
> 
>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 22 Sep 2020 18:41:56 +0100

Jonathan, Wes, list,

Sorry, I've been dead to the L4S conversations on this list for weeks 
now. But I've resolved to get back involved now. I stopped interacting 
on L4S over the summer because I became burned out by the L4S 
conversation being dominated by the same two or three people who do not 
want the agreed deliverables at all. I have never had to work on a 
technology before where I am faced with a continual stream of blocking 
messages, rather than constructive criticism or solutions being offered.

It has always been recognized that there are not enough bits in the IP 
header for a perfect encoding scheme, so that we will have to choose the 
least worst compromise. We had a consensus call in April, in which the 
questions of Classic ECN fallback (#16) and false positives in the 
current algo (#29) were well understood by all concerned. The result was 
a decision not to go with the alternative solution (SCE) that inherently 
addresses issues #16 & #29, but instead to continue with the solution 
(L4S) that partially addresses these issues and fully addresses other 
concerns. The L4S design team were asked to prepare a new ops guidelines 
draft as part of that decision.

I became disheartened about working further on the Classic ECN AQM 
fallback algo when all the work we'd done was just dismissed as futile. 
It became apparent that no amount of work on it would ever satisfy the 
few people who have been vocally opposing L4S as a whole. Now that's a 
shame, because I/we had worked out an additional heuristic to address 
the large bulk of the false positives that appeared at low link capacity 
(note that is different from low flow rate). I admit that this solution 
would not have addressed the few cases that arise in apparently random 
circumstances. And I admit that, without having tried it, it might not 
work at all.

I now prefer to work on Classic ECN AQM detection more than fallback. My 
logic for this is that L4S can be /unfair/ against Classic flow(s) in a 
Classic ECN AQM, but it is not /unsafe/. Unsafe means starvation, but 
Classic flows always still progress against L4S. So, it should be 
sufficient for L4S senders to /monitor/ for Classic AQMs, at least in 
the initial stages of the L4S experiment.

This can then become part of the monitoring infrastructure that the L4S 
ops guidance proposes for the L4S experiment. It can highlight where 
classic ECN AQMs might have been deployed. Then, if it produces false 
positives, further digging should discover that there is not actually 
any classic ECN AQM where it says there is. Then at least false 
positives don't harm performance.

If it turns out that there are a lot of classic ECN AQMs out there, it 
then becomes important to improve the detection algo so it can be used 
in-band for fall-back. On the other hand, if we find there is very 
little, it becomes preferable to use the network configuration 
techniques described in the ops guidance to alter the classic ECN AQMs 
to isolate classic ECN and L4S ECN traffic.

Regards

Bob

PS. There are also the cases where L4S flows don't perform well 
(particularly around start-up behaviour) in a Classic ECN AQM within an 
FQ scheduler (e.g. the experiments Pete did with L4S over FQ_CoDel and 
with Cake). Over the summer Joakim and I have been working on faster 
response to marking in the Prague algo, and we need to combine that with 
delay measurements for these cases.

I would also like to see a commitment to improving the CoDel and Cobalt 
control laws to increase ECN marking more rapidly in response to 
consistently increasing delay. Because this has always been a problem 
with unresponsive flows, irrespective of whether L4S had ever appeared 
on the scene. I'd be willing and interested to help with that sort of 
work,... assuming my day job left sufficient time for this.

On 31/07/2020 19:03, Jonathan Morton wrote:
>> On 31 Jul, 2020, at 6:41 pm, Wesley Eddy <[email-masked]> wrote:
>>
> Presently, the Prague congestion control algorithm is unsafe on the Internet without a properly functioning classic-bottleneck detection heuristic.  Additionally, the classic-bottleneck detection heuristic that has been published and demonstrated does not function properly.  This combination of facts absolutely *is* a blocker for WGLC.
>
> Therefore, this issue should not be closed until it has been concretely addressed.
>
>   - Jonathan Morton
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Pete Heist <[email-masked]>
Date: Wed, 23 Sep 2020 14:19:09 +0200

Hi Koen,

I can definitely understand the need to turn bottleneck detection on
and off for testing, or for additional knobs during development.

Overall, I suspect that there will be more questions about potential
problems if bottleneck detection is not a MUST for implementations in
the draft, or not baked into the final implementation in a way that
works well under a wide variety of conditions without additional
configuration.

On an easier topic, I wonder if we shouldn't change the "false
negatives" and "false positives" terminology to something clearer, like
"mis-identification of RFC 3168 bottlenecks as L4S", or "mis-
identification of L4S bottlenecks as RFC 3168", respectively. I might
have opened up a can of worms there in trying to save a few words. :)

Pete

On Tue, 2020-09-15 at 12:43 +0000, De Schepper, Koen (Nokia -
BE/Antwerp) wrote:
> Hi Wes, Pete,
> 
> I think to make progress on avoiding both false negatives and false
> positives, a good view on the conditions that cause problems is
> needed. So we better have the means to detect the real life impact of
> Classic-ECN-FIFO deployments. This means we need to be able to switch
> off Classic ECN detection (under controlled or even known
> conditions).
> 
> Another point is that it would be useful also to have all control
> variables of the existing implementation configurable for everyone
> willing to further experiment (without necessarily needing to change
> code). As I understood, the right tuning of these can bring a lot of
> further improvement opportunities. Also depending on a typical
> deployment, these parameters could be tuned for that specific
> targeted case.
> 
> So the resolution of this issue is exactly to facilitate further
> improving the detection algorithm (preferably via tuning), and being
> able to disable it when conditions are controlled or safe to avoid
> these false negatives.
> 
> One thing I noticed during testing was that the current
> implementation of TCP Prague in Linux allows disabling bottleneck
> detection through the prague_ecn_fallback kernel module parameter (
> https://github.com/L4STeam/linux/blob/0e7cf8acb318873c3f61084453f8da15b2e398be/net/ipv4/tcp_prague.c
> , line 158). I don’t know if that was left in only for testing.
> 
> In section 6.3.3 of l4s-arch, there is discussion around classic
> bottleneck detection. Since I don’t see an explicit MUST that it
> remain enabled (although I do see the text “an L4S sender will have
> to fall back to…”), it’s not completely clear to me if it’s actually
> required to be implemented and permanently enabled in all
> implementations. If it is, I suppose the implementation should
> reflect that also.
> 
> While I feel it best that detection identifies both types of queues
> accurately, if bottleneck detection were both an explicit MUST in the
> text *and* not possible to disable in any implementation, I think
> that would make the misidentification of L4S queues as classic ECN
> queues less of a safety concern, since it would be impossible to turn
> off. It would remain an issue for the architecture overall though.
> 
> Hope that helps...
> 
> Pete
> 
> > Hello, ticket #29 for the L4S documents is about classic bottleneck
> > detection misidentifying L4S queues as classic ECN queues.
> > 
> > 	• It is specific to classic bottleneck detection algorithm,
> > which is planned to be worked on in the Prague ICCRG draft.
> > 	• The result is sometimes failing to achieve the best possible
> > L4S behavior, but doesn't seem to be an Internet safety
> > issue.  This resulting in people turning off classic bottleneck
> > detection would be a different issue, and something maybe the
> > operator guidelines would address.
> > 	• It seems like it can be worked on further in the course of
> > L4S experimentation, without negative effects to others.
> > So, I believe we should track this work in the ICCRG, and close the
> > ticket here.  Please let me know in the next week if I've
> > misunderstood any aspect of this and it should remain open.
> > 
> >

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 23 Sep 2020 15:46:39 +0000

Hi Pete,

I don't think the goal is to wait for a perfect RFC3168 detection solution and we already have an implementation that works well under a wide variety of conditions without additional configuration. This also means that it can still be improved, which I think can be done more appropriate when facing real world issues. The goal of the operational guidelines is to provide a larger set of tools for solving problems with classic ECN bottlenecks, exactly to avoid the need to rely on perfect end-host detection mechanism only.

Related to the "false negatives" and "false positives" naming, I agree that it is very confusing. As the goal is to detect Classic ECN network AQM behavior, maybe better and shorter names could be "false-detect" and "false-non-detect"?

Koen.

-----Original Message-----
From: Pete Heist <[email-masked]> 
Sent: Wednesday, September 23, 2020 2:19 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; Wesley Eddy <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] plan for L4S issue #29

Hi Koen,

I can definitely understand the need to turn bottleneck detection on and off for testing, or for additional knobs during development.

Overall, I suspect that there will be more questions about potential problems if bottleneck detection is not a MUST for implementations in the draft, or not baked into the final implementation in a way that works well under a wide variety of conditions without additional configuration.

On an easier topic, I wonder if we shouldn't change the "false negatives" and "false positives" terminology to something clearer, like "mis-identification of RFC 3168 bottlenecks as L4S", or "mis- identification of L4S bottlenecks as RFC 3168", respectively. I might have opened up a can of worms there in trying to save a few words. :)

Pete

BE/Antwerp) wrote:
> off Classic ECN detection (under controlled or even known conditions).
> 
> deployment, these parameters could be tuned for that specific targeted 
> case.
> 
> One thing I noticed during testing was that the current implementation 
> of TCP Prague in Linux allows disabling bottleneck detection through 
> the prague_ecn_fallback kernel module parameter ( 
> https://github.com/L4STeam/linux/blob/0e7cf8acb318873c3f61084453f8da15
> b2e398be/net/ipv4/tcp_prague.c , line 158). I don’t know if that was 
> left in only for testing.
> 
> remain enabled (although I do see the text “an L4S sender will have to 
> fall back to…”), it’s not completely clear to me if it’s actually 
> implementations. If it is, I suppose the implementation should reflect 
> that also.
> 
> text *and* not possible to disable in any implementation, I think that 
> would make the misidentification of L4S queues as classic ECN queues 
> less of a safety concern, since it would be impossible to turn off. It 
> would remain an issue for the architecture overall though.
> 
> Hope that helps...
> 
> Pete
> 
> > 	• It is specific to classic bottleneck detection algorithm, which 
> > is planned to be worked on in the Prague ICCRG draft.
> > 	• The result is sometimes failing to achieve the best possible L4S 
> > behavior, but doesn't seem to be an Internet safety issue.  This 
> > resulting in people turning off classic bottleneck detection would 
> > be a different issue, and something maybe the operator guidelines 
> > would address.
> > 	• It seems like it can be worked on further in the course of L4S 
> > experimentation, without negative effects to others.

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Wed, 23 Sep 2020 09:08:40 -0700 (PDT)

> Related to the "false negatives" and "false positives" naming, I agree that it is very confusing. As the goal is to detect Classic ECN network AQM behavior, maybe better and shorter names could be…

Though I like these terms better, can we go one further and make it plain english "failure to detect" and "incorrectly detected"?
My issue is mostly with the double negative construct of combining "false" and "non" in the same term, which is most likely the same reason people have proplems with the original terms as in the case of "false negatives".

Regards,
Rod

> > b2e398be/net/ipv4/tcp_prague.c , line 158). I don?t know if that was 
> > bottleneck detection. Since I don?t see an explicit MUST that it 
> > remain enabled (although I do see the text ?an L4S sender will have to 
> > fall back to??), it?s not completely clear to me if it?s actually 
> > > 	? It is specific to classic bottleneck detection algorithm, which 
> > > 	? The result is sometimes failing to achieve the best possible L4S 
> > > 	? It seems like it can be worked on further in the course of L4S 

-- 
Rod Grimes                                                 [email-masked]

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 23 Sep 2020 17:17:10 +0100

> My issue is mostly with the double negative construct of combining "false" and "non" in the same term, which is most likely the same reason people have proplems with the original terms as in the ca…

Why can't we just say "does not detect"?

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 23 Sep 2020 19:47:07 +0200

Hi Koen,

> On Sep 23, 2020, at 17:46, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Pete,
> 
> I don't think the goal is to wait for a perfect RFC3168 detection solution and we already have an implementation that works well under a wide variety of conditions without additional configuration.

	How confident are we that the "wide variety of conditions" under which this supposedly works is actually representative for the existing internet?

> This also means that it can still be improved, which I think can be done more appropriate when facing real world issues. The goal of the operational guidelines is to provide a larger set of tools for solving problems with classic ECN bottlenecks, exactly to avoid the need to rely on perfect end-host detection mechanism only.

	I grudgingly agree that trying to fix L4S's AQM short comings by mandating fancy heuristics in the end points seems to be a loosing proposition. 

> Related to the "false negatives" and "false positives" naming, I agree that it is very confusing. As the goal is to detect Classic ECN network AQM behavior, maybe better and shorter names could be…

	I do not think that that is helpful, false-positive and false positive have much better well-known definitions that actually apply here. So if other nomenclature should be used, let's make it clearly different or better yet use already established terms. 

	We could switch to signal detection theory terms (which is just one way to think about a classifier with just two options), then the four combinations of truth and classification for our RFC3168 detector would be described with the following terms:

rfc3168 present; rfc3168 detected: 	hit (true positive)
rfc3168 present; rfc3168 not-detected:	miss	(false negative)
rfc3168 absent; rfc3168 detected:		false alarm (false positive)
rfc3168 absent; rfc3168 not-detected:	correct rejection (true negative)

these terms are pretty much standard for similar detection problems (and also offer a decent approach to assess the effectiveness of the detector).

Let's stick to some already established nomenclature, whether that is the true/false positive/negative one or the DST one, please (see e.g. https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers).

The bigger issue I have with this is that this is still just a "fudge" to be able write in the internet draft "transports are required to implement RFC3168 detection, as demonstrated in TCP Prague" even though everybody here should know that that is not going to happen. People will mark whatever with ECT(1) and we will have to live with the fall-out. IMHO this is a sufficient reason to put the smarts in the AQM, instead of the current approach of assuming that end-points will just do this out of goodness of their heart.

That is why I believe that rfc3168 detection in TCP Prague is a red herring that distracts from fixing L4S true issues. Like demonstrating that the current implementation actually performs as inteded over long holti-hop high RTT high Bandwdth links, over asymmetric links, and over uni- and bidirectionally saturated links.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 23 Sep 2020 22:00:53 +0200

Dear Bob,

a few questions in-line, below.

> On Sep 22, 2020, at 19:41, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan, Wes, list,
> 
> Sorry, I've been dead to the L4S conversations on this list for weeks now. But I've resolved to get back involved now. I stopped interacting on L4S over the summer because I became burned out by the L4S conversation being dominated by the same two or three people who do not want the agreed deliverables at all.

	[SM] Which deliverables did you agree to and with whom? 

> I have never had to work on a technology before where I am faced with a continual stream of blocking messages, rather than constructive criticism or solutions being offered.

	[SM] I take it you ignored the IPv4/v6 discussions then?

> 
> It has always been recognized that there are not enough bits in the IP header for a perfect encoding scheme, so that we will have to choose the least worst compromise.

	[SM] Least by which criterion?

> We had a consensus call in April,

	[SM] Did we? As far as I remember the decision back then was always a "vote" of confidence for L4S over SCE, to make team L4S continue with the experient, but it did not (even try to) answer the question whether L4S is ready for deployment.

> in which the questions of Classic ECN fallback (#16) and false positives in the current algo (#29) were well understood by all concerned. The result was a decision not to go with the alternative solution (SCE) that inherently addresses issues #16 & #29, but instead to continue with the solution (L4S) that partially addresses these issues and fully addresses other concerns. The L4S design team were asked to prepare a new ops guidelines draft as part of that decision.

	[SM] Yes, that fleetingly appeared on the list, I commented up it and did not head much of that thereafter.

> 
> I became disheartened about working further on the Classic ECN AQM fallback algo when all the work we'd done was just dismissed as futile.
> It became apparent that no amount of work on it would ever satisfy the few people who have been vocally opposing L4S as a whole.

	[SM] I have a feeling you are targeting in my direction here, but I have always maintained that I can be convinced by data, as long as the data was reasonably representable for to be expected "bad" conditions over the open internet (that is to say pure proofs of principle that something can work under some carefully selected conditions are not enough, no matter under how many iterations known good conditions are shuffled).

> Now that's a shame, because I/we had worked out an additional heuristic to address the large bulk of the false positives that appeared at low link capacity (note that is different from low flow rate). I admit that this solution would not have addressed the few cases that arise in apparently random circumstances. And I admit that, without having tried it, it might not work at all.

	[SM] Not sure that piling up heuristic on top of heuristic is a safe way to engineer robust and reliable solutions, sorry.

> 
> I now prefer to work on Classic ECN AQM detection more than fallback. My logic for this is that L4S can be /unfair/ against Classic flow(s) in a Classic ECN AQM, but it is not /unsafe/. Unsafe means starvation, but Classic flows always still progress against L4S.

	[SM] Could you please give a definition of "starvation", in terms of share of a bottleneck. Without a proper numeric definition that sub thread is going to de-rail quickly. Here is mine: giving a flow less than 1:8 of its equitable share on a bottleneck is starvation in my book. (I accept that too many flows will result in almost unusable throughput for everybody, but that is a different starvation criterion, absolute minimal throughput, which for the discussion at hands seems less suited).

> So, it should be sufficient for L4S senders to /monitor/ for Classic AQMs, at least in the initial stages of the L4S experiment.

	[SM] Or better keep the L4S experiment isolated from the rest of the internet, then you can do whatever you want, no? 

> 
> This can then become part of the monitoring infrastructure that the L4S ops guidance proposes for the L4S experiment. It can highlight where classic ECN AQMs might have been deployed.

	[SM] How that? You will need numbers for your classifiers performance versus the ground truth to assess that, but by running over the uncontrolled internet, you will not have ground truth information. So please elaborate how you intend to use your classifier's output to assess the quality of said classifier itself?

> Then, if it produces false positives,

	[SM] How do you plan to figure out whether a detection is a hit, or a false positive and whether a non-detection is a correct rejection or a miss, if you do not run the experiment over a know network with reliable information about the exact rfc3168-ness of each AQM along the path?

> further digging should discover that there is not actually any classic ECN AQM where it says there is. Then at least false positives don't harm performance.

	[SM] This is IMHO the wrong mindset, do not optimise prematurely for you preferred case, but rather first see how reality looks and then optimize... 

> 
> If it turns out that there are a lot of classic ECN AQMs out there, it then becomes important to improve the detection algo so it can be used in-band for fall-back. On the other hand, if we find there is very little, it becomes preferable to use the network configuration techniques described in the ops guidance to alter the classic ECN AQMs to isolate classic ECN and L4S ECN traffic.

	[SM] This seems all quite optimistic, unless we get an error free rfc3168 detector first to assess the likelihood of rfc3168 AQMs on a typical internet path, no? Really, I must be missing something here, but this does not look like engineering for a proper solution here.

Best Regards
	Sebastian

> 
> Regards
> 
> 
> 
> Bob
> 
> PS. There are also the cases where L4S flows don't perform well (particularly around start-up behaviour) in a Classic ECN AQM within an FQ scheduler (e.g. the experiments Pete did with L4S over FQ_CoDel and with Cake). Over the summer Joakim and I have been working on faster response to marking in the Prague algo, and we need to combine that with delay measurements for these cases.
> 
> I would also like to see a commitment to improving the CoDel and Cobalt control laws to increase ECN marking more rapidly in response to consistently increasing delay. Because this has always been a problem with unresponsive flows, irrespective of whether L4S had ever appeared on the scene. I'd be willing and interested to help with that sort of work,... assuming my day job left sufficient time for this.
> 
>

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 23 Sep 2020 23:12:33 +0300

> On 23 Sep, 2020, at 11:00 pm, Sebastian Moeller <[email-masked]> wrote:
> 

It was a consensus call over whether ECT(1) should have input or output semantics, which is a somewhat different (though related) question than "L4S vs SCE".  Regardless of any subsequent spin, it ended with no consensus established for any single option presented.

It was also established at the same time that L4S in its current form was *not* sufficiently robust for deployment, and thus unlikely to succeed if forwarded to later stages of the IETF standardisation process.  That was not merely the opinion of myself and others in my "camp", so to speak, but was explicitly stated by the Chairs.  An opportunity was given for L4S to undergo further development with the aim of resolving these concerns.

If any concrete progress has been made in that direction, I have not yet seen it.  But then, Bob has apparently been out of the loop for a while.  In that context, perhaps he will consider the above a friendly memory refreshment exercise.

Of course, if I have somehow got this wrong, I'll hear corrections.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 28 Sep 2020 09:31:42 +0000

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Jonathan Morton
Sent: Wednesday, September 23, 2020 10:13 PM
To: Sebastian Moeller <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] plan for L4S issue #29

[K] There was a big consensus to proceed with the L4S semantic.

It was also established at the same time that L4S in its current form was *not* sufficiently robust for deployment, and thus unlikely to succeed if forwarded to later stages of the IETF standardization process.  That was not merely the opinion of myself and others in my "camp", so to speak, but was explicitly stated by the Chairs.  An opportunity was given for L4S to undergo further development with the aim of resolving these concerns.

[K] Operational guidelines draft was proposed, because a sender-host-only resolution (the Classic ECN AQM detection) should not be the only means to resolve potential real world problems. All involved parties should be able to take actions if problems might appear.

[K] Greg prepared a first Ops Guidelines draft, collecting all ideas.  I guess next, we should be able to close the issues they are sufficiently covered by guidelines to resolve them.

Koen.

 - Jonathan Morton

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 28 Sep 2020 09:54:51 +0000

Hi Sebastian,

>>	How confident are we that the "wide variety of conditions" under which this supposedly works is actually representative for the existing internet?

Only real world deployments will answer this question. I guess any other answer will be purely speculation...

>>	rfc3168 absent; rfc3168 not-detected:	correct rejection (true negative)

Thanks for this. I like this classification and the tabular representation a lot. I guess it is ok then to use in the draft "missed detection" and "false detection"? I assume the other correct cases will be less referred to.

>>	That is why I believe that rfc3168 detection in TCP Prague is a red herring that distracts from fixing L4S true issues. Like demonstrating that the current implementation actually performs as inte…

These are all known problems that also exist in Classic TCP on FIFO queues. Solutions are known for several (all?) of them, which can as well be implemented for L4S (Prague CCs and L4S AQMs). So I wouldn't blame L4S for those problems or expect that a reference implementation is including all solutions to all known problems.

Koen.

-----Original Message-----
From: Sebastian Moeller <[email-masked]> 
Sent: Wednesday, September 23, 2020 7:47 PM
To: De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Cc: Pete Heist <[email-masked]>; Wesley Eddy <[email-masked]>; [email-masked]
Subject: Re: [tsvwg] plan for L4S issue #29

Hi Koen,

Best Regards
	Sebastian

> To: De Schepper, Koen (Nokia - BE/Antwerp) 
> <[email-masked]>; Wesley Eddy 
> <[email-masked]>
> Cc: [email-masked]
> negatives" and "false positives" terminology to something clearer, 
> like "mis-identification of RFC 3168 bottlenecks as L4S", or "mis- 
>> https://github.com/L4STeam/linux/blob/0e7cf8acb318873c3f61084453f8da1
>> 5 b2e398be/net/ipv4/tcp_prague.c , line 158). I don’t know if that 
>> was left in only for testing.
>>

---

From: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
Date: Mon, 28 Sep 2020 10:17:50 +0000

I assume Bob means the L4S drafts.

My main criterions were: Incentive to get ECN finally widely deployed by having a "big" advantage compared to non-ECT traffic, and having a simple enough network AQM that would be good enough to Classic traffic (better than tail-drop) and very good-enough for low latency (and throughput) to the new-ECN users.

Thanks for this. Your comments were noted and will certainly be taken into account by Greg (who is editing the draft). I understood from Greg that any contributions of text (filling the todo-gaps and rephrasing the existing text) are welcome.

Koen.

-----Original Message-----
From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
Sent: Wednesday, September 23, 2020 10:01 PM
To: Bob Briscoe <[email-masked]>
Cc: [email-masked]
Subject: Re: [tsvwg] plan for L4S issue #29

Dear Bob,

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 28 Sep 2020 13:43:07 +0200

Hi Koen,

thanks, more below in-line.

> On Sep 28, 2020, at 11:31, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

	[SM] That was the phrasing back then, I agree, but without an actual consumer no decision would have been necessary. So in other words, the input/output question in reality just presented an indirect "L4S or something else" choice. And many of the pro-input comments were motivated with agreement to L4S' goals, IIRC. But let's just not dwell on this, but rather figure out whether ECT(1) as input actually is a sane and safe design ;). I still wonder whether, that was not putting the cart before the horse, to commit to input versus output before assessing the respective consequences... 

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 28 Sep 2020 14:06:37 +0200

Hi Koen,

> On Sep 28, 2020, at 11:54, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 
> Hi Sebastian,
> 

	[SM] Question to all L4S proponents: You certainly most have performed experiments over the wider internet already (between carefully selected endpoints) already (as there really is no requirement for having an RFC to perform a test). Could you please share this data with the list. The open questions are robustness, safety and sharing behaviour (with non ECT(1)) for the following course conditions:
1) high RTT, multi-hop paths
2) asymmetric multi-hop paths
3) uni-laterally saturated multi-hop paths
4) bi-directionally saturated multi-hop paths

Thanks in advance.

> Thanks for this. I like this classification and the tabular representation a lot. I guess it is ok then to use in the draft "missed detection" and "false detection"? I assume the other correct case…

	[SM] Sounds okay, and probably requires less verbiage to explain.

> These are all known problems that also exist in Classic TCP on FIFO queues. Solutions are known for several (all?) of them, which can as well be implemented for L4S (Prague CCs and L4S AQMs). So I…

	[SM] Not exactly, I am referring to issues, that L4S/TCP Prague inherited from DCTCP, long ramp up times over high RTT links, and the heightened sensitivity of the reference dual queue coupled AQM has for equitable sharing on short RTTs. But I really want to see L4S TCP Prague work robustly and reliably under normal internet conditions, and at least do no harm. So I am fine with L4S choking itself (not ideal, but acceptable), but I am not happy with L4S flows crowding out non L4S flows. And for me, crowding out/starve means giving <= 1/8th of the equitable share on a link ;)

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 28 Sep 2020 21:09:31 +0300

> On 28 Sep, 2020, at 12:31 pm, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
> 

This is a straightforward factual question which, it is clear, I remember very differently from Koen and Bob.  I will quote David Black in reply to Paul Vixie, on 26th May:

> Going back to the original expression of concerns, and writing as a WG chair:
> 
>> ... i hope the chairs can explain why choice 3, more testing, was not the rough consensus they think they witnessed,
>> and also, how the time value of a choice of 1 or 2 "now" is higher than the merit value of getting to an evidence-based consensus "later".
> 
> In reverse order, the goal is to get to an evidence-based consensus "later", e.g., as indicated by the following text from the announcement email:
> 
> 	•	A successful WGLC on the drafts will require TSVWG rough consensus that the L4S experiment is safe to perform on the Internet.
> 
> I am confident that TSVWG does not currently have rough consensus that the L4S experiment is safe to perform on the Internet.  RFC 7282 is relevant background reading on rough consensus.
> 
> The only way that I could see choice 3 (more testing) being viewed as the outcome of the consensus call would be to assume that almost everyone who expressed support of choice 1 (ECT(1) as input) would also support choice 3.  Not only was that not the structure of the choices, but I am also confident that such an assumption would be incorrect.  Please keep in mind that choice 3 was not "more testing" in general - choice 3 specifically asked for specific tests:
> 
> 	3.  	There is a specific test or tests I need to see before making a decision about ECT(1). Please be specific about the tests in your response.
> 
> The time value in choosing a direction now is that the WG has spent most of the past year not getting much done in this area to the consternation (and more) of many of the people involved on all sides.  From a WG management perspective, it would be irresponsible of the WG chairs to allow this situation to continue for another year.
> 
> Thanks, --David (as a TSVWG chair)

In short, the decision to proceed with L4S for the time being was an executive one by the Chairs in the interest of unsticking the WG as a whole, and not a consensus position *of* the WG as a whole.  I would appreciate confirmation from the Chairs that I have characterised that correctly, but it seems to be supported by the above quotation.

At any rate, the prerequisite for closing blocking issues *must* be that the issues have actually been resolved, not merely that they are being moved to a forum where they have not received as much scrutiny.

 - Jonathan Morton

---

From: "Gorry (erg)" <[email-masked]>
Date: Mon, 28 Sep 2020 20:07:31 +0100

Since you asked...

> On 28 Sep 2020, at 19:10, Jonathan Morton <[email-masked]> wrote:
> 
> 
>> 
That was all that was required at the time. The decision was to focus on the working group work items and we confirmed that.

> At any rate, the prerequisite for closing blocking issues *must* be that the issues have actually been resolved, not merely that they are being moved to a forum where they have not received as much…

At some point the working group needs to publish the spec. - This final stage is taking longer than I would hope, and
I do hope that will be seeing a WGLC soon.

Gorry

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 28 Sep 2020 21:21:23 +0200 (CEST)

On Mon, 28 Sep 2020, Gorry (erg) wrote:

> At some point the working group needs to publish the spec. - This final 
> stage is taking longer than I would hope, and I do hope that will be 
> seeing a WGLC soon.

Do we actually?

I still haven't ruled out that we decide not to use these bits, for now, 
because we don't know enough how it will affect the entire Internet.

This means declaring failure and say "no" to both SCE and L4S until 
further notice.

I think both L4S and SCE has problems and I think proponents for both are 
glossing over some of these problems.

What happened to the L4S issue # 16 discussion? It seems to have died in 
June with no conclusion?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 28 Sep 2020 22:30:07 +0300

> On 28 Sep, 2020, at 10:21 pm, Mikael Abrahamsson <swmike=[email-masked]> wrote:
> 
> I think both L4S and SCE has problems and I think proponents for both are glossing over some of these problems.

The SCE team has merely stepped back a bit, watching to see if L4S actually manages to make any technical progress.  Meanwhile we are continuing work in our own direction.  We know SCE is imperfect, but we still think it is a better foundation for development.

> What happened to the L4S issue # 16 discussion? It seems to have died in June with no conclusion?

The L4S team tried to close it without actually resolving it, same as #29 right now.  Both issues need to remain open until a resolution is actually achieved.

 - Jonathan Morton

---

From: Pete Heist <[email-masked]>
Date: Tue, 29 Sep 2020 10:26:44 +0200

On Mon, 2020-09-28 at 09:54 +0000, De Schepper, Koen (Nokia -
BE/Antwerp) wrote:
> Hi Sebastian,
> 
> > > 	How confident are we that the "wide variety of conditions"
> > > under which this supposedly works is actually representative for
> > > the existing internet?
> 
> Only real world deployments will answer this question. I guess any
> other answer will be purely speculation...
> 
> Thanks for this. I like this classification and the tabular
> representation a lot. I guess it is ok then to use in the draft
> "missed detection" and "false detection"? I assume the other correct
> cases will be less referred to.

Personally, "missed detection" and "false detection" sound fine to me,
as I get the right sense of both the bottleneck and the result in both
cases.

> > > 	That is why I believe that rfc3168 detection in TCP Prague is a
> > > red herring that distracts from fixing L4S true issues. Like
> > > demonstrating that the current implementation actually performs
> > > as inteded over long holti-hop high RTT high Bandwdth links, over
> > > asymmetric links, and over uni- and bidirectionally saturated
> > > links.
> 
> These are all known problems that also exist in Classic TCP on FIFO
> queues. Solutions are known for several (all?) of them, which can as
> well be implemented for L4S (Prague CCs and L4S AQMs). So I wouldn't
> blame L4S for those problems or expect that a reference
> implementation is including all solutions to all known problems.
> 
> Koen.
> 
> > On Sep 23, 2020, at 17:46, De Schepper, Koen (Nokia - BE/Antwerp) <
> > [email-masked]> wrote:
> > 
> > Hi Pete,
> > 
> > I don't think the goal is to wait for a perfect RFC3168 detection
> > solution and we already have an implementation that works well
> > under a wide variety of conditions without additional
> > configuration.
> 
> 	How confident are we that the "wide variety of conditions"
> under which this supposedly works is actually representative for the
> existing internet?
> 
> 
> > This also means that it can still be improved, which I think can be
> > done more appropriate when facing real world issues. The goal of
> > the operational guidelines is to provide a larger set of tools for
> > solving problems with classic ECN bottlenecks, exactly to avoid the
> > need to rely on perfect end-host detection mechanism only.
> 
> 	I grudgingly agree that trying to fix L4S's AQM short comings
> by mandating fancy heuristics in the end points seems to be a loosing
> proposition. 
> 
> > Related to the "false negatives" and "false positives" naming, I
> > agree that it is very confusing. As the goal is to detect Classic
> > ECN network AQM behavior, maybe better and shorter names could be
> > "false-detect" and "false-non-detect"? 
> 
> 	I do not think that that is helpful, false-positive and false
> positive have much better well-known definitions that actually apply
> here. So if other nomenclature should be used, let's make it clearly
> different or better yet use already established terms. 
> 
> 	We could switch to signal detection theory terms (which is just
> one way to think about a classifier with just two options), then the
> four combinations of truth and classification for our RFC3168
> detector would be described with the following terms:
> 
> these terms are pretty much standard for similar detection problems
> (and also offer a decent approach to assess the effectiveness of the
> detector).
> 
> Let's stick to some already established nomenclature, whether that is
> the true/false positive/negative one or the DST one, please (see e.g.
> https://en.wikipedia.org/wiki/Evaluation_of_binary_classifiers).
> 
> The bigger issue I have with this is that this is still just a
> "fudge" to be able write in the internet draft "transports are
> required to implement RFC3168 detection, as demonstrated in TCP
> Prague" even though everybody here should know that that is not going
> to happen. People will mark whatever with ECT(1) and we will have to
> live with the fall-out. IMHO this is a sufficient reason to put the
> smarts in the AQM, instead of the current approach of assuming that
> end-points will just do this out of goodness of their heart.
> 
> That is why I believe that rfc3168 detection in TCP Prague is a red
> herring that distracts from fixing L4S true issues. Like
> demonstrating that the current implementation actually performs as
> inteded over long holti-hop high RTT high Bandwdth links, over
> asymmetric links, and over uni- and bidirectionally saturated links.
> 
> Best Regards
> 	Sebastian
> 
> > Koen.
> > 
> > I can definitely understand the need to turn bottleneck detection
> > on and off for testing, or for additional knobs during development.
> > 
> > Overall, I suspect that there will be more questions about
> > potential problems if bottleneck detection is not a MUST for
> > implementations in the draft, or not baked into the final
> > implementation in a way that works well under a wide variety of
> > conditions without additional configuration.
> > 
> > > > In contrast to other issues, it doesn't seem like this should
> > > > block 
> > > > a WGLC on the L4S drafts.
> > > >

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Tue, 29 Sep 2020 08:49:57 -0700 (PDT)

> because we don't know enough how it will affect the entire Internet.

I can support the "we don't know enough" statement.

Sadly this was not presented as a choice during the "vote" on
ECT(1) input vs output, but it should have been.  In fact I thought it
was going to be.  Alisha Cooper clearly mentioned it during her suggestion
to NOT attempt to take a consensus call during the end of the virtual
meeting due to lack of time.  Yet what came out was a different
3rd option, which personally I just do not understand as it was
extremely narrow.  Further what was presented to SCE as a selection
between ECT(1) as an input or output without any concern over this
being L4S vs SCE was clearly NOT presented to the WG in that light,
but instead clearly presented as exactly that, L4S vs SCE.

> This means declaring failure and say "no" to both SCE and L4S until 

Given that SCE is not an adopted work this is more or less
the defacto state for SCE, as it only gets WG time if the
chairs so decide it is worthy of spending it.  On the other
hand L4S has consumed a large amount of not only TSVWG time, but
also of TCPM and ICCRG, and issues with it identified long ago
have not progressed.  Is it time to put L4S to bed as a failure?
Perhaps, it claims to have a large group of supporters, yet the work
on it seems to be proceeding at glacial speed, and only by a few
visible people.

> glossing over some of these problems.

I do not believe that SCE has glossed over any problems,
when we see one we have noted it and infact gone to great
efforts to find problems, characterize them and seek solutions.
It has been often that during that process we have identified
problems with L4S, characterized them and provided feedback
to the WG on what we have found.

We are well aware of the defects in SCE and continue to work
on resolving them.  We fully admit that SCE, as it is documented
today, is not ready for deployment at scale, BUT we argue at least
it is SAFE to deploy at scale.

I would ask that you please identify a problem that SCE
has glossed over, so that we might make certain it is on
our known issues list and that we might address it.

> What happened to the L4S issue # 16 discussion? It seems to have died in 

Someone else in the thread has commented on this.

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 29 Sep 2020 17:04:17 +0100

See below.

>>> stage is taking longer than I would hope, and I do hope that will be
Yes.
>> because we don't know enough how it will affect the entire Internet.

Still possible, if the WG as a whole decides that.

Gorry

---

From: Greg White <[email-masked]>
Date: Tue, 29 Sep 2020 22:48:18 +0000

On 9/29/20, 10:05 AM, "tsvwg on behalf of Gorry Fairhurst" <[email-masked] on behalf of [email-masked]> wrote:

    See below.
    
    Yes.

We're talking about WGLC to begin an experiment. The interest by the community in achieving the expected benefits of L4S was well represented at the consensus call. I believe that the WG needs to honor the consensus position and move forward with planning the experiment. With appropriate guidance I believe the experiment can commence and we can begin to understand whether any of the concerns raised are real, and if they are, how to resolve them. For the L4S Operational Guidance draft, I would appreciate constructive input from those who are interested in making the experiment a success (thanks already to Sebastian for his comments).  Important aspects include: If a sender wishes to test an L4S congestion control algorithm (Prague or otherwise) what should they be monitoring to understand how much of a benefit are they getting, what impact are they causing to non-L4S flows, etc.? If a network operator wishes to test an L4S compliant network element, what should they monitor?  What actions should either entity take with the resulting information?  

In my opinion, Issue 29 can be closed. RFC3168 detection should continue to evolve and should be referenced in the Operational Guidance draft, but fallback should not be required in the experimental protocol drafts for the reasons outlined in Issue 29, and on the mailing list.

-Greg
    
[snip]

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 30 Sep 2020 06:29:37 +0300

> On 30 Sep, 2020, at 1:48 am, Greg White <[email-masked]> wrote:
> 
> I believe that the WG needs to honor the consensus position

Let me be absolutely crystal clear:  There was NO consensus position established.  Please go back and read the statements by the Chairs if you are still unclear on that point.

The Chairs made an executive decision, in the absence of consensus, to allow work to continue on L4S.  Progress is therefore expected on resolving the open issues to the satisfaction of the WG as a whole.  The SCE team is respecting that decision for the time being, but we will continue to call out deficiencies that have not been rectified and any further attempts to bypass the processes of the WG.

> I am confident that TSVWG does not currently have rough consensus that the L4S experiment is safe to perform on the Internet.  RFC 7282 is relevant background reading on rough consensus.

That is the central problem that L4S needs to resolve.  If it cannot do so, it should step aside so that alternative approaches to high-fidelity congestion control can be tried.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 30 Sep 2020 08:50:11 +0100

> That is the central problem that L4S needs to resolve.  If it cannot do so, it should step aside so that alternative approaches to high-fidelity congestion control can be tried.

Specifically on the Interim outcomes: The currently chartered work items 
on L4S remain chartered, and the Chairs expect these to be requested for 
publication as RFCs, following a WGLC. That WGLC will (as usual) 
consider safety of the method for the proposed experimental deployment. 
There are also other ECN specifications that the WG is updating, which 
need to be finalised first.

I am confident there is energy in the Working Group to complete the 
current set of ECN-related specs, and I'd strongly encourage people to 
focus on offering text that allows this to now complete.

Gorry

(TSVWG Co-Chair)

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 30 Sep 2020 10:00:22 +0200

Dear Greg,

more below in-line, prefixed with [SM]...

> On Sep 30, 2020, at 00:48, Greg White <[email-masked]> wrote:
> 
> 
> 
> We're talking about WGLC to begin an experiment.

	[SM] I believe that we are conflating two "experiments" here:

1) the absolutely required experiments to what degree L4S will realize its claimed charcteristics (or its promises) under a number of conditions that are relevant/prevalent in the real world.

2) how to deploy L4S in the real world.

If we look carefully at these two experiments it becomes obvious, that an RFC in the experimental track seems necessary for 2), but 1) can and should proceed long before 2) is being addressed. To be blunt, if experiment 1) is not clearly demonstrating improvements over the state of the art, then 2) becomes moot.

@CHAIRS: Could you please let me know, if you agree?

> The interest by the community in achieving the expected benefits of L4S was well represented at the consensus call. I believe that the WG needs to honor the consensus position and move forward with planning the experiment.

	[SM] That is a call for experiment 1), but the drafts really aim for experiment 2).

> With appropriate guidance I believe the experiment can commence and we can begin to understand whether any of the concerns raised are real, and if they are, how to resolve them.

	[SM] I disagree, at the current time, neither the drafts (L4S and operational guidance) are properly tailored for type 1) experiments.

> For the L4S Operational Guidance draft, I would appreciate constructive input from those who are interested in making the experiment a success (thanks already to Sebastian for his comments).  Important aspects include: If a sender wishes to test an L4S congestion control algorithm (Prague or otherwise) what should they be monitoring to understand how much of a benefit are they getting, what impact are they causing to non-L4S flows, etc.? If a network operator wishes to test an L4S compliant network element, what should they monitor?  What actions should either entity take with the resulting information?  

	[SM] These are all good points. Let me add a few thoughts here.

Let's assume an endpoint wants to participate in testing it will require a few things:
1) a L4S-compatible transport that sets ECT(1), L4S-incompatible transports that set ECT(1), as well as standards compliant transports (ECT(0)/ no ECN)
2) one/more L4S-compatible remote endpoints
3) a L4S AQM at a bottleneck, that will allow monitoring all traffic (the only way endpoints will be able to assess the side-effects of L4S traffic is to actually terminate all such flows).

Point 3) alone makes it clear that for this kind of endpoint driven experiment, only small scale AQMs will be suitable, like AQMs on an end-user internet access link, that are exclusively managing that links traffic and do not interact with other access links' AQM instances. 

And given these constraints, all that seems required for type 1) experiments IMHO is to put the access link AQMs under end-user control (preferably with three states: AQM off, all traffic, all traffic is treated like ET(0), full L4S-compliant behavior). If such an AQM defaults to AQM off, this is already save to deploy today, without requiring any additional RFC...
To be explicit, this opt-in approach allows to control the fall-out quite well; any side-effects of the L4S AQability testingM misbehaving will be mostly restricted to the end-node that opted-in. Sure that is not 100% safe, but it will allow the crucial type 1) experiments to proceed without having to commit to ECT(1) at the IETF RFC level.

Now, realistically there is nothing so far, that would have made that course of action impossible even today, so I wonder why such a setting has not already been used to make the required robustness and reliability tests that L4S is lacking (since almost a decade now?)

As is, what I predict is going to happen, is that ECT(1)/L4S is going to end up as RFC without proper robustness and reliability testing, heck without even a clear indication that its promises will realize over anything but short RTT, low hop-count links, by sheer impulse conservation. 

Some comments by the chairs seem to indicate that we are already well along that path... but there is still time to do it right. L4S has been slow enough that waiting another X years can not be a showstopper, not that the required type 1) experiments necessarily would take years...

Best Regards
        Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 30 Sep 2020 09:21:53 +0100

See below.

On 30/09/2020 09:00, Sebastian Moeller wrote:
> 1) the absolutely required experiments to what degree L4S will realize its claimed characteristics (or its promises) under a number of conditions that are relevant/prevalent in the real world.
Let me try as one chair who has seen several TSVWG experiments over the 
years:

The WG chose to adopt this some time ago, presumably because they saw 
the potential to offer benefit over the state of the art. We now have a 
set of working group documents.

The primary consideration for progression of this type of specification 
is understanding of the safety of the method for at-scale deployment.  
Specifically, for such an experiment, appropiate controls need to be in 
place to actively prevent congestion collapse, and avoid 
security-related pitfalls. The potential set of issues need to be 
teased-out for network operators and the WG to understand how to 
evaluate this.

Technical evaluation of performance, latency, fairness, etc is always 
welcome - including discussion of published papers and presentations of 
results (e.g., in ICCRG). This is however, not the purpose of the TSVWG 
specification.

In the future, those choosing to implement and deploy the specification 
will provide most useful input to any future progression along the 
standards track, and to inform any resulting best current practice - or 
indeed, for this WG to understand whether the experiment is ultimately 
deemed successful or unsuccessful.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 30 Sep 2020 11:09:30 +0200

Hi Gory,

> On Sep 30, 2020, at 10:21, Gorry Fairhurst <[email-masked]> wrote:
> 
> See below.
> 
> Let me try as one chair who has seen several TSVWG experiments over the years:
> 
> The WG chose to adopt this some time ago, presumably because they saw the potential to offer benefit over the state of the art.

	[SM] Please elaborate. In the ECT(0) versus ECT(1) I did not have the impression that the audience had either deep knowlegde about either L4S nor the state of the art... I base this on the comments that talked about the potential benefits, which without first running the type 1) experiments first really are just wishful thinking. Don't get me wrong, often a initial gut-feeling is a good impulse to start an experiment on, but one needs to keep open for the real possibility that a noice idea might crumble once it gets in contact with reality.

> We now have a set of working group documents.

	[SM] As I indicated in the past, the way the WG acted has set us on an almost unstoppable path to actually publish these drafts as RFC come hell or high water, without any substantial changes. All that based on the feeling that it would be nice if something like that would work over the internet. Color me not impressed.

> 
> The primary consideration for progression of this type of specification is understanding of the safety of the method for at-scale deployment.

	[SM] I would argue that the first question should be is at-scale deployment actually a viable course of action, which IMHO requires more diverse testing first. Neither of the current drafts pays any consideration what happens if the experiment fails and how to un-roll it. Following that course we will end up with L4S AQMs deployed (but potentially not activated) and the ECT(!) codepoint consumed, independent of whether L4S will actually see any meaningful use.
So the LEAST I would expect, if we insist upon publishing these RFC before doing the safety experiments to describe in the drafts how to un-roll each of them and explicitly describe in what time frame the scarce resource (IP header codepoint) will be able to be returned for other uses. 
	The way I see it, either first make sure that no roll-back is going to be likely (by performing type 1) experiments to convince ourselves the L4S as currently designed is up fo the job) or have an actionable plan of how to remove the fall-out of a failed experiment, and that action plan needs to be part of the internet draft RFC, especially in regards to the future use of ECT(1) in case of failure or lack of market adoption of L4S.

@CHAIRS: Since this seems so obvious to me, but also quite contrary to what the censensus in the WG is, could you please explain, what I am missing and why the current approach, "no proper experiments before an RFC and no documented rules how to undo the experiment" is actually aligned with the WG's aims and goals.

>  Specifically, for such an experiment, appropiate controls need to be in place to actively prevent congestion collapse, and avoid security-related pitfalls. The potential set of issues need to be teased-out for network operators and the WG to understand how to evaluate this.

	[SM] I would agree, but as said before, this is well possible right now without and of the drafts progressing to RFC status.

> 
> Technical evaluation of performance, latency, fairness, etc is always welcome - including discussion of published papers and presentations of results (e.g., in ICCRG). This is however, not the purpose of the TSVWG specification.

	[SM] IMHO this is backwards, without something demonstrably working robustly and reliably (over the existing internet) writing and publishing an RFC seems quite speculative, and in this case actually dangerous, as https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-10#page-20 states

"6.  L4S Experiments

   [I-D.ietf-tsvwg-aqm-dualq-coupled] sets operational and management
   requirements for experiments with DualQ Coupled AQMs.  General
   operational and management requirements for experiments with L4S
   congestion controls are given in Section 4 and Section 5 above, e.g.
   co-existence and scaling requirements, incremental deployment
   arrangements.

   The specification of each scalable congestion control will need to
   include protocol-specific requirements for configuration and
   monitoring performance during experiments. Appendix A of [RFC5706]
   provides a helpful checklist.

   Monitoring for harm to other traffic, specifically bandwidth
   starvation or excess queuing delay, will need to be conducted
   alongside all early L4S experiments.  It is hard, if not impossible,
   for an individual flow to measure its impact on other traffic.  So
   such monitoring will need to be conducted using bespoke monitoring
   across flows and/or across classes of traffic."

There is exactly zero mentioning how to undo the L4S experiment in case of the expected failure*. There is also zero mentioning how ECT(1) is going to be recycled. To be explicit, if the consensus is to allow the L4S experiment to spoil the ECT(1) codepoint for years to come, this absolutely MUST be spelled out explicitly in the RFC, so that everybody knows up front what is at stake.

*) I say expected failure as by now, I take the lack of data demonstration robust and reliable functionality of L4S over rather normal internet conditions (outside of the overtested "close CDN to end-host" condition) as an indirect sign, that this data is hard to produce, indicating that L4S will not keep its promises over longer haul paths but we will be left to carry its considerable cost.

> 
> In the future, those choosing to implement and deploy the specification will provide most useful input to any future progression along the standards track, and to inform any resulting best current practice - or indeed, for this WG to understand whether the experiment is ultimately deemed successful or unsuccessful.

	[SM] If we nominally still allow for data to convince us that the experiment (what ever the experiments actually is) was unsuccessful, I believe the drafts need explicit sections discussing in detail what should happen on experiment termination. This has been brought up in the past by others but has not made it into the drafts as far as I can see. 

Best Regards
	Sebastian

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 30 Sep 2020 23:03:09 -0400

Replying to your question to the chairs:

> @CHAIRS: Could you please let me know, if you agree?

We've heard very strongly from the working group that enabling 
low-latency is very important to a wide number of stakeholders right 
now.  Several teams have been experimenting with this, and it's been 
reported that the industry is actively working on L4S implementations.

We also have clearly heard a number of people that are so far 
unconvinced of the utility or capabilities in their analysis.  I think 
this shows that the operational considerations work that Greg has 
started editing is important.  It can help people experimenting with 
methodology for collecting data and having appropriate risk awareness 
and management plans.

So, from my viewpoint as a chair, I don't think the working group seems 
to want to block right now on convincing every single person to love 
L4S, but rather to make sure that it's setup to be sufficiently safe for 
the wider experimentation plans that we've heard of in prior meetings.