Subject: [PANRG] On the new text on ECN: draft-irtf-panrg-what-not-to-do-16

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 11 Mar 2021 15:06:03 +0000

HI,

Two observations on ECN in what may have been learned?

* I think the "one chance" is per one /Generation/ of equipment for 
hardware, possibly less time for software updates - but hard to 
eliminate deployed critical bugs completely.

* "Measure widely before, but important also gently test during initial 
deployment", to avoid the pain when there are issues and therefore to 
provide a chance to refine the design if there is a problem, and reduce 
the pain of trying.

I like the additional thought from Eric (in the RG meeting) on 
mitigating user pain by using fallback techniques, so they do not share 
your pain when something happens to go wong.

————

Is this a typo?: /Cannot be recovered at TCP layer/
- it seems like a partial sentence, does need a /This.../

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 11 Mar 2021 16:17:32 +0000

Resend: On 11/03/2021 15:06, Gorry Fairhurst wrote:
> * "Measure widely before, but important also gently test during 
> initial deployment", to avoid the pain when there are issues and 
> therefore to provide a chance to refine the design if there is a 
> problem, and reduce the pain of trying.
>
> mitigating user pain by using fallback techniques, so they do not 
> share your pain when something happens to go wong.
>
> ————
>
> _______________________________________________
> Panrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/panrg

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Mar 2021 17:12:52 +0000

Gorry, and panrg,

I'd agree with Gorry, about one chance per generation.

I'd also say that there are two main prongs to the ECN story. Not just 
the "one chance" point.
The other is "Outperforming End-to-end Protocol Mechanisms"

When I was asked to write the business case for deploying ECN in BT, my 
colleagues in the tech strategy team pushed me on this point of 
outperforming e2e mechanisms. Jamal Salim's performance evaluation 
[RFC2884] showed ECN gave no performance benefit, except for short 
flows. I had to admit that e2e FEC for short flows would be more likely 
to win out. And at the time, I'd just found Damon Wischik paper about 
how little the extra percentage of traffic volume would be, if every 
sender just duplicated the first few packets of each flow [Wischik08].

So the business case flat-lined. End systems had long since worked out 
tonnes of loss-hiding tricks. There was far too great a risk that, by 
the time the 3-part deployment had got anywhere (sender, receiver and 
network), the problem would have been solved e2e, and the high-risk 
high-cost investment would all have been wasted.

This was actually the start of my journey to realize that the "ECN = 
drop" rule was the problem, 'cos it disallowed the real benefit of ECN - 
to cut queuing delay, by providing a finer-grained signal than loss. I 
did some calculations to work out that the noise in the delay signal was 
too great to get queue delay down as low as would be achievable with ECN 
(which could use virtual queues once ISPs realized that bandwidth had 
become plentiful enough). That was actually when I started working on 
chirping (bringing in Mirja as a research fellow) and came to the 
conclusion that we could only get a better delay signal out of the noise 
by creating more noise with the chirps. That's when I realized the 
chirps should only be used at start-up, and ECN was the only way to keep 
queueing extremely low under load. Today you can see the limits of using 
e2e delay to reduce queuing in BBR, although of course BBR is unlikely 
to be the last word in e2e delay reduction.

I know it's unlikely that every ISP went through such a rigorous 
exercise, but still none deployed it - probably intuitively reaching the 
same conclusion. The main reason being the deployment pain wasn't worth 
the small performance gain. Which is a TL;DR summary of all the above.

The issue with Linksys home routers crashing wasn't a biggie in that 
assessment of ECN - there were few enough of that model around by then 
that it could be worked round with pre-deployment validation testing 
from the OS. This supports Gorry's "one chance per generation" point.

Reference
[Wischik08] Wischik, D. "Short Messages" Philosophical Transactions of 
the Royal Society A, 2008, 366, 1941-1953

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 11 Mar 2021 17:37:11 -0600

Thanks for continuing this discussion on the mailing list. If I might add
this ...

> I'd agree with Gorry, about one chance per generation.

My understanding (and we're talking about this to understand it better) is
that there are two things in play with the early ECN deployment experiment.

Yes, routers crashing when they say non-zero ECN bits was in a device which
(at the time) wasn't likely to be updated or replaced for a number of years
(see "equipment generation"), but the second point (and I think it was
Kireeti who said this well) was that once you've updated or replaced all of
the problematic equipment, the people who turned ECN off didn't immediately
turn ECN back on (I remember the phrase "a bad taste in their mouths"
mentioned multiple times.

So, as Kireeti and others observed, generations between updates are getting
shorter, but it's not clear that the ability of people to forgive and
forget past experiences is getting shorter in the same way. I'm remembering
the quote from Oscar Wilde, "Second marriage is the triumph of hope over
experience" - I think we're talking about the same thing as "second
marriage".

Best,

Spencer

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 12 Mar 2021 00:41:27 +0000

Spencer,

The apocrypha in the draft needs to tell the whole story. The current 
message is that everything was done right, except we got blind-sided by 
a bug. No.

ECN involved a /three/-part deployment, client, server /and/ bottleneck 
AQM. The crashing home routers pushed back the client part, and servers 
until they finally realized they could enable ECN without making anyone 
more vulnerable to the home router crashes. But my email was about why 
the network part never got deployed (until the more recent FQ-CoDel and 
CAKE deployments).

Even once the bug was out of the way, the deployment pain for networks 
wasn't worth the small performance gain. In the language of the draft, 
ECN didn't "Outperform End-to-end Protocol Mechanisms", which had had 
plenty of time to mask losses in other ways before ECN arrived  (all 
evidenced in my original posting).

The final deployment of the network part in FQ-CoDel and CAKE is more 
difficult to explain. I think that was down to the nature of the open 
source communities that built them. A network operator would have 
assessed whether ECN's performance gain was worth the cost of deployment 
(as in the story in my original posting). But for the open source 
community the ability to function efficiently was enough. It wasn't 
driven by the more traditional cost-benefit analysis approach of 
commercial operators.

In contrast, if you look at most machines at the broadband access 
bottleneck from the major vendors (Nokia, Ericsson, Siemens, etc), they 
didn't implement AQM, let alone ECN, 'cos the operators they were 
selling to tendered for QoS features, which they understood as something 
the network alone provides, not something it helps end-systems to 
provide for themselves (as in AQM & ECN).

Bob

On 11/03/2021 23:37, Spencer Dawkins at IETF wrote:
> Thanks for continuing this discussion on the mailing list. If I might 
> add this ...
>
> On Thu, Mar 11, 2021 at 11:13 AM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Gorry, and panrg,
>
> My understanding (and we're talking about this to understand it 
> better) is that there are two things in play with the early ECN 
> deployment experiment.
>
> Yes, routers crashing when they say non-zero ECN bits was in a device 
> which (at the time) wasn't likely to be updated or replaced for a 
> number of years (see "equipment generation"), but the second point 
> (and I think it was Kireeti who said this well) was that once you've 
> updated or replaced all of the problematic equipment, the people who 
> turned ECN off didn't immediately turn ECN back on (I remember the 
> phrase "a bad taste in their mouths" mentioned multiple times.
>
> So, as Kireeti and others observed, generations between updates are 
> getting shorter, but it's not clear that the ability of people to 
> forgive and forget past experiences is getting shorter in the same 
> way. I'm remembering the quote from Oscar Wilde, "Second marriage is 
> the triumph of hope over experience" - I think we're talking about the 
> same thing as "second marriage".
>
> Best,
>
> Spencer
>
>     >> [email-masked] <mailto:[email-masked]>
>     <https://www.irtf.org/mailman/listinfo/panrg>
>     >
>     >
>     > [email-masked] <mailto:[email-masked]>
>     <https://www.irtf.org/mailman/listinfo/panrg>
>
>     -- 
>     Bob Briscoe http://bobbriscoe.net/ <http://bobbriscoe.net/>
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.irtf.org/mailman/listinfo/panrg>
>
>

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 12 Mar 2021 07:24:40 +0000

Hi panrg,

I think I agree with Bob’s main point, if I’ve understood it correctly.

A note worth highlighting for the innocent ECN bystanders here:
Most of the benefits from FQ-codel and CAKE come from running an AQM and making it into a throughput bottleneck, which (I think uncontroversially) seems to have given substantial latency benefits to most of the people who’ve deployed it.

The ECN marking itself (as opposed to the AQM that’s doing the marking, but could just as easily drop the packet instead of marking) usually provides only the marginal benefit of avoiding one retransmitted packet, usually on a TCP fast retransmit (plus a few benefits from corner cases where you wouldn’t have gotten 3 dup acks, or a lost packet otherwise would have hurt more than usual).

There’s a fundamental problem that derives from treating loss as the right congestion signal to use, and then defining the CE marking to mean “respond the same as loss”.  In some sense the big problem underneath this came from landing on Reno instead of Vegas (along with policing aggressive senders) [1].

(It might even be right to say that almost all our later congestion-related problems derive from that early “Reno vs. Vegas” outcome, the fallout from which still hasn’t really been fixed yet.  This for instance is a key reason that large tail-drop buffers consistently cause poor latency, and I can state with pretty good confidence that it dramatically complicated the deployment of alternative congestion controllers.)

Anyway, this point about ECN performance is well-taken IMO, and it makes me think maybe there’s a good addition to the “test gently during initial deployment” advice that’s something like “characterize and quantify the achievable benefits for the bulk of the traffic that’s supposed to get benefits before nailing down semantics for scarce codepoints” [2].

Best regards,
Jake

PS:  I’m not vouching for Bob’s side comments about characterizing the reasoning behind the ISP and open-source communities’ decision processes; I imagine those a little differently but just as speculatively, so I won’t debate them here.  I’m just agreeing with the specific point about the realizable performance gains from 3168 ECN as compared to what’s achievable without using up the codepoints.

[1] I’ll just give one reference to probably my favorite explanation of the dynamics here, for those who haven’t read up on this:
“Analysis and improvement of fairness between TCP Reno and Vegas for deployment of TCP Vegas to the Internet”, Hasegawa et.al.
https://ieeexplore.ieee.org/abstract/document/896302

[2] Worth noting is that on Sally Floyd’s ECN page, referenced from RFC 3168, the first question in the list of open ECN issues is about quantifying the benefits of ECN for TCP:
http://www.icir.org/floyd/ecn.html#issues
“Open issues for ECN:
-          What are the quantitative benefits of ECN for TCP?”

From: Bob Briscoe <[email-masked]>
Date: Thursday, March 11, 2021 at 4:41 PM
To: Spencer Dawkins at IETF <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [PANRG] On the new text on ECN: draft-irtf-panrg-what-not-to-do-16

Spencer,

The apocrypha in the draft needs to tell the whole story. The current message is that everything was done right, except we got blind-sided by a bug. No.

ECN involved a /three/-part deployment, client, server /and/ bottleneck AQM. The crashing home routers pushed back the client part, and servers until they finally realized they could enable ECN without making anyone more vulnerable to the home router crashes. But my email was about why the network part never got deployed (until the more recent FQ-CoDel and CAKE deployments).

Even once the bug was out of the way, the deployment pain for networks wasn't worth the small performance gain. In the language of the draft, ECN didn't "Outperform End-to-end Protocol Mechanisms", which had had plenty of time to mask losses in other ways before ECN arrived  (all evidenced in my original posting).

The final deployment of the network part in FQ-CoDel and CAKE is more difficult to explain. I think that was down to the nature of the open source communities that built them. A network operator would have assessed whether ECN's performance gain was worth the cost of deployment (as in the story in my original posting). But for the open source community the ability to function efficiently was enough. It wasn't driven by the more traditional cost-benefit analysis approach of commercial operators.

In contrast, if you look at most machines at the broadband access bottleneck from the major vendors (Nokia, Ericsson, Siemens, etc), they didn't implement AQM, let alone ECN, 'cos the operators they were selling to tendered for QoS features, which they understood as something the network alone provides, not something it helps end-systems to provide for themselves (as in AQM & ECN).

Bob
On 11/03/2021 23:37, Spencer Dawkins at IETF wrote:
Thanks for continuing this discussion on the mailing list. If I might add this ...

On Thu, Mar 11, 2021 at 11:13 AM Bob Briscoe <[email-masked]<mailto:[email-masked]>> wrote:
Gorry, and panrg,

My understanding (and we're talking about this to understand it better) is that there are two things in play with the early ECN deployment experiment.

Yes, routers crashing when they say non-zero ECN bits was in a device which (at the time) wasn't likely to be updated or replaced for a number of years (see "equipment generation"), but the second point (and I think it was Kireeti who said this well) was that once you've updated or replaced all of the problematic equipment, the people who turned ECN off didn't immediately turn ECN back on (I remember the phrase "a bad taste in their mouths" mentioned multiple times.

So, as Kireeti and others observed, generations between updates are getting shorter, but it's not clear that the ability of people to forgive and forget past experiences is getting shorter in the same way. I'm remembering the quote from Oscar Wilde, "Second marriage is the triumph of hope over experience" - I think we're talking about the same thing as "second marriage".

Best,

Spencer

Reference

Bob

On 11/03/2021 16:17, Gorry Fairhurst wrote:
>> [email-masked]<mailto:[email-masked]>
>> https://www.irtf.org/mailman/listinfo/panrg<https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>
> [email-masked]<mailto:[email-masked]>
> https://www.irtf.org/mailman/listinfo/panrg<https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>

_______________________________________________
Panrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/panrg<https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>

Panrg mailing list

[email-masked]<mailto:[email-masked]>

https://www.irtf.org/mailman/listinfo/panrg<https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>

--

Bob Briscoe                               http://bobbriscoe.net/<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 12 Mar 2021 09:06:18 +0000

This is helpful ... yesterday I was refelecting on the benefits of using 
AQM/RED (a different story) ... however, I'd agree with Jake: to me 
successful deployment of features that require new configuration needs 
operators to perceive the achievable benefits for the bulk of the traffic.

Gorry

On 12/03/2021 07:24, Holland, Jake wrote:
> Most of the benefits from FQ-codel and CAKE come from running an AQM 
> and making it into a throughput bottleneck, which (I think 
> uncontroversially) seems to have given substantial latency benefits to 
> most of the people who’ve deployed it.
>
> The ECN marking itself (as opposed to the AQM that’s doing the 
> marking, but could just as easily drop the packet instead of marking) 
> usually provides only the marginal benefit of avoiding one 
> retransmitted packet, usually on a TCP fast retransmit (plus a few 
> benefits from corner cases where you wouldn’t have gotten 3 dup acks, 
> or a lost packet otherwise would have hurt more than usual).
>
> There’s a fundamental problem that derives from treating loss as the 
> right congestion signal to use, and then defining the CE marking to 
> mean “respond the same as loss”.  In some sense the big problem 
> underneath this came from landing on Reno instead of Vegas (along with 
> policing aggressive senders) [1].
>
> (It might even be right to say that almost all our later 
> congestion-related problems derive from that early “Reno vs. Vegas” 
> outcome, the fallout from which still hasn’t really been fixed yet.  
> This for instance is a key reason that large tail-drop buffers 
> consistently cause poor latency, and I can state with pretty good 
> confidence that it dramatically complicated the deployment of 
> alternative congestion controllers.)
>
> Anyway, this point about ECN performance is well-taken IMO, and it 
> makes me think maybe there’s a good addition to the “test gently 
> during initial deployment” advice that’s something like “characterize 
> and quantify the achievable benefits for the bulk of the traffic 
> that’s supposed to get benefits before nailing down semantics for 
> scarce codepoints” [2].
>
> Best regards,
>
> Jake
>
> PS:  I’m not vouching for Bob’s side comments about characterizing the 
> reasoning behind the ISP and open-source communities’ decision 
> processes; I imagine those a little differently but just as 
> speculatively, so I won’t debate them here.  I’m just agreeing with 
> the specific point about the realizable performance gains from 3168 
> ECN as compared to what’s achievable without using up the codepoints.
>
> [1] I’ll just give one reference to probably my favorite explanation 
> of the dynamics here, for those who haven’t read up on this:
>
> “Analysis and improvement of fairness between TCP Reno and Vegas for 
> deployment of TCP Vegas to the Internet”,**Hasegawa et.al.
>
> [2] Worth noting is that on Sally Floyd’s ECN page, referenced from 
> RFC 3168, the first question in the list of open ECN issues is about 
> quantifying the benefits of ECN for TCP:
>
>       -What are the quantitative benefits of ECN for *TCP*?”
>
> *From: *Bob Briscoe <[email-masked]>
> *Date: *Thursday, March 11, 2021 at 4:41 PM
> *To: *Spencer Dawkins at IETF <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>
> *Subject: *Re: [PANRG] On the new text on ECN: 
> draft-irtf-panrg-what-not-to-do-16
>
> Spencer,
>
> message is that everything was done right, except we got blind-sided 
> by a bug. No.
>
> ECN involved a /three/-part deployment, client, server /and/ 
> bottleneck AQM. The crashing home routers pushed back the client part, 
> and servers until they finally realized they could enable ECN without 
> making anyone more vulnerable to the home router crashes. But my email 
> was about why the network part never got deployed (until the more 
> recent FQ-CoDel and CAKE deployments).
>
> assessed whether ECN's performance gain was worth the cost of 
> deployment (as in the story in my original posting). But for the open 
> source community the ability to function efficiently was enough. It 
> wasn't driven by the more traditional cost-benefit analysis approach 
> of commercial operators.
>
> bottleneck from the major vendors (Nokia, Ericsson, Siemens, etc), 
> they didn't implement AQM, let alone ECN, 'cos the operators they were 
> selling to tendered for QoS features, which they understood as 
> something the network alone provides, not something it helps 
> end-systems to provide for themselves (as in AQM & ECN).
>
>
> Bob
>
>     Yes, routers crashing when they say non-zero ECN bits was in a
>     device which (at the time) wasn't likely to be updated or replaced
>     for a number of years (see "equipment generation"), but the second
>     point (and I think it was Kireeti who said this well) was that
>     once you've updated or replaced all of the problematic equipment,
>     the people who turned ECN off didn't immediately turn ECN back on
>     (I remember the phrase "a bad taste in their mouths" mentioned
>     multiple times.
>
>     So, as Kireeti and others observed, generations between updates
>     are getting shorter, but it's not clear that the ability of people
>     to forgive and forget past experiences is getting shorter in the
>     same way. I'm remembering the quote from Oscar Wilde, "Second
>     marriage is the triumph of hope over experience" - I think we're
>     talking about the same thing as "second marriage".
>
>     Best,
>
>     Spencer
>
>         <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>         >
>         >
>         <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>         -- 
>         <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>
>
>         <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>
>
>     https://www.irtf.org/mailman/listinfo/panrg  <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 12 Mar 2021 05:47:22 -0600

So, having had a few minutes to sleep on this discussion (which would never
happen in a face-to-face IETF meeting, of course) ...

> operators to perceive the achievable benefits for the bulk of the traffic.

ECN is (quite) a bit different than most of the other contributions to
draft-irtf-panrg-what-not-to-do, because its (non-)deployment history spans
a decade or two (as Gorry said during the meeting, ST2 didn't achieve
deployment, and the Internet moved on, but ECN is different).

I THINK I have the opportunity to propose new text that breaks ECN out into
a broader story - what happens when you don't achieve deployment, for what
should have been a good idea, and you keep trying to achieve deployment,
and stumble over more lessons. This is different from what-not-to-do has
mostly been about, because ECN is still an active conversation in IETF
working groups.

On the other hand, I think (as Bob said) the story of ECN (non-)deployment
has had multiple chapters, so another option would be to say that
explicitly, and confine the scope of the new text on ECN to be only on the
first chapter, and focus on the new Lessons Learned, that we talked about
during the PANRG session, amd in this thread.

I THINK I have a preference to add the caveat about the ECN history
continuing into the present day, because most of the Lessons Learned from
the continuing story are already listed in the document, just not pointing
to ECN as another case where they are displayed.

But I'd like to hear the thoughts of others about this.

Best,

Spencer

> Most of the benefits from FQ-codel and CAKE come from running an AQM and
> making it into a throughput bottleneck, which (I think uncontroversially)
> seems to have given substantial latency benefits to most of the people
> who’ve deployed it.
>
>
>
> The ECN marking itself (as opposed to the AQM that’s doing the marking,
> but could just as easily drop the packet instead of marking) usually
> provides only the marginal benefit of avoiding one retransmitted packet,
> usually on a TCP fast retransmit (plus a few benefits from corner cases
> where you wouldn’t have gotten 3 dup acks, or a lost packet otherwise would
> have hurt more than usual).
>
>
>
> There’s a fundamental problem that derives from treating loss as the right
> congestion signal to use, and then defining the CE marking to mean “respond
> the same as loss”.  In some sense the big problem underneath this came from
> landing on Reno instead of Vegas (along with policing aggressive senders)
> [1].
>
>
>
> outcome, the fallout from which still hasn’t really been fixed yet.  This
> for instance is a key reason that large tail-drop buffers consistently
> cause poor latency, and I can state with pretty good confidence that it
> dramatically complicated the deployment of alternative congestion
> controllers.)
>
>
>
> Anyway, this point about ECN performance is well-taken IMO, and it makes
> me think maybe there’s a good addition to the “test gently during initial
> deployment” advice that’s something like “characterize and quantify the
> achievable benefits for the bulk of the traffic that’s supposed to get
> benefits before nailing down semantics for scarce codepoints” [2].
>
>
>
> Best regards,
>
> Jake
>
>
>
> reasoning behind the ISP and open-source communities’ decision processes; I
> imagine those a little differently but just as speculatively, so I won’t
> debate them here.  I’m just agreeing with the specific point about the
> realizable performance gains from 3168 ECN as compared to what’s achievable
> without using up the codepoints.
>
>
>
>
>
> [1] I’ll just give one reference to probably my favorite explanation of
> the dynamics here, for those who haven’t read up on this:
>
> deployment of TCP Vegas to the Internet”, Hasegawa et.al.
>
> [2] Worth noting is that on Sally Floyd’s ECN page, referenced from RFC
> 3168, the first question in the list of open ECN issues is about
> “Open issues for ECN: -          What are the quantitative benefits of
> ECN for *TCP*?”
>
>
>
>
>
>
>
> *From: *Bob Briscoe <[email-masked]> <[email-masked]>
> <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]> <[email-masked]> <[email-masked]>
> message is that everything was done right, except we got blind-sided by a
> bug. No.
>
> more vulnerable to the home router crashes. But my email was about why the
> network part never got deployed (until the more recent FQ-CoDel and CAKE
> deployments).
>
> wasn't worth the small performance gain. In the language of the draft, ECN
> didn't "Outperform End-to-end Protocol Mechanisms", which had had plenty of
> time to mask losses in other ways before ECN arrived  (all evidenced in my
> original posting).
>
> source communities that built them. A network operator would have assessed
> whether ECN's performance gain was worth the cost of deployment (as in the
> story in my original posting). But for the open source community the
> ability to function efficiently was enough. It wasn't driven by the more
> traditional cost-benefit analysis approach of commercial operators.
>
> didn't implement AQM, let alone ECN, 'cos the operators they were selling
> to tendered for QoS features, which they understood as something the
> network alone provides, not something it helps end-systems to provide for
> themselves (as in AQM & ECN).
>
>
> Bob
>
> which (at the time) wasn't likely to be updated or replaced for a number of
> years (see "equipment generation"), but the second point (and I think it
> was Kireeti who said this well) was that once you've updated or replaced
> all of the problematic equipment, the people who turned ECN off didn't
> immediately turn ECN back on (I remember the phrase "a bad taste in their
> mouths" mentioned multiple times.
>
>
>
> getting shorter, but it's not clear that the ability of people to forgive
> and forget past experiences is getting shorter in the same way. I'm
> remembering the quote from Oscar Wilde, "Second marriage is the triumph of
> hope over experience" - I think we're talking about the same thing as
> "second marriage".
>
>
>
> Best,
>
>
>
> Spencer
>
>
>
> Bob Briscoe                               http://bobbriscoe.net/ <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>
>
>
> Panrg mailing [email-masked]://www.irtf.org/mailman/listinfo/panrg
>
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 22 Mar 2021 10:55:49 -0500

Hi, Gorry,

I'm starting to work on the updates to the ECN section
of draft-irtf-panrg-what-not-to-do, and wanted to reply to this question
specifically --

> - it seems like a partial sentence, does need a /This.../

Yeah. That is a quote from
https://www.ietf.org/proceedings/68/slides/tsvarea-3/sld1.htm, so it does
need a "This" or something like that, but since it's a quote, I left it and
added "(sic)" afterwards.

Thanks for the careful read, and for asking before the RFC Editor asks the
same question :-)

Best,

Spencer

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 22 Mar 2021 13:03:49 -0500

Dear All,

I've worked through the discussion from IETF 110 in CoviMD, on the e-mail
list, and in my own notes from our meeting.

I said I would submit a -19 at the end of this week, and I still plan to do
that, but if you want a preview, here's what I've got for the biggest
changes.

   - The new "Lesson Learned" is "Plan for Failure". It's still Invariant,
   in Table 1. The text for it is "If early implementers discover severe
   problems with a new feature, that feature is likely to be disabled, and
   convincing implementers to re-enable that feature can be very difficult,
   and can require years or decades. *In addition to testing, partial
   deployment for a subset of users, implementing instrumentation that will
   detect degraded user experience, and even "failback" to a previous version
   or "failover" to an entirely different implementation are likely to be
   helpful. (See {{ecn}})."*
   - I reflected Bob Briscoe's points about AQM deployment being part of
   the ECN story, and emphasized that the ECN material focused on experiences
   that provided Lessons Learned that were not covered by other contributions.
   - I added material to the Lesson Learned, to bring it up to date. That
   section now continues

*During discussion of ECN at {{PANRG-110}}, we noted that "you get one
chance to get it right" isn't quite correct today, because so operating
systems on so many host systems are frequently updated, which was not at
all true in the early 2000s. We think that these restatements of the ECN
Lessons Learned are more useful for current implementers: *

*- Even if you do everything right, you may trip over implementation bugs
in devices you know nothing about, that will cause severe problems that
prevent successful deployment of your path aware technology. Testing before
deployment isn't enough to ensure successful deployment. It is also
necessary to "deploy gently", which often means deploying for a small
subset of users to gain experience, and implementing feedback mechanisms to
detect that user experience is being degraded. *
*- After implementations disable your Path Aware technology, it may take
years, or even decades, to convince implementers to re-enable it by
default. This might be based on the difficulty of distributing
implementations that enable it by default, but are just as likely to be
based on the "bad taste in the mouth" that implementers have after an
unsuccessful deployment attempt that degraded user experience. *

*With these expansions, the two lessons, taken together, could be more
helpfully summarized as "plan for failure" - anticipate what your next step
will be, if initial deployment is unsuccessful. *

*ECN deployment was also hindered by non-deployment of AQM in many devices,
because of operator interest in QoS features provided in the network,
rather than using the network to assist end systems in providing for
themselves. But that's another story, and the AQM Lessons Learned are
already covered in other contributions in {{Contributions}}.*

Full details are at
https://github.com/panrg/draft-dawkins-panrg-what-not-to-do/pull/7/commits/85ed46f5f7285346672d4d81861b028d9ed09072,
of course, and comments here or in Github are always welcome.

Best,

Spencer

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 22 Mar 2021 18:06:19 +0000

On 22/03/2021 15:55, Spencer Dawkins at IETF wrote:
> of draft-irtf-panrg-what-not-to-do, and wanted to reply to this 
> question specifically --
>
> On Thu, Mar 11, 2021 at 10:13 AM Gorry Fairhurst <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     ————
>
> https://www.ietf.org/proceedings/68/slides/tsvarea-3/sld1.htm 
> <https://www.ietf.org/proceedings/68/slides/tsvarea-3/sld1.htm>, so it 
> does need a "This" or something like that, but since it's a quote, I 
> left it and added "(sic)" afterwards.
>
> Thanks for the careful read, and for asking before the RFC Editor asks 
> the same question :-)
>
> Best,
>
> Spencer

ACK,

Gorry

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 24 Mar 2021 09:45:28 -0500

Dear PANRG,

Just a reminder that I plan to submit roughly this text, this coming
Friday, and ask that -19 be approved for publication (Colin's waiting on
the update to approve it).

If you have clever thoughts about it, sooner is better than later.

And thanks to a bunch of people who improved the text, and especially the
Lessons Learned from ECN. I think this was a useful late addition to a doc
that had already left the research group.

Best,

Spencer

On Mon, Mar 22, 2021 at 1:03 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

> I said I would submit a -19 at the end of this week, and I still plan to
> do that, but if you want a preview, here's what I've got for the biggest
> changes.
>
>    - The new "Lesson Learned" is "Plan for Failure". It's still
>    Invariant, in Table 1. The text for it is "If early implementers discover
>    severe problems with a new feature, that feature is likely to be disabled,
>    and convincing implementers to re-enable that feature can be very
>    difficult, and can require years or decades. *In addition to testing,
>    partial deployment for a subset of users, implementing instrumentation that
>    will detect degraded user experience, and even "failback" to a previous
>    version or "failover" to an entirely different implementation are likely to
>    be helpful. (See {{ecn}})."*
> *ECN deployment was also hindered by non-deployment of AQM in many
> devices, because of operator interest in QoS features provided in the
> network, rather than using the network to assist end systems in providing
> for themselves. But that's another story, and the AQM Lessons Learned are

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Mar 2021 15:52:05 +0000

Thanks for doing this Spencer!

This particular bit doesn't seem to read correctly:

On 24/03/2021 14:45, Spencer Dawkins at IETF wrote:
> chance to get it right" isn't quite correct today, because so 
> operating systems on so many host systems are frequently updated, 
> which was not at all true in the early 2000s. We think that these 
> restatements of the ECN Lessons Learned are more useful for current 
> implementers: *

Is there a sprurious "so"?

- I think the argument that OS "*are frequently updated" does not sit so 
well with me. True, there are many updates that simply were not as often 
in the early 2000s. It's still quite painful introducing innovation in 
some parts of the OS.
*

*- Also perhaps more important, QUIC argued these were not enough to 
allow innovation so moved to userland stacks.
*

*I wonder then whether the words "operating systems" are actually better 
written as "protocol stacks"?*

*Gorry
*

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 25 Mar 2021 08:57:46 -0500

Hi, Gorry,

> Is there a sprurious "so"?

I think that's a missing "many", but it's borked - thanks for catching that!

> - I think the argument that OS "
> *are frequently updated" does not sit so well with me. True, there are
> many updates that simply were not as often in the early 2000s. It's still
> quite painful introducing innovation in some parts of the OS. *
>
>
> *- Also perhaps more important, QUIC argued these were not enough to allow
> innovation so moved to userland stacks. *
>

I think you're correctly capturing what I was thinking of (without typing
it). We talked about OS upgrade patterns during our discussion two
weeks ago, but the movement to userland protocol stacks is also a
significant factor. I'll name both in -19.

This is what I committed to Girhub.

During discussion of ECN at {{PANRG-110}}, we noted that "you get one
chance to get it right" isn't quite correct today, because operating
systems on so many host systems are frequently updated, and transport
protocols like QUIC {{I-D.ietf-quic-transport}} are being implemented in
user space, and can be updated without touching installed operating
systems. Neither of these factors were true in the early 2000s.

We think that these restatements of the ECN Lessons Learned are more useful
for current implementers:

Best,

Spencer

> *Gorry *