Subject: [tsvwg] David Black (individual) on safety of L4S for the Internet

From: "Black, David" <[email-masked]>
Date: Fri, 8 May 2020 20:01:01 +0000

This the longer explanation of why I find myself in group 2 on the ECT(1) question because "If you believe L4S will not be safe for the internet without significant architectural changes, you are in this group."

Current practice is not to mix DCTCP-like traffic (1/p-class congestion control) with TCP-like traffic (1/sqrt(p)-class congestion control, e.g., NewReno) in the same queue because the DCTCP-like traffic outcompetes the TCP-like traffic to the point of starvation of the latter.  By using ECT(1) as the only classifier for low-latency traffic, L4S causes mixing of those two classes of traffic at network nodes that have not been modified to use ECT(1) as a classifier.  Hence the L4S experiment proponents are responsible for the safety of that traffic mixing at unmodified nodes.  I understand the L4S approach to achieving safety at unmodified nodes to be use of TCP Prague, specifically its heuristics that detect RFC 3168 bottlenecks.  As previously discussed, this traffic mixing does not cause problems at otherwise unmodified nodes that use FQ.

For the proposed Internet-wide L4S experiment, the technology to be tested is necessarily experimental, but the safety of the experiment needs to be fundamental to avoid serious damage if things go wrong.  I do not currently see a strong safety case for L4S+TCP Prague - an example of such a strong safety case can be found in QUIC congestion control, where the fact that it is based on NewReno congestion control provides strong assurance that QUIC congestion control is safe for the Internet.   In contrast, TCP Prague is still research (it's great research and I applaud the people who have invested their time and effort in it for what has been achieved), but a safety case ought not to be based on not-yet-stable research, and I don't think another 6 months of testing and bug fixing will leave TCP Prague as anything other than research (ruling out group 3 for me).  Further, the L4S low-latency service will likely be used by protocols other than TCP Prague, which may not receive the level of scrutiny and testing that has been applied to TCP Prague.  The safety case needs to come from elsewhere.

If ECT(1)-marked traffic starts causing problems, network operators are not going to wait for TCP Prague or other protocol implementations to be patched - they're going to take action, e.g., as Neal Cardwell describes (and I've heard about this sort of potential consequence in other conversations, so please don't blame Neal for this):

> - L4S flows potentially causing unfairness in RFC3168 ECN bottlenecks has been mentioned as a potential concern. However, a robust RFC3168 ECN bottleneck
> should already have a mechanism to avoid unfairness caused by flows that are marked as ECT(0|1) and yet not performing RFC3168 responses. In particular,
> many of the large sources of known deployments of  RFC3168 --  Linux fq_codel and cake -- are already deployed with fair queueing. In such bottlenecks L4S
> traffic should not cause harm to other non-L4S flows. Furthermore, if there really are ISPs with deployments of RFC3168 bottlenecks that have neither FQ nor
> any other protection from non-RFC3168-ECT(1) flows, then they can bleach incoming ECT(1) code points to Not-ECT and treat L4S as Not-ECT (ISPs typically
> already transform the DSCP byte at their ingress anyway). So I do not see harm to RFC3168 ECN bottlenecks as a prohibitive concern.

In contrast to reliance on TCP Prague, that combination of FQ and bleaching looks like it could provide a robust safety case.  It would be most unfortunate if things wind up depending on this, especially bleaching, as the fact that ECT(1) is not bleached at network boundaries was a major reason for selection of ECT(1) as the identifier for low-latency L4S traffic.  Among the implications is that if the L4S experiment fails, that bleaching could be around much longer, making it infeasible to use ECT(1) for anything else Internet-wide.  In my view, it is the responsibility of the L4S proponents to design the L4S  technology and structure the experiment in a fashion that makes ECT(1) bleaching unlikely and/or likely to be limited in scope if it happens.

Thanks, --David
----------------------------------------------------------------
David L. Black, Sr. Distinguished Engineer, Dell EMC
Dell Technologies, Infrastructure Systems Group
176 South St., Hopkinton, MA  01748
+1 (774) 350-9323<tel:+17743509323>           Mobile: +1 (978) 394-7754<tel:+19783947754>
[email-masked]<mailto:[email-masked]>
----------------------------------------------------------------

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 8 May 2020 23:15:46 +0300

> On 8 May, 2020, at 11:01 pm, Black, David <[email-masked]> wrote:
> 

This point bears emphasis: if ECT(1) is bleached or blackholed, it will become *unavailable* for a future Internet experiment.  And I think that if network operators feel a need to do that, it will represent a failure of the L4S experiment sufficient to justify its termination, and attempt the reclamation of ECT(1).

The combination of these two facts suggests that if the L4S experiment goes ahead, it will permanently consume ECT(1).  Any choice to do so should therefore not be made lightly.

 - Jonathan Morton

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 8 May 2020 18:04:15 -0400

On Fri, May 8, 2020 at 4:01 PM Black, David <[email-masked]> wrote:
> Current practice is not to mix DCTCP-like traffic (1/p-class congestion
> control) with TCP-like traffic (1/sqrt(p)-class congestion control, e.g.,
> NewReno) in the same queue because the DCTCP-like traffic outcompetes the
> TCP-like traffic to the point of starvation of the latter.

For my education, does someone have a pointer to the experiments
underlying that description of DCTCP outcompeting Reno/CUBIC? I'm not
aware of that being a fundamental property of DCTCP. The DCTCP
algorithm specifies a Reno-style response to loss -
https://tools.ietf.org/html/rfc8257#section-3.5 - and has Reno-style
increase functions as well. So the algorithm should be Reno friendly,
AFAIK.

The description of DCTCP outcompeting Reno reminds me of the behavior
of the Linux DCTCP implementation, before this 2019 bug fix by Koen De
Schepper and Olivier Tilmans:

  https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=aecfde23108b8e637d9f5c5e523b24fb97035dc3
  tcp: Ensure DCTCP reacts to losses

AFAIK, Linux DCTCP with this fix no longer outcompetes Reno.

> In contrast to reliance on TCP Prague, that combination of FQ and bleaching
> looks like it could provide a robust safety case.

AFAICT bleaching ECT(1) to Not-ECT should be sufficient for safety,
without also requiring FQ. An L4S flow that has  ECT(1) bleached to
Not-ECT should then be treated Like Reno by the bottleneck in
question. And, like DCTCP, the L4S specs specify a Reno-compatible
loss response for TCP Prague:
  https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-09#appendix-A.1.3

So at least to a first approximation it seems like if an L4S flow is
bleached by an ISP to Not-ECT, is then treated by the ISP's
bottlenecks like non-ECN Reno, and then (per the L4S spec) behaves
like non-ECN Reno, L4S should be safe.

Apologies for the noise if I have missed something.

best regards,
neal

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 9 May 2020 01:37:42 +0300

> On 9 May, 2020, at 1:04 am, Neal Cardwell <ncardwell=[email-masked]> wrote:
> 

The problem with DCTCP is not its response to loss, though there was a serious bug that apparently went unnoticed for a long time.  Rather, it has to do with the response to CE marks applied by an AQM.  We can analyse this qualitatively and show a clear difference which will always result in severe unfairness.  This is also easy to confirm by experiment.

Assume that a single queue exists at the bottleneck, with a single ECN AQM which does not discriminate between flows and classes of traffic.  This queue and AQM are shared by diverse traffic flows, some of which adhere to RFC3168 and RFC-8511, some of which are Not-ECT, and some of which are DCTCP.  The Not-ECT packets will, in accordance with RFC-3168, be dropped if they would have been marked CE.

The steady state for both Not-ECT and RFC-3168/8511 transports (which include NewReno and CUBIC) is multiple RTTs between CE marks.  Each round-trip containing a CE mark (or a packet loss) causes a Multiplicative Decrease to between 50% and 85% of the previous congestion window, as defined by RFC-8511 and earlier well-known specifications.  It then takes several RTTs to grow the congestion window to its previous value, where a further congestion signal can be expected.

The steady state for DCTCP is two CE marks per round-trip.  Each CE mark causes half a segment to be subtracted from the congestion window, on average.  This is qualitatively different from the response specified by RFC-8511.  In particular, to obtain the 50% reduction to a single CE mark that NewReno implements, an entire congestion window's worth of segments must be marked CE by the AQM.

Two results should be obvious from the above.  First, an AQM response that causes RFC-8511 compliant transports to back off substantially may cause very little response from DCTCP.  Second, an AQM response that produces the steady state in DCTCP will also apply CE marks (or packet loss) to almost every round-trip of other competing flows, which will cause conventional TCPs' congestion windows to collapse to very small values.

That is why DCTCP is specified for deployment only in controlled environments where its interactions with conventional TCP can be strictly managed.

 - Jonathan Morton

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 8 May 2020 15:59:59 -0700

I can't parse the logic of

1) L4S / TCP-prague can have implementation bugs/violations that cause
ECT(1) to be disabled/ignored forever, hence fq+bleaching would make
it safe.
2) QUIC uses Reno so it's safe

What prevents these "safety" mechanisms from having bugs too? I have
seen vendor fq implementations that're really not fq at all. QUIC
senders can in reality run any congestion control they like.

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 9 May 2020 02:02:11 +0300

> like non-ECN Reno, L4S should be safe.

Further to my earlier reply, the fact is that a bug causing loss to be *ignored* for the purpose of congestion control has arisen at least twice in the context of 1/p schemes (both DCTCP and TCP Prague).  So relying on loss as a fallback might not be *entirely* reliable.

And I think Yuchung Cheng has noticed the same aspect of this problem.

 - Jonathan Morton

---

From: "Black, David" <[email-masked]>
Date: Sat, 9 May 2020 00:42:59 +0000

[still as an individual]

> 2) QUIC uses Reno so it's safe

The two mechanisms (Prague and Reno) are at very different stages of maturity.  There is a lot of deployment and operational experience with Reno, which is a proven mechanism at this juncture,  whereas Prague is brand new research.  In time I hope/expect to see Prague become as mature and solid as Reno, but the two are at rather different states of maturity now.

> senders can in reality run any congestion control they like.

Nothing prevents that - bugs are inevitable ;-), but with a well-understood mechanism like Reno with which there is a lot of experience, it's obvious that a bug is a bug.  There's still much to be learned about Prague, so "interesting" behavior may be a bug, an opportunity to better tune the heuristic or maybe even a feature.  Experimenting with Prague to better understand it is a good thing, but something else ought to be providing protection from the experiment going off the rails.  That something else doesn't have to be FQ+bleaching.

Thanks, --David

> -----Original Message-----
> From: tsvwg <[email-masked]> On Behalf Of Yuchung Cheng
> Sent: Friday, May 8, 2020 7:00 PM
> To: Jonathan Morton
> Cc: Neal Cardwell; [email-masked]
> Subject: Re: [tsvwg] David Black (individual) on safety of L4S for the Internet
> 
> 
> [EXTERNAL EMAIL]
> 
> > > On 9 May, 2020, at 1:04 am, Neal Cardwell
> <ncardwell=[email-masked]> wrote:
> > >
> > The problem with DCTCP is not its response to loss, though there was a
> serious bug that apparently went unnoticed for a long time.  Rather, it has to do
> with the response to CE marks applied by an AQM.  We can analyse this
> qualitatively and show a clear difference which will always result in severe
> unfairness.  This is also easy to confirm by experiment.
> >
> > Assume that a single queue exists at the bottleneck, with a single ECN AQM
> which does not discriminate between flows and classes of traffic.  This queue
> and AQM are shared by diverse traffic flows, some of which adhere to RFC3168
> and RFC-8511, some of which are Not-ECT, and some of which are DCTCP..  The
> Not-ECT packets will, in accordance with RFC-3168, be dropped if they would
> have been marked CE.
> >
> > The steady state for both Not-ECT and RFC-3168/8511 transports (which
> include NewReno and CUBIC) is multiple RTTs between CE marks.  Each round-
> trip containing a CE mark (or a packet loss) causes a Multiplicative Decrease to
> between 50% and 85% of the previous congestion window, as defined by RFC-
> 8511 and earlier well-known specifications.  It then takes several RTTs to grow
> the congestion window to its previous value, where a further congestion signal
> can be expected.
> >
> > The steady state for DCTCP is two CE marks per round-trip.  Each CE mark
> causes half a segment to be subtracted from the congestion window, on
> average.  This is qualitatively different from the response specified by RFC-
> 8511.  In particular, to obtain the 50% reduction to a single CE mark that
> NewReno implements, an entire congestion window's worth of segments must
> be marked CE by the AQM.
> >
> > Two results should be obvious from the above.  First, an AQM response that
> causes RFC-8511 compliant transports to back off substantially may cause very
> little response from DCTCP.  Second, an AQM response that produces the
> steady state in DCTCP will also apply CE marks (or packet loss) to almost every
> round-trip of other competing flows, which will cause conventional TCPs'
> congestion windows to collapse to very small values.
> >
> > That is why DCTCP is specified for deployment only in controlled
> environments where its interactions with conventional TCP can be strictly
> managed.
> >
> >  - Jonathan Morton
> >

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 8 May 2020 22:02:03 -0400

On Fri, May 8, 2020 at 6:37 PM Jonathan Morton <[email-masked]> wrote:
> The problem with DCTCP is not its response to loss, though there was a serious
> bug that apparently went unnoticed for a long time. Rather, it has to do with
> the response to CE marks applied by an AQM. We can analyse this qualitatively
> and show a clear difference which will always result in severe unfairness. This
> is also easy to confirm by experiment.

I see. Thanks for clarifying. I'm aware of the issues with DCTCP
operating in an RFC3168 AQM. The sentence introducing this thread
("DCTCP-like traffic outcompetes the TCP-like traffic") did not
specify an AQM (RFC3168 or RFC8257), so absent the mention of an AQM I
was imagining this was referring to the default case of a non-ECN
single-queue drop-tail FIFO.

However, I suppose I should have been able to deduce from the
structure of the argument that there was implicitly an RFC3168 AQM in
this scenario, so my apologies for not deducing that.

best,
neal

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 9 May 2020 12:42:06 +0300

> On 9 May, 2020, at 5:02 am, Neal Cardwell <[email-masked]> wrote:
> 

It's a bit more subtle than that: the unfairness exists regardless of the type of AQM, so long as it cannot distinguish a DCTCP flow from a conventional flow (for which FQ is sufficient).  The unfairness stems from a different interpretation of the common congestion signal, and is ultimately caused by DCTCP, not by the AQM.

And that is the case for the "shallow marking" switches you mentioned earlier.  Even though they were designed with DCTCP in mind, they are not designed to provide fairness between DCTCP (or a transport that resembles it, as TCP Prague does) and conventional TCP (or a transport that resembles it, such as TCP-SCE or QUIC).  So while a conventional TCP passing through such a switch my merely experience poor utilisation due to the shallow buffer, a conventional TCP crossing DCTCP traffic in that same switch may find itself completely starved.

It's crucial that everyone in this debate fully understands this point, because DCTCP-like behaviour underpins every aspect of L4S.

A consequence of that is that L4S expects the network to be upgraded, in full, to either understand its classification signal or implement FQ.  The *whole Internet*, from datacentre to CDN to core to backhaul to last-mile to third-world farmer just trying to check his e-mail.  That is not going to happen within any reasonable timeframe.

There are alternatives which recognise the existence of AQMs already deployed, and explicitly distinguish the high-fidelity and conventional congestion signals so that they aren't confused with each other upon receipt.  The SCE team has proposed one, and I now see that Jake Holland has proposed another.  I think both of these alternatives should be given serious consideration.

 - Jonathan Morton

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 9 May 2020 09:54:59 -0400

On Sat, May 9, 2020 at 5:42 AM Jonathan Morton <[email-masked]> wrote:
> And that is the case for the "shallow marking" switches you mentioned earlier.  ...
>  a conventional TCP crossing DCTCP traffic in that same switch may find
> itself completely starved.

I guess here "conventional TCP" is referring to a TCP sender using
RFC3168 ECN? The sites that have deployed shallow-marking ECN switches
know who they are, and know not to (and already ensure that they do
not) mix RFC3168 ECN and DCTCP/L4S-style shallow-threshold ECN senders
in the same queue. Part of my point is that because of such issues,
such sites will likely decide they can only deploy L4S, and not
RFC3168 or SCE.

> A consequence of that is that L4S expects the network to be upgraded, in
> full, to either understand its classification signal or implement FQ. The
> *whole Internet*, from datacentre to CDN to core to backhaul to last-mile to
> third-world farmer just trying to check his e-mail. That is not going to
> happen within any reasonable timeframe.

It is not the case that L4S expects "the whole Internet" to be
upgraded. AFAICT the L4S ecosystem is expecting something like the
following:

*If* you are one of the few ISPs that has deployed RFC3168 ECN
bottlenecks that do not have FQ or other mechanisms to ensure robust
fairness properties in the face of flows that mark ECT(0|1) and do not
follow RFC3168, *and* you are not comfortable with the L4S mechanisms
to infer the presence of RFC3168 bottlenecks, then you should either
(a) upgrade your RFC3168 ECN bottlenecks to something with more robust
fairness properties (e.g. fq_codel or cake), or (b) simply change your
existing DSCP rewriting rules at the ISP ingress to bleach ECT(1) to
Not-ECT (or any other mechanism of your chosing to treat ECT(1) as
Not-ECT).

neal

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 9 May 2020 19:17:43 +0200

Hi Neal,

> On May 9, 2020, at 15:54, Neal Cardwell <ncardwell=[email-masked]> wrote:
> 
>> [...]

	[SM] IMHO this expectation really is demanding that others clean up the mess L4S will have made, no?
Other AQMs that presently work well and standards-compliant will need to change to allow L4S to work outside of existing RFC3168 rules. And that for rather little gains L4S offers over the state of the art ... (fq_codel based AQM can deliver a median of around 5ms under load with bog-standard 1/sqrt(p)type TCP flows, sure 1ms < 5 ms, and what ever many nines L4S secifies is larger than the median, but we are still talking relative small differences compared to the multiple 100s of miliseconds without any AQM, but that later is the wrong reference point to judge the L4S costs against IMHO)

	Your proposed remedy, using FQ to isolate the fall-out of L4S flows lack of CE responsiveness to themselves, is exactly what L4S itself eschews on the argument that FQ is computationally too expensive. Now, I am not convinced that that is actually technically correct (and assume L4S FQ-allergy is mostly driven by Bob Briscoe's known position on FQ) but still that is a self-inconsistent position to take IMHO: to allow L4S to get by cheaply, existing users are required to up their game at a considerable computational cost. 

I am not saying that this is not a valid position to take, but the consequences should be explicitly discussed.

Also, as a nit, the 2 ECN bits live outside the 6-bit DSCP field, so DSCP rewriting rules might not be the right term?

Best Regards
	Sebastian

---

From: Luca Muscariello <[email-masked]>
Date: Sat, 9 May 2020 22:20:35 +0200

> Not-ECT (or any other mechanism of your chosing to treat ECT(1) as

The network operations you describe above assume that the issues
are just about the heuristics used by DCTCP in L4S to fallback
to ECN RFC3168 or the separation of DCTCP and TCP.

The behaviour of L4S at the bottleneck is actually
well defined by dualQ
https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-11
<https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-11>
So, the safety issues are more complex as I wrote
in my rationale that summarizes, I think, many other concerns
in the message below.

https://mailarchive.ietf.org/arch/msg/tsvwg/T3CLKTE6gRjV_6cBjTO7ALe4h7Q/

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 12 May 2020 17:01:50 +0200

Hi Jonathan,

A small nitpick:

 > In particular, to obtain the 50% reduction to a single CE mark that

The DCTCP sender has to believe that one window's worth of segments was
received with CE set; This is incidentally the backwards compatibility
mode of DCTCP against RFC3168 receivers, which will latch ECE until an
(acceptable) CWR has been received. (And based on the readily available
DCTCP / RFC3168 implementations, this is something that tends to happen
frequently).

You are, of course, correct in stating that shallow-threshold CE marking
is detremental to 1/sqrt(p) CC. But I can also support Neal's claim from
independent findings, that hardware is being deployed which only
supports 1/p marking in heterogeneous environments today (well, since 2
years ago).

Having an classifier which is more persistent across the internet than
DiffServ is IMHO key, if a consistent low-latency service is something
we strive for.

(And yes, L4S as it currently stands, may have issues to varying degrees
in certain scenarios. Nevertheless, I remain convinced, that those
problems can be resolved in a feasible way.

Richard

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 12 May 2020 19:08:47 +0300

> On 12 May, 2020, at 6:01 pm, Scheffenegger, Richard <[email-masked]> wrote:
> 

I'm perfectly aware of that, but I was referring to the case where DCTCP senders are matched with DCTCP receivers, as that is both the intended deployment scenario for DCTCP, and the one which exposes the problem.  In engineering it is necessary to consider the worst case, not merely the best case, and risk-assess the consequences of that worst case occurring in practice.

In fact, this particular backwards-compatibility mode has a hidden danger to it.  A sender may select DCTCP behaviour and see no detriment at first, because all the receivers observed behave conventionally and this backwards compatibility mode is in effect.  And a receiver may select DCTCP behaviour and see no detriment at first, because all the senders observed also behave conventionally.  But the behaviour then changes dramatically when the two happen to meet, and by this time both the sender's and receiver's operators have stopped paying attention.

That's why protocol standards exist to be followed.  I really shouldn't have to spell that out in the context of an IETF WG debate.

> hardware is being deployed which only
> supports 1/p marking in heterogeneous environments today

This is irrelevant, as I noted before.  An AQM does not enforce 1/p behaviour or otherwise.  When faced with mixed traffic types, both types of AQM will trigger the unfair behaviour, unless this is explicitly mitigated.

An AQM may be *optimised* for a particular traffic type, of course - in the case of 1/p behaviour on a short path, the optimisation is to mark at a shallower depth and a higher frequency than for 1/sqrt(p) behaviour on a longer path.  But if it is optimised for the wrong thing, then the AQM is badly deployed.  And, to state the obvious, currently all RFC-compliant traffic on the Internet is 1/sqrt(p) on a relatively long path.

 - Jonathan Morton

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 14 May 2020 17:03:56 +0000 (UTC)

Hi David, 
 you write "In contrast to reliance on TCP Prague, that combination of FQ and bleaching looks like it could provide a robust safety case." 

Some of those who have responded to the consensus call seem to assume  that a safety case should not necessitate any action on the part of third parties, eg operators who are running RFC3168 AQMs. That you reject bleaching not on that ground, but because it would nobble the use of ECT(1), makes me wonder if your position is more nuanced. 

Some responders are evidently more  relaxed about third parties needing to act in order to maintain effective  congestion control on the internet, as evidenced by their willingness to contemplate ECT(1) bleaching.

If action on the part of third parties is allowed to be necessary as part of a safety case, as long as that action is sufficiently easy, then it seems possible that an action could be found which does make L4S safe without making ECT(1) bleaching likely, or make it  limited in scope. For example, maybe there is a simple way for operators to signal to endpoints that RFC3168 AQMs are in use. So I was wondering what your position is, regarding third party actions being necessary  as part of a safety case - what limitations should there be, on  third party actions that a safety case may necessitate?

Alex

On Friday, May 8, 2020, 9:10:18 PM GMT+1, Black, David <[email-masked]> wrote: 

This the longer explanation of why I find myself in group 2 on the ECT(1) question because “If you believe L4S will not be safe for the internet without significant architectural changes, you are in this group.”

For the proposed Internet-wide L4S experiment, the technology to be tested is necessarily experimental, but the safety of the experiment needs to be fundamental to avoid serious damage if things go wrong.  I do not currently see a strong safety case for L4S+TCP Prague – an example of such a strong safety case can be found in QUIC congestion control, where the fact that it is based on NewReno congestion control provides strong assurance that QUIC congestion control is safe for the Internet.   In contrast, TCP Prague is still research (it’s great research and I applaud the people who have invested their time and effort in it for what has been achieved), but a safety case ought not to be based on not-yet-stable research, and I don’t think another 6 months of testing and bug fixing will leave TCP Prague as anything other than research (ruling out group 3 for me).  Further, the L4S low-latency service will likely be used by protocols other than TCP Prague, which may not receive the level of scrutiny and testing that has been applied to TCP Prague.  The safety case needs to come from elsewhere.

Thanks, --David

+1 (774) 350-9323           Mobile: +1 (978) 394-7754

[email-masked]

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 14 May 2020 21:54:23 +0100

Alex, David, all,

Firstly an appeal to everyone to stop talking about wiping the ECN field 
as if it is acceptable. Bleaching ECN is not a safety mechanism, except 
as a last resort. Treating ECT(1) /as if/ it is Not-ECT is far 
preferable to wiping it so it /is/ Not-ECT.

Consider an RFC3168 AQM in network A, and an L4S or RFC3168 ECN AQM in 
network B downstream. If network A wipes the ECN field, but the 
bottleneck is in network B, then the wiping destroyed the ECN service in 
network B, with no safety benefit in network A.

Now to your point, Alex, pls see inline...

> If action on the part of third parties is allowed to be necessary as part of a safety case, as long as that action is sufficiently easy, then it seems possible that an action could be found which d…

[BB] Regarding safety actions by nodes or networks not taking part in 
the ECT(1) experiment; I don't think the real question is whether a 
safety case is 'allowed' to include these. As long as a proposal clearly 
states which measures would need non-participant action, then we can all 
decide how much weight to give their proposal.

Some non-participating networks will be willing to actively take such 
measures. Others won't. Some might be unaware of the L4S experiment. 
Clearly a safety case is weaker if it requires action by 
non-participants. But that doesn't rule it out.

This whole ECT(1) question does not have a formula that gives a correct 
answer. Different people weigh different factors more strongly than 
others, and the current quest is to find out which factors weigh most 
heavily for most people.

For instance, if there is a simple way to detect any problems and 
configure RFC3168 AQMs to be safer, some people will find that to be an 
acceptable compromise if they believe that single-queue RFC3168 AQMs are 
likely to be rare, even though they would not accept such a compromise 
if these AQMs were widespread. Others see safety as an absolute matter. 
For them, an inherently safe signalling system is paramount, 
particularly if they are unsure whether single-queue RFC3168 AQMs are 
rare. Others don't see this as a safety issue anyway, because the 
unfairness disappears at low link rates.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Black, David" <[email-masked]>
Date: Fri, 15 May 2020 00:03:01 +0000

Bob,
[still posting as an individual, not as WG chair]

> preferable to wiping it so it /is/ Not-ECT.

I concur with that appeal ... but ...

... We've been through an analogous adventure with DSCP bleaching, as RFC 2474 recommends that behavior for DSCPs (treat unknown DSCPs as best effort without changing the DSCP) but the network operators went ahead and bleached DSCPs at network boundaries, as described in Section 1 of RFC 8100 (https://www.rfc-editor.org/rfc/rfc8100.html#section-1).  I strongly suggest reviewing that history towards understanding why things turned out that way.

> network B downstream. If network A wipes the ECN field, 

To be specific, my understanding of the wiping under discussion is to rewrite ECT(1) to Not-ECT, not zero out all ECN field values (e.g., rewriting CE to Not-ECT is very wrong and dangerous).  In particular, ECT(0) would not be wiped.

> but the

But there may be a safety benefit in network B.  Suppose a flow uses DCTCP and the RFC 3168 bottleneck in network B drops a Not-ECT packet that it would have been marked CE if network A had not wiped its ECT(1) mark to Not-ECT.  That drop triggers DCTCP's drop reaction, which is safe for an RFC3168 bottleneck, in contrast to DCTCP's CE-mark reaction, which is not.  Please refer to the prior discussion with Neal Cardwell on DCTCP's differing reactions to drops vs. CE-marks.

> non-participants. But that doesn't rule it out.

I generally agree.  Over in the IETF Security area, there's a long track record of dealing with this sort of situation, where a counter-measure may be vitally necessary in some cases, but not in others, via the use of "MUST implement, MAY use" requirements.  Such requirements ensure that the counter-measure will be available if a network operator discovers that it is needed.  Among the reasons that this approach makes sense here is that the measures under discussion may be needed by a network operator to counter denial-of-service attacks that take advantage of the L4S low-latency service.

Returning to Alex's original question:

> So I was wondering what your position is, regarding third party actions being necessary  as part of a safety case -
> what limitations should there be, on  third party actions that a safety case may necessitate?

At a minimum, the third parties MUST have the tools readily available to take any safety actions that may become necessary.   Analogy - a fire inspector is not interested in whether the building owner could buy a fire extinguisher at a nearby store - the fire extinguisher has to be installed and maintained so that it's readily available if needed.

Beyond that, I welcome discussion on how to reduce the occurrence of situations in which such actions become necessary.

Thanks, --David (as an individual)

> From: Bob Briscoe <[email-masked]>
> Sent: Thursday, May 14, 2020 4:54 PM
> To: [email-masked]; [email-masked]; Black, David
> >   you write "In contrast to reliance on TCP Prague, that combination of FQ and
> bleaching looks like it could provide a robust safety case."
> >
> > Some of those who have responded to the consensus call seem to
> assume  that a safety case should not necessitate any action on the part of third
> parties, eg operators who are running RFC3168 AQMs. That you reject
> bleaching not on that ground, but because it would nobble the use of ECT(1),
> makes me wonder if your position is more nuanced.
> >
> > Some responders are evidently more  relaxed about third parties needing to
> act in order to maintain effective  congestion control on the internet, as
> evidenced by their willingness to contemplate ECT(1) bleaching.
> >
> > If action on the part of third parties is allowed to be necessary as part of a
> safety case, as long as that action is sufficiently easy, then it seems possible
> that an action could be found which does make L4S safe without making ECT(1)
> bleaching likely, or make it  limited in scope. For example, maybe there is a
> simple way for operators to signal to endpoints that RFC3168 AQMs are in use.
> So I was wondering what your position is, regarding third party actions being
> necessary  as part of a safety case - what limitations should there be, on  third
> party actions that a safety case may necessitate?
> 
> > This the longer explanation of why I find myself in group 2 on the ECT(1)
> question because “If you believe L4S will not be safe for the internet without
> significant architectural changes, you are in this group.”
> >
> >
> >
> > Current practice is not to mix DCTCP-like traffic (1/p-class congestion control)
> with TCP-like traffic (1/sqrt(p)-class congestion control, e.g., NewReno) in the
> same queue because the DCTCP-like traffic outcompetes the TCP-like traffic to
> the point of starvation of the latter.  By using ECT(1) as the only classifier for
> low-latency traffic, L4S causes mixing of those two classes of traffic at network
> nodes that have not been modified to use ECT(1) as a classifier.  Hence the L4S
> experiment proponents are responsible for the safety of that traffic mixing at
> unmodified nodes.  I understand the L4S approach to achieving safety at
> unmodified nodes to be use of TCP Prague, specifically its heuristics that detect
> RFC 3168 bottlenecks.  As previously discussed, this traffic mixing does not
> cause problems at otherwise unmodified nodes that use FQ.
> >
> >
> >
> > For the proposed Internet-wide L4S experiment, the technology to be tested is
> necessarily experimental, but the safety of the experiment needs to be
> fundamental to avoid serious damage if things go wrong.  I do not currently see
> a strong safety case for L4S+TCP Prague – an example of such a strong safety
> case can be found in QUIC congestion control, where the fact that it is based on
> NewReno congestion control provides strong assurance that QUIC congestion
> control is safe for the Internet.   In contrast, TCP Prague is still research (it’s
> great research and I applaud the people who have invested their time and
> effort in it for what has been achieved), but a safety case ought not to be based
> on not-yet-stable research, and I don’t think another 6 months of testing and
> bug fixing will leave TCP Prague as anything other than research (ruling out
> group 3 for me).  Further, the L4S low-latency service will likely be used by
> protocols other than TCP Prague, which may not receive the level of scrutiny
> and testing that has been applied to TCP Prague.  The safety case needs to
> come from elsewhere.
> >
> >
> >
> > If ECT(1)-marked traffic starts causing problems, network operators are not
> going to wait for TCP Prague or other protocol implementations to be patched
> – they’re going to take action, e.g., as Neal Cardwell describes (and I’ve heard
> about this sort of potential consequence in other conversations, so please don’t
> blame Neal for this):
> >
> >
> >
> >> - L4S flows potentially causing unfairness in RFC3168 ECN bottlenecks has
> been mentioned as a potential concern. However, a robust RFC3168 ECN
> bottleneck
> >> should already have a mechanism to avoid unfairness caused by flows that
> are marked as ECT(0|1) and yet not performing RFC3168 responses. In
> particular,
> >> many of the large sources of known deployments of  RFC3168 --  Linux
> fq_codel and cake -- are already deployed with fair queueing. In such
> bottlenecks L4S
> >> traffic should not cause harm to other non-L4S flows. Furthermore, if there
> really are ISPs with deployments of RFC3168 bottlenecks that have neither FQ
> nor
> >> any other protection from non-RFC3168-ECT(1) flows, then they can bleach
> incoming ECT(1) code points to Not-ECT and treat L4S as Not-ECT (ISPs typically
> >> already transform the DSCP byte at their ingress anyway). So I do not see
> harm to RFC3168 ECN bottlenecks as a prohibitive concern.
> >
> >
> looks like it could provide a robust safety case.  It would be most unfortunate if
> things wind up depending on this, especially bleaching, as the fact that ECT(1) is
> not bleached at network boundaries was a major reason for selection of ECT(1)
> as the identifier for low-latency L4S traffic.  Among the implications is that if
> the L4S experiment fails, that bleaching could be around much longer, making
> it infeasible to use ECT(1) for anything else Internet-wide.  In my view, it is the
> responsibility of the L4S proponents to design the L4S  technology and structure
> the experiment in a fashion that makes ECT(1) bleaching unlikely and/or likely
> to be limited in scope if it happens.
> >
> >
> >
> > Thanks, --David
> >

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 15 May 2020 10:59:21 +0100

See in-line

On 15/05/2020 01:03, Black, David wrote:
>> as a last resort. Treating ECT(1)/as if/  it is Not-ECT is far
>> preferable to wiping it so it/is/  Not-ECT.
Well, some operators have chosen a model that does that, I can't resist 
self-citing some research that shows this might not be the full story 
..."Exploring DSCP modification pathologies in the Internet" (2018).

https://www.sciencedirect.com/science/article/pii/S0140366417312835

The reasons for field bleaching seem to vary, (and, alas, a significant 
proportion still seems to be due to configured ToS semantics).

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 15 May 2020 13:26:23 +0200

Hi Gory,

> On May 15, 2020, at 11:59, Gorry Fairhurst <[email-masked]> wrote:
> 
> See in-line
> 
>>> as a last resort. Treating ECT(1) 
>>> /as if/
>>>  it is Not-ECT is far
>>> preferable to wiping it so it 
>>> /is/
>>>  Not-ECT.
>>> 
>> .... We've been through an analogous adventure with DSCP bleaching, as RFC 2474 recommends that behavior for DSCPs (treat unknown DSCPs as best effort without changing the DSCP) but the network operators went ahead and bleached DSCPs at network boundaries, as described in Section 1 of RFC 8100 (
>> https://www.rfc-editor.org/rfc/rfc8100.html#section-1
>> ).  I strongly suggest reviewing that history towards understanding why things turned out that way.
>> 
>> 
> Well, some operators have chosen a model that does that, I can't resist self-citing some research that shows this might not be the full story ... "Exploring DSCP modification pathologies in the Internet" (2018).
> 

	[SM] Well, my take on that is that there is a 80% or higher chance (transparent plus Reset ToS prec. plus Reset ToS prec. CS6/CS7)  for DSCPs with 3 leading zero bits to survive internet traversal. That seems like a viable starting position for a LL classifier IMHO. In reality L4S will only work well over short paths anyway (mostly CDN -> end-user) and there the number of Diffserve domain crossings should be on the lower side and SLAs might be an option to make an L4S-dscp actually survive well.

	As the bleaching ECT(1) argument demonstrates any marker on a packet will be considered fair game to manipulation if that still allows delivery of the packets and fiddling it gives advantages to either senders or to intermediate hops, so the superiority of ECT(1) in internet traversal really is just a snapshot due to the circumstances that nobody got sufficiently annoyed by ECT(1) yet. IMHO if L4S works and is really worth its salt (meets its promises), it should be able to pull its own weight, by making networks operators take care to not bleach its reference DSCP, instead of forcing ECT(1) on everybody. 

> 
> The reasons for field bleaching seem to vary, (and, alas, a significant proportion still seems to be due to configured ToS semantics).

	[SM] Clearly one of the reasons is that so far there is absolutely zero value to conserve an unhandled DSCP and treat it like BE for the operator in question. According to the L4S RFCs however, everybody is going to profit from L4S everywhere, so surely those operator straggling today to leave DSCPs alone will now do so for the L4S-dscp since that will give them better network performance (even if not participating in the L4S experiment, they could optimistically expect less bursty traffic patterns, allowing to drive the gear at higher average load). Either that or that "everybody is going to profit" claim is being falsified, either way an interesting result.

Best Results
	Sebastian

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 15 May 2020 13:20:47 +0100

Hi,

On 15/05/2020 12:26, Sebastian Moeller wrote:
> Hi Gorry,

> 	As the bleaching ECT(1) argument demonstrates any marker on a packet will be considered fair game to manipulation if that still allows delivery of the packets
Not really the IETF recommendation.
>   and fiddling it gives advantages to either senders or to intermediate hops, so the superiority of ECT(1) in internet traversal really is just a snapshot due to the circumstances that nobody got sufficiently annoyed by ECT(1) yet. IMHO if L4S works and is really worth its salt (meets its promises), it should be able to pull its own weight, by making networks operators take care to not bleach its reference DSCP, instead of forcing ECT(1) on everybody.
Equipment still configured for ToS Semantics is hard to change. 
Although, useful to do so.
> 	[SM] Clearly one of the reasons is that so far there is absolutely zero value to conserve an unhandled DSCP
DSCPs might be utilised by other network segments and hence could add 
value to the traffic.
> and treat it like BE for the operator in question.
Perhaps - because mapping to a PHB is different to changing the DSCP.
> According to the L4S RFCs however, everybody is going to profit from L4S everywhere, so surely those operator straggling today to leave DSCPs alone will now do so for the L4S-dscp since that will give them better network performance (even if not participating in the L4S experiment, they could optimistically expect less bursty traffic patterns, allowing to drive the gear at higher average load). Either that or that "everybody is going to profit" claim is being falsified, either way an interesting result.
>
> Best Results
> 	Sebastian
>
>
>> Gorry

-- 
G. Fairhurst, School of Engineering

I'm not sure how a transport would ensure that the bottleneck links are using a DSCP in their classifier, and how you would operators would police such a scheme
- excepet perhaps to confine L4S as an ISP experiment for their own customer base (which they can already do using a private DSCP).
This just involves more components in the deployment. I thought the WG has gone
beyond the idea of doing an experiment within an ISP to proposing a new architecture, so I really don't understand how
the use of a DSCP would help deploy that experiment.

Gorry.

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 15 May 2020 17:53:01 +0200

Hi Gory,

> On May 15, 2020, at 14:20, Gorry Fairhurst <[email-masked]> wrote:
> 
> Hi,
> 

	[SM] That was not my claim, I understand that the RFCs and reality diverged, but it is an argument that Tom Herbert made in the encrypted transport header discussion that seems appropriate here. Both endpoints, and network elements will fiddle with ECT(1) if that conveys an advantage, independent of whether it is recommended or not.

> Equipment still configured for ToS Semantics is hard to change. Although, useful to do so.

	[SM] Well, that equipment, according to the cited paper, will also (with low probability) bleach the ECN field (interpreting it as TOS) so that issue already exists for L4S with ECT(1), no?

> DSCPs might be utilised by other network segments and hence could add value to the traffic.

	[SM] Yes theoretically true, but as reality seems to demonstrate not a strong enough driver for bleachers/remappers to change their way. A L4S DSCP/PHB that directly implies desirable traffic properties to each network node (less burstyness or more re-ordering tolerance) might change that value equation.

> Perhaps - because mapping to a PHB is different to changing the DSCP.

	[SM] I am not fully understanding this argument. Sure these are different things, but the recommendation, remap locally unused DSCPs to the BE PHB, seems not to have gotten too much traction according to the data. Unless we assume that each network only ever re-mapps those input DSCPs that it uses for its own internal purposes, and that by pure chance DSCPs in the decimal ~8-48 range (see Figure 2) all get remapped, because networks either use a lot of dscps internally or different networks use mostly not overlapping sets of DSCPs. Honestly, I would bet on networks simply re-mapping everthing to zero and then upmark those paket with their internal DSCPs. It is always safer to re-map to zero and only elevate packets with known properties...

Best Regards
	Sebastian

P.S.: I am really not seeing how your paper does support the case for ECT(1) for L4S at all. For the one critical segment, mobile, I see quite enough involvement of @mobile-corp email addresses here, that getting the mobile segment honor a L4S-DSCP seems in no way insurmountable, IFF they are actually as interested in HFCC.

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 16 May 2020 00:12:52 +0100

Sebastian,

On 15/05/2020 12:26, Sebastian Moeller wrote:
> 	[SM] Well, my take on that is that there is a 80% or higher chance (transparent plus Reset ToS prec. plus Reset ToS prec. CS6/CS7)  for DSCPs with 3 leading zero bits to survive internet traversal. That seems like a viable starting position for a LL classifier IMHO.

[BB]
I don't think you've read the paper closely enough. The measurements you 
quote did not cross any access network. They were  from a couple of 
dozen vantage points in Digital Ocean data centres to large numbers of 
popular web server destinations. That doesn't measure what typical 
access networks do to the DSCP.

The paper does include measurements from 107 vantage points connected to 
Internet access networks, but solely mobile. Here, for any DSCP except 
best efforts, the % unchanged is never better than 38%.

For how fixed access networks mangle Diffserv, you'd need to measure 
from vantage points on something like PlanetLab. Reference [31] measured 
from both MONROE and PlanetLab vantage points to characterize mobile and 
fixed access, but that was looking more at the effect on ECN than 
Diffserv. We did take some Diffserv measurements to look for correlation 
with ECN wiping. But from memory I think we only added the Diffserv test 
over mobile, 'cos that's where nearly all the ECN wiping was. I'll take 
a look at the data when I have some time (ha ha!)

> In reality L4S will only work well over short paths anyway (mostly CDN -> end-user)

[BB] Is this a fact, or an alternative fact?

> and there the number of Diffserve domain crossings should be on the lower side and SLAs might be an option to make an L4S-dscp actually survive well.
>
> 	As the bleaching ECT(1) argument demonstrates any marker on a packet will be considered fair game to manipulation if that still allows delivery of the packets and fiddling it gives advantages to either senders or to intermediate hops, so the superiority of ECT(1) in internet traversal really is just a snapshot due to the circumstances that nobody got sufficiently annoyed by ECT(1) yet.
>

[BB] The bleaching ECT(1) argument doesn't /demonstrate/ anything. It's 
your conjecture.

Currently, for fixed networks:
* according to our measurements (ref [31] in Gorry's study), from 54 
PlanetLab vantage points on 25 networks in 22 countries to the Alexa top 
500k web servers, consistent with previous studies, the ECN codepoint is 
wiped on 0.23% of paths.
* in contrast, according to Gorry's study measured at a similar time, 
from a couple of dozen vantage points in data centres, there is some 
form of DSCP pathology on 70% of paths.
* Gorry's study also found a high correlation between clearing the DSCP 
and clearing the ECN field, which supports the hypothesis that a 
well-documented bug reported on more than one brand of equipment is 
still prevalent, where the ECN field is cleared as an unintentional 
side-effect of clearing the DSCP.

For mobile:
* Our study from vantage points in 18 mobile networks found 10.5 always 
wiped the ECN codepoint, and the others never did.
* Gorry's study from 107 vantage points, in an unstated number of mobile 
networks, found only between about 5% and 25% of paths preserved the DSCP.

When we put a huge amount of effort into measurement studies, it's 
because data is important. When the data contradicts what you want to 
believe, you are welcome to say that the data might change in the 
future. And you can make up reasons why this might happen. But I'm not 
going to make the same points against your arguments again because, when 
you've said the same made-up things once, and twice and three times and 
four times, it achieves nothing other than demonstrating that you like 
the sound of your own keyboard.

> IMHO if L4S works and is really worth its salt (meets its promises), it should be able to pull its own weight, by making networks operators take care to not bleach its reference DSCP, instead of forcing ECT(1) on everybody.

[BB] There is no reference DSCP for L4S. ECT(1) already inherently 
identifies L4S packets. If a DSCP were used as well, networks supporting 
L4S would ignore it, given it's redundant anyway, and it might sometimes 
be bleached.

It is poor practice to use 2 identifiers for one feature, because both 
then have to be checked and cases where they are inconsistent have to be 
handled, potentially opening security vulnerabilities.

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 16 May 2020 16:30:57 +0200

Hi Bob,

more below in-line, prefixed with [SM]. 

> On May 16, 2020, at 01:12, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> I don't think you've read the paper closely enough. The measurements you quote did not cross any access network. They were  from a couple of dozen vantage points in Digital Ocean data centres to large numbers of popular web server destinations. That doesn't measure what typical access networks do to the DSCP.

	[SM] Oh, sure, But for L4S to go anywhere the edge ISPs need to be on-board anyway, so that final level of potential bleaching, is IMHO less of a hard obstacle and more of a test whether there is actually demand for L4S' promises and deliverables. That applies both to the fixed-line and the mobile edge. 

> 
> The paper does include measurements from 107 vantage points connected to Internet access networks, but solely mobile. Here, for any DSCP except best efforts, the % unchanged is never better than 38%.

	[SM] Yes, but either the carriers are interested in L4S ad then passing a single DSCP will not be an unsurmountable problem, or the market will have spoken about L4S.

> 
> For how fixed access networks mangle Diffserv, you'd need to measure from vantage points on something like PlanetLab. Reference [31] measured from both MONROE and PlanetLab vantage points to characterize mobile and fixed access, but that was looking more at the effect on ECN than Diffserv. We did take some Diffserv measurements to look for correlation with ECN wiping. But from memory I think we only added the Diffserv test over mobile, 'cos that's where nearly all the ECN wiping was. I'll take a look at the data when I have some time (ha ha!)

	[SM] That would surely be interesting in itself, but I believe orthogonal to the question whether DSCPs are a viable way to signal intent end-2-end. IMHO that will work if the intermediate networks can also profit, and according to L4S promises they should. (I am less optimistic about these advantages offered by L4S, but I am happy if I should be proven wrong about this by reality).

> [BB] Is this a fact, or an alternative fact?

	[SM] Future tense should have made clear that this is a prediction, sorry, but as a non-native speaker of english I sometimes miss subtle points and assumed no "In reality I predict, L4S will...". 
	I base this prediction on ancient and recent rounds of testing from both team L4S and team SCE. The amount of realistic scenarios where L4S falls way short of how reasonable minds will read its promises, and the lack of sufficiently adversarial testing from team L4S, together with the hopes and wishes that ECT(1) will not be abused (instead of hard requirements and/or analysis). 
	I still think that on reasonable short paths things might work reasonably well most of the time though, and hence conceded positive uitility for L4S for short paths. I note that when I asked about bi-directional saturating tests, you did not deliver data, but just accepted that L4S will do badly under those circumstances.

> [BB] The bleaching ECT(1) argument doesn't /demonstrate/ anything. It's your conjecture.

	[SM] Erm, sorry, I did not introduce ECT(1)-bleaching, it were other's supportive of L4S that brought up that in case of failure bleaching ECT(1) might be an option. All I note above is that if bleaching ECT(1) gives an intermediate node an advantage it will be done, no amount of words in the L4S RFCs to the opposite. I also predict L4S to fail at least of non-short paths and hence assume ECT(1) bleaching to become a reality. Call it conjecture, but your hypothesis that ECT(1) will not be bleached is based on the same incomplete information and hence also just a conjecture.

> * according to our measurements (ref [31] in Gorry's study), from 54 PlanetLab vantage points on 25 networks in 22 countries to the Alexa top 500k web servers, consistent with previous studies, the ECN codepoint is wiped on 0.23% of paths.
> * in contrast, according to Gorry's study measured at a similar time, from a couple of dozen vantage points in data centres, there is some form of DSCP pathology on 70% of paths.

	[SM] Yes, but proper reading of that data indicates that by thoughtfully selecting a DSC with the 3 TOS precedence bits set to zero, significantly increases that percentage to above 80%. I did mention that explicitly, so let's stick to my numbers please.

> * Gorry's study also found a high correlation between clearing the DSCP and clearing the ECN field, which supports the hypothesis that a well-documented bug reported on more than one brand of equipment is still prevalent, where the ECN field is cleared as an unintentional side-effect of clearing the DSCP.

	[SM] Sure, but that does not change the fact the ECT(1) is not qualitatively different in that regard.

> 
> For mobile:
> * Our study from vantage points in 18 mobile networks found 10.5 always wiped the ECN codepoint, and the others never did.
> * Gorry's study from 107 vantage points, in an unstated number of mobile networks, found only between about 5% and 25% of paths preserved the DSCP.

	[SM] Yes that looks less attractive, but with mobile carriers interested in L4S' offerings, that should be relatively easy to change, given the size and range of the big mobile carriers action of a few players will "unlock" L4S for a considerable number of users, even if other ISPs struggle behind.

> 
> When we put a huge amount of effort into measurement studies, it's because data is important. When the data contradicts what you want to believe, you are welcome to say that the data might change in the future.

	[SM] I conceptually agree, except I offered a pretty close to the numbers reading of the paper. So let's stick to objective arguments, please. 

> And you can make up reasons why this might happen. But I'm not going to make the same points against your arguments again because, when you've said the same made-up things once, and twice and three times and four times, it achieves nothing other than demonstrating that you like the sound of your own keyboard.

	@chairs: I wonder whether this is the level of discourse to aim for on this list?

> [BB] There is no reference DSCP for L4S.

	[SM] I apologize for being to terse here, I am talking about a hypothetical DSCP L4S could use as signal from the end-points to request 1/p-congestion signaling. I assumed that everybody participating in this discussion is sufficiently fluent about the L4S drafts to understand that. Your argument indicates that I was wrong, so thanks for giving me an opportunity to clarify this.

> ECT(1) already inherently identifies L4S packets.

	[SM] That is the plan, but we have gone though this ad nauseam, so please tell me L4S sees a CE marked packet, what kind of flow does this come from? I do not ask whether that is important to know, I ask how would an L4S node deduce this from a non-existent identifier.

> If a DSCP were used as well, networks supporting L4S would ignore it, given it's redundant anyway, and it might sometimes be bleached.

	[SM] Well, you probably missed my position, I am of the opinion that ECT(1) should not be used as input, and hence that situation would not really happen. I have given my rationale in the past, so will not repeat that here (but it is related to my prediction above).

> 
> It is poor practice to use 2 identifiers for one feature, because both then have to be checked and cases where they are inconsistent have to be handled, potentially opening security vulnerabilities.

	[SM] I am with you on this Bob, we just differ in which "identifier" we would jettison respectively.

Best Regards	

Sebastian