Subject: [tsvwg] Follow-up to your DSCP and ECN codepoint comments at tsvwg interim

From: Bob Briscoe <[email-masked]>
Date: Sat, 7 Mar 2020 00:34:22 +0000

Steve,

I had audio problems during the chairs slides at the tsvwg interim.
https://datatracker.ietf.org/meeting/interim-2020-tsvwg-01/session/tsvwg
Now I see the following two comments of yours in the minutes, I'd like 
to follow-up:

> - Steve Blake mentioned that L4S could be made compatible with each.
Can you say more about what's behind this idea? or point me to where 
you've already said it?

> - Steve Blake noted anyone running these experiments will tinker with 
> boxes, so may handle the DSCP bleaching concern.
Unfortunately, that's only partially true. It's fairly true for 
CDN->user applications. However, for (say) conversational video or 
online gaming, AR, VR, etc, which are particularly important targets 
apps, the path is often either accessISP-accessISP or 
accessISP-cloud-accessISP. Altho most access ISPs nowadays are pretty 
tightly meshed with other access ISPs (with less intermediaries than 
there used to be), there are still significant numbers of cases where 
two access ISPs are not directly connected.

Anyway, if user B's downstream is the bottleneck for user A, even if ISP 
B doesn't bleach Diffserv, the DSCP is still no use if ISP A bleaches, 
or if an intermediary does.

See 
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-09#appendix-B.2 
for write-up of the pros and cons.

Network traversal is already a difficult problem, even with just a very 
few ECN problems. We really don't want to make this impossible. With a 
3-part (sender-bottleneck-receiver) L4S deployment, there is already 
enough deployment barrier without adding Diffserv traversal to make 4 or 
more parts (sender-border-bottleneck-receiver). For SCE, there's already 
4-parts to deploy (tunnels added), so let's not countenance making that 
5 or more.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 7 Mar 2020 20:16:40 +0100

Dear All.

I fully understand the challenges of using DSCPs end to end, and yet DSCPs still seem to be the IETF's endorsed intra-domain and end-to-end marking mechanism (see RFC 2474). Any other marking use-case (like the LE PHB) could lay claim on ECT(1) by the same rationale as L4S as it would be currently more reliable. Sure, L4S could claim that it also uses functionally uses ECN and hence has a slightly stronger claim, but since the L4S arch draft explicitly mentions allowing non-responsive bounded-rate (and hence essentially non ECN-aware) flows in the LL queue it seems clear that ECT(1) in addition to all it's other issues is not really fulfilling L4S's marking needs exhaustively.

That said, the only real uses L4S is likely going to see any time soon, IMHO, are those that have an immediate monetary advantage for the ISP rolling out and operating the L4S infrastructure, and hence SLAs between content suppliers and ISP that make sure to conserve the L4S DSCPs seem like a natural solution to the issue. 
	In the spirit of walking before running it would IMHO behoove the L4S effort not trying to solve all potential problems a priori, but rather experimentally demonstrate that even a restricted scope L4S deployment has actual merits in real life. 

More below in-line.

> or if an intermediary does.

	[SM] That really just would show that there is a business case for ISP A and B to connect directly in a DSCP conserving fashion....

> 4-parts to deploy (tunnels added), so let's not countenance making that

	[SM] As mentioned above hat same rationale affects all PHBs that desire end to end persistence. What makes L4S so much more deserving than other PHB's so that the IETF should grant it special preferential treatment? 

	It seems worth mentioning, that today bidirectional FQ on an end user's access link run on the CPE, does already achieve median inter-flow latency-under-load-increases of ~ 5ms, so the incremental gain for switching to L4S will be to get that latency increase down to ~1 ms for LL-flows, while increasing it to 15ms for non-LL flows. These are pretty moderate gains for considerable costs and side-effects! 
	I would love to learn which novel use-cases will be possible by shaving off these additional 4 ms of queueing related delay that will not work with a 5ms target delay? And wonder if any of those that apparently require hard real-time guarantees in the handful of milliseconds range from the network should be run over the shared internet to beginn with independent of the number of parts between the endpoints....?

IMHO, that is orthogonal to the question of how to make 1/p-cc-response type flows equitably share the internet with 1/sqrt(p) type flows. Conflating this by overloading the "I desire LL queueing" marker with the "I will respond in a 1/p-type fashion to CE markings" is really just creating additional unnecessary "coupling" with considerable side-effects. 
As the SCE proposal demonstrates, 1/p and 1/sqrt(p) signaling can concurrently equitably coexist (currently with a few restrictions on propagating signals when decapsulating tunnels), without needing to change the semantics of what CE means, so all this is left to do is solve the orthogonal problem of steering flows into "low-latency" or "besteffort-latency" queues. As far as I understand the IETF's solution for that is DSCP/PHB. 
And for a realistic scope* of an experimental deployment of L4S requiring at least to pair L4S's ECT(1) marking with a DSCP (for the experiment eg out of the "xxxx11" experimental / local use range) does not seem onerous but rather prudent and cautious.

Best Regards
	Sebastian

*) L4S is not going to work well over arbitrary internet paths, period. It does have a decent chance over well curated relativ "short" (measured in hops and/or ASs) path, but over these DSCPs can be made sufficiently reliable already.

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 8 Mar 2020 03:35:12 +0200

> On 7 Mar, 2020, at 9:16 pm, Sebastian Moeller <[email-masked]> wrote:
> 

I broadly agree with this argument.  And, as noted further down, the relatively small improvements in latency will only be significant on paths where the baseline RTT is already quite small, implying both a short geographic distance and a small number of routing hops between the endpoints.  These are exactly the conditions most likely to be able to deploy a Diffserv solution through the normal SLA process.

The challenge with L4S is that it relies on the classifier not only for low-latency service (a performance improvement, whose loss would typically be tolerable) but to configure the type of congestion signalling (a safety requirement, whose loss would be dangerous).  This is, I think, why they sought a classifier which would reliably traverse AS boundaries.  This design requirement ultimately stems from their early design decision to perpetuate the DCTCP signalling method, which changes the semantics of CE.

Unfortunately this is not the only requirement exposed by that decision.  Not only must the classifier signal *survive* throughout the path, but every potential bottleneck node must *understand* the classifier and apply the different congestion signalling.  This puts L4S back in the position of having to limit deployment to controlled environments, only now without the Diffserv "containment" mechanism to help indicate when it strays outside of such an environment.  This in turn leads to the present attempt to work around the problem with "classic detection" algorithms, which I believe will show significant detection latency and will therefore be unable to switch modes as seamlessly as SCE has already demonstrated.

With SCE we have approached the problem from a different direction from the start: Safety First.  The spare ECN codepoint is used as an additional output from the network, and CE retains its existing meaning and usage.  This means SCE can be deployed across any existing network with congestion safety, competing normally with conventional traffic - because on a conventional network an SCE flow is almost indistinguishable from a conventional flow.  The performance benefits begin to appear, seamlessly, when the bottleneck on the path is SCE enabled.

And this means that if a Diffserv codepoint is used to request low-latency service for an SCE flow, it can be given by simply adjusting AQM parameters for that traffic, *without* impairing safety should this classifier be lost en route, and with completely equivalent performance.  I think this is a very important point that the L4S team has overlooked, in their continual attempts to discredit the SCE project.

We are working on a demonstration of this capability, which we may be able to show at Vancouver (remotely), perhaps via the Hackathon.  Of course the fully-integrated code will also be publicly available at the time of demonstration, so you can replicate and play with the results.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 8 Mar 2020 14:18:17 +0000

Steve,

Exec summary:
*If you hobble an experiment too much unnecessarily, you make it fail.*

See inline for the details. And the details *matter*...

On 07/03/2020 03:49, Steven Blake wrote:
> Hi Bob. Long time, no see ...
>
> On Sat, 2020-03-07 at 00:34 +0000, Bob Briscoe wrote:
>> Steve,
> L4S could use:
> - EXP DSCP to select low-latency queue
> - Optional: EXP DSCP to select the coupled queue (for specific non-TCP
>    Prague traffic being used as part of the experiment)
> - SCE-style ECN signaling (ECT(1) and/or CE)

That seems possible,... until you realize the details of codepoint 
traversal problems round the feedback loop (below). Then the flakiness 
of codepoint traversal disappoints so many potential users that the 
whole experiment gets a rubbish reputation, even if the performance when 
it does work is cool.

Even if we'd taken that approach 5 years ago, it still wouldn't be 
working today, because of the problem below.
The approach we chose worked immediately 5 years ago, and it still works 
today. We knew that required strong IETF blessing. We got that at the 
time. OK, now the WG has decided to check that decision. But, please, 
after that, can everyone get behind whatever decision is made.

  * SCE-style signalling doesn't propagate out of (hardly) any layerings
    on the current Internet (tunnels, lower layers).
      o The original ECN tunnelling specs didn't consider ECT1 as a
        lower strength CE. So if outer/inner is ECT1/ECT0, then RFC3168
        and IPSec decapsulators will just strip the outer ECT1.
      o The drafts that are extending RFC6040 decapsulation behaviour to
        all the other layering protocols aren't even through WGLC yet
        (ecn-encap-guidelines and rfc6040update-shim). And even for
        IP-in-IP tunnels, a large proportion don't comply with rfc6040 yet.
      o Tunnels / pseudowires / layerings / VPNs / etc are often 'over
        the top' of the network operator wanting to run the experiment.
  * SCE-style signalling would still need a transport layer feedback
    protocol that gave hi-fidelity ECT1 feedback.
      o But the approach proposed for SCE TCP feedback (individual draft
        in tcpm) would be randomly decimated by all the ACK thinning in
        the real Internet (WiFi, DOCSIS, LTE, etc).
      o Using QUIC is an alternative, but we have to support TCP as
        well. All the existing apps we have demonstrated "just worked"
        when we updated the TCP stack under them. To use QUIC, all these
        existing TCP apps would have to be updated to use QUIC. That
        would be a huge extra barrier.
      o TCPM has allocated 3 bits in the main TCP header for hi-fidelity
        CE feedback 'cos those bits traverse networks fairly reliably.
        The chances of TCPM allocating these flags for ECT1 instead of
        CE, or allocating more reserved flags for an ECT1 feedback
        experiment are nil, I'd have thought
      o So, realistically, to get hi-fi ECT1 feedback, you'd need a TCP
        Option. But all the big end-system developers are not planning
        to implement the TCP Option part of AccECN, to avoid middlebox-hell.

And why would Google/Apple/Microsoft etc want to go though 
middlebox-hell for an SCE-ECT1 experiment that isn't going to traverse 
most networks anyway?
And why would tunnel developers ensure SCE-ECT1 traverses their tunnels 
for an experiment that none of the big end-system companies have bought 
into?

So SCE-style signalling would work for some people sometimes, and 
randomly fail for most people.

Steve, please, I'm not stupid. We've thought of all these things. We 
thought of them years ago.
This is why I referred you to the pros and cons in the draft.

Then, *even if* your proposed way of deploying the experiment succeeded, 
how would it migrate to full Internet deployment? If it continued to use 
SCE-ECT1, it would not be classifiable into separate queues without 
DSCPs as well.

Otherwise it would only work for FQ, precluding it from high speed kit 
(and from lower layer AQMs from where port numbers are inaccessible).

If you hobble an experiment too much unnecessarily, you make it fail.

more...
> You are proposing to do an experiment. The experiment requires
> cooperation between one or more operators that have to deploy and
> configure equipment that frankly doesn't exist yet (at least not for
> real core link speeds).
Later you complain that equipment vendors have pre-empted the IETF and 
already built this into silicon, presumably based on the L4S status 
report at the last IETF:
https://datatracker.ietf.org/meeting/106/materials/slides-106-tsvwg-sessa-72-l4s-drafts-00#page=5
You cannot also complain that implementations don't exist yet.

Most of the interest is for access networks, where most bottlenecks are. 
The DualQ is designed to scale to core speeds. But that would usually be 
pretty pointless, given bottlenecks only move to the core very 
occasionally, by good network design.

Way more operators have plans for L4S DualQ deployment (even before the 
RFCs are baked) than have ever enabled Classic ECN in the last 2 
decades. Surely you can work out that the implementations are happening 
because operators are showing commercial interest to those implementers.

(See later about whether to view this as pre-empting the IETF, or taking 
a risk while the IETF deliberates.)

> Modifying current operational practice
> regarding configuration/use of DSCPs for traffic intended to
> participate in the experiment is a minor inconvenience compared to the
> other operational changes that are required to perform the experiment.
Yes. It's the SCE part of your proposal that's much more problematic 
than the DSCP part. But the DSCP part is also just an unnecessary 
encumberance.

What's the problem with the proposed L4S approach? In a nut-shell:

  * ECT1 appears not to be currently used on the Internet.
  * The experimental traffic has to be ECN-capable.
  * So using ECT1 as the identifier of the experimental traffic is
    sufficient.
  * It works. And it is identifiable - it can be ignored/blocked/stopped
    if necessary. But there's no evidence that will be necessary.

Why cause a load of extra deployment grief by requiring another 
identifier (DSCP) as well.
And why switch to using SCE-style, which makes it not work over real 
networks, with real TCP?

> I am not even slightly convinced that we understand the best approach
> to use for AQM on >= 100 GE links with tens of thousands of active
> flows (I'm a signal processing guy; it will probably require FFTs.) :)
> In fact, and you should appreciate this, if the goal is to minimize
> queueing delay and delay variation, IMHO we should revisit some of the
> PCN ideas for detecting pre-congestion on a node, even for slower links
> (think AVQ-style AQM).
Surely you don't think I hadn't thought of this? :)
That is the longer-term plan.
https://mentor.ieee.org/802.1/dcn/18/1-18-0073-00-ICne-l4s-ultra-low-queuing-delay-for-all.pdf#page=19

Virtual queue is already possible in the DOCSIS implementations - by 
operator config - future proofed. The operator just has to configure the 
max sustained rate of the user's aggregate service flow (ie. the rate 
for both L&C queues) a little lower than the bandwidth of the channel.

> Having the ability to signal multiple levels of
> "please  decelerate" is very intriguing. I think you believe that the
> transit operators are so rarely congested that you don't have to care
> (and you don't want to have to coordinate with them), but I believe
> that this should be demonstrated, not asserted. 100 GE is already being
> used for access uplink today.
I'm not actively "not caring" about transit operators, I'm just focusing 
my energy where I think it's most needed first.
Anyway, what I choose to do doesn't limit what everyone else does. 
Nonetheless, I have been helping core folks put in place the pieces 
needed early, so these next steps can be made.

> If you guys had agreed to isolate your L4S experiment's traffic with
> experimental DSCPs, people would have shrugged and said "go ahead" and
> you could have started your experiment two years ago with IETF's
> blessing and gotten real operational data by now. Of course, people are
> free to do whatever they want with their gear and their networks, IETF
> blessing or not, so if you can get operators to agree to play with your
> proposed ECN signaling, carry on. If vendors (e.g., Nokia) had not
> already implemented the proposed L4S ECN signaling in hardware and then
> tried to use that as an argument in support of ratifying such signaling
> in the IETF (and carrying it into 3GPP, FFS),
I'm not aware that anyone has ever used that as an argument. Companies 
are taking a risk to implement before the RFC is baked. That might look 
like applying pressure on others to go the same way. But I think that's 
in the eye of the beholder. They are the ones having to take the risk.

> then I wouldn't care what
> signaling is used *for the experiment*. As it is, I oppose changing the
> definition of ECN bits
Er, but your proposal changes the definition of ECN bits to SCE. Yes, 
it's theoretically conditional on an accompanying DSCP. But, to make it 
work, silicon in TCP offload hardware has to be updated. Not to mention 
silicon in tunnel decapsulators, silicon that does the ECN marking, the 
checksum recalculations, etc. Not a reversible experiment, either way.

I understand your concern. And you're entitled to take a stand. But it's 
not as black and white as you make out. It's a balance of a number of 
factors across the layers, including whether existing *applications* 
have to be changed.

That's why we asked the IETF from the start (well, after 2 years of our 
own experiments). And I have to say that none of the factors that went 
into the WG's decision to go with ECT1 as an identifier have changed.
*
*

> until L4S (or any other experiment) successfully
> concludes, especially not when a (IMHO) reasonable alternative exists.
As above, the "reasonable alternative" is flakey at best.

*If you hobble an experiment too much unnecessarily, you make it fail.*

Bob

> Argue about the optimal ECN semantics for Internet-wide deployment
> after you've gotten some operational experience.
>
>
> Regards,
>
> // Steve
>
>
>
>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 8 Mar 2020 14:23:39 +0000

Sebastian,

> 	[SM] That really just would show that there is a business case for ISP A and B to connect directly in a DSCP conserving fashion....
This is a misunderstanding of the sequencing of adoption incentives. If 
something new doesn't work in the scenarios of interest, it will never 
take off, and never affect anyone's business cases for doing anything.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 8 Mar 2020 15:07:33 +0000

Jonathan,

> I broadly agree with this argument.  And, as noted further down, the relatively small improvements in latency will only be significant on paths where the baseline RTT is already quite small, implyi…
The benefits of removing say 20ms of tail latency easily apply in a 
country up to the size of the US (45ms RTT coast to coast in glass).

I assume that you are aware that there is more than 1 ISP in the US.
And more than 1 ISP in Europe for that matter.

Details are important. And being on the right planet is even more important.

> And this means that if a Diffserv codepoint is used to request low-latency service for an SCE flow, it can be given by simply adjusting AQM parameters for that traffic, *without* impairing safety s…
Yes, all this is known.
SCE is not the only way of doing safety though.

> We are working on a demonstration of this capability, which we may be able to show at Vancouver (remotely), perhaps via the Hackathon.  Of course the fully-integrated code will also be publicly ava…
As I said, we will publish results of another way of doing safety (that 
we also identified right from the start): Classic ECN AQM detection and 
fall-back. Probably early this week, but certainly also in Vancouver.

SCE shows that, if you hobble an experiment too much, you make it 
unworkable over real networks with real tunnels, real transport 
protocols and real applications.

For instance, is there any interest from the major end-system stacks in 
implementing SCE rather than L4S, given how often SCE is unlikely to work?

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 08 Mar 2020 16:34:18 +0100

Hi Bob,

More below.

On March 8, 2020 3:23:39 PM GMT+01:00, Bob Briscoe <[email-masked]> wrote:
>> 	[SM] That really just would show that there is a business case for
>ISP A and B to connect directly in a DSCP conserving fashion....
>>

[SM] That might pass as a cogent argument, but for the fact that ISPs (and CDNs) already use SLAs to define their interactions, e g. for VoIP traffic. The incremental effort to also include a LL-type traffic class/dscp seems pretty minor to me.

I also strongly believe that the only incentive for an ISP to roll-out L4S would be if that additional work/expense can be monetized, and the most viable route for that I can see, is to discrimination-free sell L4S access to the deploying ISP's eyeballs to all interested content suppliers, be it directly or via CDNs. In that case an SLA is not only not going to be a stumbling block, but rather a mandatory component of making this work.

What kind of deployment are you realistically expecting where this is going to be an issue?

And especially for the scope of the experimental deployment? The experiment is required to make sure that L4S can deployed in a safe and backward compatible fashion and that it can deliver it's promises under real-world conditions.
The experiment is NOT about how to deploy something in a fashion that offers the least part of resistance/the highest adoption rate. As a rule of thumb, I would assume the IETF be interested in the technical aspects and not the marketing side....

Regards
        Sebastian

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 8 Mar 2020 18:27:37 +0000

Seabstian,

> [SM] That might pass as a cogent argument, but for the fact that ISPs (and CDNs) already use SLAs to define their interactions, e g. for VoIP traffic. The incremental effort to also include a LL-ty…
Companies don't just do these things lightly, especially not for 
applications that might not be possible any other way (both low latency 
and capacity-seeking high bandwidth has traditionally been considered 
impossible). It took decades to drag companies, kicking and screaming, 
into the VoIP models you see now. Technical ability was never the 
problem. I have a VHS video somewhere of a multi-party voice conference 
over ARPANET, including a radio link, and a leg from a public call box. 
I've been involved in interconnect QoS (economics and technology) for 
longer than I'd care to say (not as far back as ARPANET I should add!).

But my point was different. No-one has working technology until we have 
ECT(1) in an RFC. If we add DSCPs as yet another barrier to deployment, 
it appears flakey and the experiment fails. Then it never gets near to 
any commercial considerations, 'cos it doesn't exist.

>
> I also strongly believe that the only incentive for an ISP to roll-out L4S would be if that additional work/expense can be monetized, and the most viable route for that I can see, is to discrimination-free sell L4S access to the deploying ISP's eyeballs to all interested content suppliers, be it directly or via CDNs. In that case an SLA is not only not going to be a stumbling block, but rather a mandatory component of making this work. What kind of deployment are you realistically expecting where this is going to be an issue?
I don't think anyone at the IETF should be expecting Internet businesses 
to adopt one particular business model.
I believe our goal should be to produce technology over which all 
different business models can be built.

For that, I believe a strongly end-to-end technology (i.e. signal) needs 
to be the foundation, but different networks can derive different 
commercial offerings from it in different ways, without interfering with 
the underlying signal. {Note 1}

Diffserv is starting from the pit of the valley in this respect. It is 
expected to be wiped unless otherwise agreed.

In contrast, ECN starts from the high ground, starting from strongly e2e 
semantics. Any operator that wants to wipe or block an ECN signal from 
their competitors has to justify messing with e2e congestion control. 
{Note 2}

The distinction is between a technology that ISPs can use to degrade 
their competitors (race to the bottom), vs one they can only use to 
enhance themselves.

You only have to look at my publications pages over the last 20 years to 
see how much I've thought about the integrity of congestion signals as a 
basis for economic models.

Bob

{Note 1}: E.g. initially some might only classify ECT1 into the L queue 
for business customers, or as a product for a higher tier of customers, 
or solely for the operator's own services (if allowed in their 
jurisdiction). Other ISPs, like you say, will want to use it as a 
differentiator for their whole regular service (see draft-ietf-l4s-arch).

{Note 2}: Of course, certain ISPs might pervert the ECN signal, but I 
think that's less likely, 'cos they can only access to traffic in their 
own network, which inherently hits the e2e congestion control of their 
own customers. If we think that's a possibility, L4S senders could cover 
the least significant bit of the ECN field with integrity protection to 
raise the bar against network interference, by tying it to the integrity 
failure of each whole packet.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 8 Mar 2020 20:52:58 +0100

Hi Bob,

more below in-line.

> On Mar 8, 2020, at 19:27, Bob Briscoe <[email-masked]> wrote:
> 
> Seabstian,
> 
> Companies don't just do these things lightly,

[SM] Which does not really matter, if the companies already have a SLA in place. If L4S offers something the market desires it will be included into the SLAs otherwise if not even those parties that could maximally profit (even from the systematic sharing failures) are willing to put in a modicum of effort it tells us something about L4S.

> especially not for applications that might not be possible any other way (both low latency and capacity-seeking high bandwidth has traditionally been considered impossible).

	[SM] The bigger point is IMHO, is greedy and low-latency actually a desirable combination of features?

> It took decades to drag companies, kicking and screaming, into the VoIP models you see now.

	[SM] Yes? What is your point?

> Technical ability was never the problem. I have a VHS video somewhere of a multi-party voice conference over ARPANET, including a radio link, and a leg from a public call box. I've been involved in interconnect QoS (economics and technology) for longer than I'd care to say (not as far back as ARPANET I should add!).

	[SM] Bob, this are nice anecdotes but at best tangentially related to the L4S/ECT(1) fiasco in the making.

> 
> But my point was different. No-one has working technology until we have ECT(1) in an RFC. If we add DSCPs as yet another barrier to deployment, it appears flakey and the experiment fails.

	[SM] Failure of the L4S experiment is exactly my prediction, but due to its own internal inconsistencies and short-coming, nothing related to dscps,, but I would really hate it if L4S flame-out would both spoil ECT(1) for the immediate future and discredit 1/p type congestion signaling. 

> Then it never gets near to any commercial considerations, 'cos it doesn't exist.

	[SM] Slightly different prediction, no commercial considerations because it does not nearly deliver what it promises.

> I don't think anyone at the IETF should be expecting Internet businesses to adopt one particular business model.

	[SM] Exactly, the IETF should judge the L4S proposal on its technical merits alone, and there L4S simply is lacking.

> I believe our goal should be to produce technology over which all different business models can be built.
> 
> For that, I believe a strongly end-to-end technology (i.e. signal) needs to be the foundation, but different networks can derive different commercial offerings from it in different ways, without interfering with the underlying signal. {Note 1}

	[SM] But here is the rub, the L4S architecture makes in unduly hard to separate low-latency queueing from 1/p-type congestion control, as I fully agree these are ORTHOGONAL to each other, so artificially coupling them, e.g. by making ECT(1) both denote a 1/--typre response to CE AND requestimg admission to the LL-queue is exactly the wrong thing to do. Glad that you agree.

> 
> Diffserv is starting from the pit of the valley in this respect. It is expected to be wiped unless otherwise agreed.

	[SM] Well, in practice probably yes, but according to RFC2474DSCPs seem intended for "end-to-end or intra-domain" use. But in a sense an all wiped until agreed otherwise will give an additional safety mechanism for the roll-out of something new.

> 
> In contrast, ECN starts from the high ground, starting from strongly e2e semantics. Any operator that wants to wipe or block an ECN signal from their competitors has to justify messing with e2e congestion control. {Note 2}

	[SM] Again you seem to argue for convenience over technical merit.

> 
> The distinction is between a technology that ISPs can use to degrade their competitors (race to the bottom), vs one they can only use to enhance themselves.

	[SM] As so often you are charmingly naive about what is possible: all an ISP needs to do is to upmark a high volume (ideally non-responsive bursty) flow going to a competitor's network all ECT(1): "mischief managed". In the other directions we do not go as far, just pipe an L4S flow into a single queue RFC3168 AQM an watch the fire-works (and with your approach to L4S first, even your 3168 detection will in all likelihood cause transient queueing and hence increased latency for all other flows traversing that node).

> 
> 
> You only have to look at my publications pages over the last 20 years to see how much I've thought about the integrity of congestion signals as a basis for economic models.

	[SM] I perused a fair share of your prose in the last two years, but I buy into very little of your conclusions, as I believe you are working mostly from premises I do not agree with/share (the biggest one being the idea a congested node had any reliable information about the relative value of different packets it can use to optimize the assignment of bandwidth in any meaningful way, but I digress). That seems all irrelevant to L4S though, and especially why gauarding L4S use of ECT(1) with a bespoke DSCP would hinder EXPERIMENTAL roll-out of L4S in any meaningful way. As far as I can tell, you seem to think about how to deploy L4S as a standards RFC, but how about first making sure there is a viable path to it becoming a standard first?

Best Regards
	Sebastian

> 
> 
> 
> Bob
> 
> {Note 1}: E.g. initially some might only classify ECT1 into the L queue for business customers, or as a product for a higher tier of customers, or solely for the operator's own services (if allowed in their jurisdiction). Other ISPs, like you say, will want to use it as a differentiator for their whole regular service (see draft-ietf-l4s-arch).
> 
> {Note 2}: Of course, certain ISPs might pervert the ECN signal, but I think that's less likely, 'cos they can only access to traffic in their own network, which inherently hits the e2e congestion control of their own customers. If we think that's a possibility, L4S senders could cover the least significant bit of the ECN field with integrity protection to raise the bar against network interference, by tying it to the integrity failure of each whole packet.
> 
> 
>>

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 9 Mar 2020 09:10:08 +0100

Dear All,

a request for clarification from the chairs below.

> On Mar 8, 2020, at 15:18, Bob Briscoe <[email-masked]> wrote:
> 
> Steve,
> 
> Exec summary: 

Reading this makes me wonder what experimental RFCs are designed for.
What is the scope/rationale for granting an RFC experimental status? According to https://www.ietf.org/standards/process/informational-vs-experimental/:
"4.2.1 Experimental

The "Experimental" designation typically denotes a specification that is part of some research or development effort. Such a specification is published for the general information of the Internet technical community and as an archival record of the work, subject only to editorial considerations and to verification that there has been adequate coordination with the standards process (see below). An Experimental specification may be the output of an organized Internet research effort (e.g., a Research Group of the IRTF), an IETF Working Group, or it may be an individual contribution.
[...]
	• If the IETF may publish something based on this on the standards track once we know how well this one works, it's Experimental. This is the typical case of not being able to decide which protocol is "better" before we have experience of dealing with them from a stable specification. Case in point: "PGM Reliable Transport Protocol Specification" (RFC 3208)
	• If the document contains implicit or explicit success/failure criteria, and it's clear that the outcome can be used as the basis for a recommendation to the IETF community, it's Experimental. Case in point: RFC 1797 "Class A Subnet Experiment" which led to RFC 1879 "Class A Subnet Experiment Results and Recommendations"

"

I fail to see how "ease of deployment" fits into this framework as rationale for granting experimental status (IMHO if an experiment is successful deployment issues are worth discussing, if elevation into a full internet standard seems merited). However, safe coexistence with the existing internet, and proper test of functionality under diverse/adverse real-world conditions seem good arguments for granting experimental status. But for this testing requiring an abundance of caution and additional safety features and a big fat kill-switch for something with as many known side-effects as L4S is just good engineering practice. 

QUESTION @Chairs: What are the conditions and aims for the TSVW WG in deciding to grant an RFC like L4S experimental status? I would love to get a clarification please.

Many Thanks in advance

Regards
	Sebastian

[snipped text]

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 11 Mar 2020 11:50:01 -0400

> QUESTION @Chairs: What are the conditions and aims for the TSVW WG in deciding to grant an RFC like L4S experimental status? I would love to get a clarification please.

There are 3 chairs, and I only speak for 1, but the bottom-line is 
simply "WG rough consensus" (decently described in RFC 7282).

Beyond that, I don't think we are adding or removing any explicit 
criteria.  The RFCs on congestion control evaluation cover things that 
have traditionally been of interest/concern to the community in 
developing consensus that technologies were safe enough for experimental 
deployment on the Internet.  In the case of the technologies that are 
proposing to use ECT(1) for experiments today, I don't think this is any 
different.

As a chair, when sending to the IESG for recommended publication, I also 
want to see that:

(1) There is wide interest in deploying the technology.

(2) The concerns with it are understood well enough to be considered by 
people deploying it, and they are aware of what they might want to 
measure or what means are possible to mitigate problems.

Sometimes the "experiments" don't clearly succeed (global deployment) 
nor fail (nil deployment), but wind up very useful in some cases, and 
not of interest in others.  In the case of ECT(1) (considered by itself 
without any DSCP-scoping), there probably should be some bit of a plan 
or analysis around what happens if the technology persists in pockets 
where it's useful, but doesn't have 100% deployment over the Internet.  
We will need to consider the status of the ECT(1) codepoint, and how it 
can be used in the future whether these experimental specs are 
successful (widely deployed), partially successful (deployed in 
pockets), or failed (not deployed, or found defective and eventually 
un-deployed).

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 12 Mar 2020 08:52:42 +0100

Hi Wes,

Thank you! More below in-line.... 

On March 11, 2020 4:50:01 PM GMT+01:00, Wesley Eddy <[email-masked]> wrote:
>> QUESTION @Chairs: What are the conditions and aims for the TSVW WG in
> deciding to grant an RFC like L4S experimental status? I would love to
> get a clarification please.
> 
> 

	[SM] Oha, that RFC was in equal parts interesting, enlightening, and unhelpful ;) 

> proposing to use ECT(1) for experiments today, I don't think this is

	[SM] I guess then I want to start a discussion about the starvation issue. RFC2914 about CC principles indicates strongly, that starving-out flows is unacceptable and equitable sharing of bandwidth (modulo some conditions) is desirable. But I will try to start that discussion in a separate thread about whether or not the current L4S implementation and design meets these principles sufficiently.

> measure or what means are possible to mitigate problems.

[SM] Quite measured and convincing arguments.

> pockets), or failed (not deployed, or found defective and eventually 

[SM] That seems to acknowledge the fact that even a failed L4S experiment might spoil ECT(1) for a long time. This IMHO is supporting the idea that the experiment should only be started with mechanisms in place that allows a quick abort and recovery of ECT(1) on failure (given that there is an alternative proposal for ECT(1) usage in front of the WG). What are the technical arguments against rolling-out L4S experimentally in a cautious, measured safe and un-do-able way? The biggest arguments I hard so far were convenience (don't want to do more work than necessary) and easy of internet-wide deployment. IMHO the first is understandable but weak, while the second seems rather pre-mature given that L4S experiment and L4S as a a standard can be deployed using quite different methods and restrictions (so experiment under strict control and safety mechanisms and on success deployment under relaxed conditions seems like the obvious strategy).
Especially since the core questions L4S as an experiment is supposed to answer are a) will the positive lab-results also transfer into and materialize in the real-world  internet (functionality) and b) will they do so in a safe and compatible way with the existing internet (safety). 

Best Regards
         Sebastian

---

From: Lars Eggert <[email-masked]>
Date: Thu, 12 Mar 2020 10:18:50 +0200

Hi,

On 2020-3-11, at 17:50, Wesley Eddy <[email-masked]> wrote:
> There are 3 chairs, and I only speak for 1, but the bottom-line is simply "WG rough consensus" (decently described in RFC 7282).

I wonder (1) what the current status is and (2) if it can be summarized in the way to let me decide if I personally have an opinion.

I'm at least somewhat clueful about ECN and queuing, and I actually care about the topic (I want QUIC to work well in the presence of this, ideally taking advantage of it).

However, I have long ago given up on following the torrent of emails on this topic. There seem to be two camps that each believe they have a superior solution, fighting themselves to death over the details. More detail and more argumentation is unlikely to get me to a point where I feel comfortable on doing a hum.

I'd much rather see some short answers on some basic questions, such as:

Is the solution incrementally deployable?
Is it simple?
Does it offer better performance, to participating and ideally non-participating flows?
Is it at worst a no-op, or can it degrade performance for non-participating flows?
Can a partially deployed solution be rolled back?
Can it stay deployed but become a no-op if a different mechanism wins out in the end?
etc.

Because without some basic information along these lines, I fear I'd at the moment hum to not do anything, and that feels like we'd waste an opportunity.

Lars

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 12 Mar 2020 13:45:14 +0200

> On 12 Mar, 2020, at 10:18 am, Lars Eggert <[email-masked]> wrote:
> 

I think the question posed by David Black's slides (as presented and briefly discussed at the interim meeting) is intended to elicit relevant answers to these sorts of questions.  But since you asked, I'll have a go at answering them standalone, for the SCE proposal.

> Is the solution incrementally deployable?

Yes.  Each component of SCE is inherently safe to deploy into an existing network, and will have no detrimental effect on conventional traffic.  SCE flows traversing an existing network will behave conventionally and in an RFC-compliant manner.  This is not merely aspirational, but baked fundamentally into the design.

> Is it simple?

Yes.  I could summarise the technicalities on two slides - or one, if I really had to.  Implementing SCE also does not require very intrusive or extensive changes to existing network stacks; a year ago, we got a basic proof of concept running in about a man-week.

> Does it offer better performance, to participating and ideally non-participating flows?

Yes - adding SCE to an existing ECN network usually improves throughput, peak latency and jitter for SCE flows, and improves total link goodput when at least one SCE flow is present.  Adding SCE to a dumb-FIFO network also brings significant benefits to conventional traffic, as any good AQM deployment would.

> Is it at worst a no-op, or can it degrade performance for non-participating flows?

Non-SCE flows see only an RFC-3168 compliant AQM on the path, and ignore the SCE signals.  SCE flows at worst have a tendency to defer to existing traffic - the opposite to what you fear - but with a well-designed bottleneck qdisc, this effect is limited or eliminated.  The SCE proposal presents several such qdiscs as examples.

> Can it stay deployed but become a no-op if a different mechanism wins out in the end?

This is probably the most complex question to answer.  Un-deploying an experiment is always more difficult than deploying it in the first place, since it's harder to prove a negative.

SCE consumes the spare ECN codepoint and one spare TCP header bit, without explicitly negotiating for them (beyond what is normal for RFC-3168 ECN), and adds a permitted ECN codepoint transition in middleboxes: ECT(0) -> ECT(1).  As far as conventional ECT flows are concerned, ECT(1) is semantically identical to ECT(0), and the spare header bits are ignored if received, so conventional traffic would not be affected.  The question is really about how to free up these consumed header resources, should that be necessary.

Reversing an SCE deployment thus means removing all sources of this new usage, which means both middleboxes and endpoints.  The reference implementation leaves SCE functionality switched off by default, but it is easy to automate switching it on and potentially then forget about it.

The following probes can be used to seek out latent deployments, and ask their operators to disable them.  This does not require a new kernel to be compiled or installed, nor any downtime, only that the SCE functionality is configured off at runtime, eg. through a sysctl or the iproute2 tc utility.

SCE enabled middleboxes are easy to detect, by passing ECT(0) traffic through it and seeing if ECT(1) marks appear alongside (or before) CE marks.  If CE marks or an AQM-like (or tail-drop-like) pattern of drops appears but ECT(1) does not, it's not an SCE enabled middlebox.  This is probably sufficient to reclaim the spare ECN codepoint.

SCE enabled TCP receivers are also easy to detect, by probing them with ECT(1) traffic and seeing if the corresponding TCP header bit is used to echo them.  Some other transports may have ECT(1) feedback designed into them, which is not affected by SCE per se, and these can be ignored.  This is probably sufficient to reclaim the TCP header bit.

SCE enabled senders are a little more difficult to detect unambiguously.  An indication can be obtained by probing them with the TCP header bit artificially set in all acks, and seeing if the resulting throughput is markedly less than if that is not done.  SCE senders behave essentially identically to normal ECN senders when the corresponding TCP header bit remains cleared in acks, so their continued presence is mostly benign, unless the header bit starts being used in some other experiment in a way that confuses the SCE sender.

If considered necessary by the WG, it would be possible to associate SCE with an experimental-series DSCP, and require that DSCP be present for SCE functionality to be enabled.  This would however limit experiments to networks that do not bleach or otherwise disrupt that DSCP, which may inhibit exploration of behaviour on longer Internet paths (which we think SCE is much better suited for than L4S).  The upside would be easier reclamation of the header resources upon termination of the experiment.

Our proposed use of DSCP is instead an optional addition, using SCE as part of providing a low-latency Diffserv PHB, but not requiring any particular DSCP to support SCE.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 12 Mar 2020 12:19:29 +0000

This was to be an important topic of the face-to-face meeting in 
Vancouver, where we have been planning to allow time for different 
viewpoints to be aired and the consensus of the group to be sought. We 
still plan to do that, and are considering how best to proceed.

Comments in-line to show the expected direction of travel, but please do 
avoid growing this discussion thread for a week or so, until we have a 
plan to conclude it (see end of email).

> Yes.  Each component of SCE is inherently safe to deploy into an existing network, and will have no detrimental effect on conventional traffic.  SCE flows traversing an existing network will behave…
This is exactly one of the topics we need to seek thought upon.
> Yes.  I could summarise the technicalities on two slides - or one, if I really had to.  Implementing SCE also does not require very intrusive or extensive changes to existing network stacks; a year…
Yes, we should capture those two slides for SCE! We will also ask the 
same for L4S.
> Yes - adding SCE to an existing ECN network usually improves throughput, peak latency and jitter for SCE flows, and improves total link goodput when at least one SCE flow is present.  Adding SCE to…
One slide on how SCE offers benefit will also be useful. Again we should 
obtain the same for L4S.
> Non-SCE flows see only an RFC-3168 compliant AQM on the path, and ignore the SCE signals.  SCE flows at worst have a tendency to defer to existing traffic - the opposite to what you fear - but with…
... Or rewrite ECT(1),  as I understand for some existing 
tunnels/encaps. I recall from the early L4S discussion that this 
consideration was also be important, and hence we charted work to try 
and improve ECN-marking in the IP layers.

> SCE consumes the spare ECN codepoint and one spare TCP header bit, without explicitly negotiating for them (beyond what is normal for RFC-3168 ECN), and adds a permitted ECN codepoint transition in…
Yes. I would love such a brief understanding of how the experiment can 
be managed if the IETF may wish in the future to terminate an 
experiment, as was done for ECN-NONCE.
> SCE enabled middleboxes are easy to detect, by passing ECT(0) traffic through it and seeing if ECT(1) marks appear alongside (or before) CE marks.  If CE marks or an AQM-like (or tail-drop-like) pa…
Probably worth writing down. I've found it hard to test for ECN-marking 
on operational paths, but insight in this is always going to be welcome.
> Our proposed use of DSCP is instead an optional addition, using SCE as part of providing a low-latency Diffserv PHB, but not requiring any particular DSCP to support SCE.

The TSVWG Chairs are now aware that we need to plan to make progress and 
will be in touch in the next week or so, with a plan move forward in the 
absence of an IETF face-to-face meeting.

Best wishes,

Gorry,

TSVWG Chairs

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 12 Mar 2020 15:07:14 +0200

> On 12 Mar, 2020, at 2:19 pm, Gorry Fairhurst <[email-masked]> wrote:
> 
>>> Is it simple?

> Yes, we should capture those two slides for SCE! We will also ask the same for L4S.

> One slide on how SCE offers benefit will also be useful. Again we should obtain the same for L4S.

We'll bear that in mind when preparing.

> ... Or rewrite ECT(1),  as I understand for some existing tunnels/encaps. I recall from the early L4S discussion that this consideration was also be important, and hence we charted work to try and improve ECN-marking in the IP layers.

I think we will need to have a focused discussion about tunnels at some point.  For the moment, I'll just note that if the SCE signal is erased (on either the forward or feedback path), then the flow will revert to conventional RFC-3168 behaviour, which is conservatively safe.

> Probably worth writing down. I've found it hard to test for ECN-marking on operational paths, but insight in this is always going to be welcome.

Granted, there is the corner case where the middlebox in question is not a bottleneck on the path under test - so SCE or ECN functionality may be present but not visible.  That would appear to be the chief difficulty with achieving full probe coverage.  But I think enough confidence could be obtained by such tests to permit reclaiming the codepoint for another experiment in a reasonable timeframe.  We can try paying more rigorous attention to this question if the WG deems it necessary.

> This was to be an important topic of the face-to-face meeting in Vancouver, where we have been planning to allow time for different viewpoints to be aired and the consensus of the group to be sought. We still plan to do that, and are considering how best to proceed.
> 
> Comments in-line to show the expected direction of travel, but please do avoid growing this discussion thread for a week or so, until we have a plan to conclude it (see end of email).

> The TSVWG Chairs are now aware that we need to plan to make progress and will be in touch in the next week or so, with a plan move forward in the absence of an IETF face-to-face meeting.

Noted.

 - Jonathan Morton

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 12 Mar 2020 13:19:43 +0000 (UTC)

Not to invite more growing of this thread, since Gorry has asked that this not happen, but I notice that L4S and SCE have taken
quite different approaches to defining their value proposition. Neither of these approaches are wrong, but his makes them hard to compare. Therefore when drafting materials for whatever meeting occurs, it would be good if the parties could try to close this gap, perhaps with hints from the chairs on how to do so. In particular:
 
- L4S has been very clear about what the end goal is (it's in the name). Other considerations, like the present state and limitations thereof, require a lot more work to understand.
- SCE has focused on the benefits shown in tested examples with the existing codebase. They have been cautious about commenting on how universal such benefits could become at different deployment stages.

Alex

On Thursday, March 12, 2020, 12:48:03 PM GMT, Gorry Fairhurst <[email-masked]> wrote: 

> Non-SCE flows see only an RFC-3168 compliant AQM on the path, and ignore the SCE signals.  SCE flows at worst have a tendency to defer to existing traffic - the opposite to what you fear - but with…
.... Or rewrite ECT(1),  as I understand for some existing 
tunnels/encaps. I recall from the early L4S discussion that this 
consideration was also be important, and hence we charted work to try 
and improve ECN-marking in the IP layers.

Best wishes,

Gorry,

TSVWG Chairs

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 16 Mar 2020 09:13:22 +0000

Jonathan, [re-sending with improved quoting, 'cos it's been pointed out 
that the quoting made it impossible to work out who was saying what]

On 08/03/2020 01:35, Jonathan Morton wrote:
>> [SM] I fully understand the challenges of using DSCPs end to end, and 
>> yet DSCPs still seem to be the IETF's endorsed intra-domain and 
>> end-to-end marking mechanism (see RFC 2474). Any other marking 
>> use-case (like the LE PHB) could lay claim on ECT(1) by the same 
>> rationale as L4S as it would be currently more reliable. Sure, L4S 
>> could claim that it also uses functionally uses ECN and hence has a 
>> slightly stronger claim, but since the L4S arch draft explicitly 
>> mentions allowing non-responsive bounded-rate (and hence essentially 
>> non ECN-aware) flows in the LL queue it seems clear that ECT(1) in 
>> addition to all it's other issues is not really fulfilling L4S's 
>> marking needs exhaustively.
>>
>> That said, the only real uses L4S is likely going to see any time 
>> soon, IMHO, are those that have an immediate monetary advantage for 
>> the ISP rolling out and operating the L4S infrastructure, and hence 
>> SLAs between content suppliers and ISP that make sure to conserve the 
>> L4S DSCPs seem like a natural solution to the issue.
>> In the spirit of walking before running it would IMHO behoove the L4S 
>> effort not trying to solve all potential problems a priori, but 
>> rather experimentally demonstrate that even a restricted scope L4S 
>> deployment has actual merits in real life.
> []JM I broadly agree with this argument. And, as noted further down, 
> the relatively small improvements in latency will only be significant 
> on paths where the baseline RTT is already quite small, implying both 
> a short geographic distance and a small number of routing hops between 
> the endpoints. These are exactly the conditions most likely to be able 
> to deploy a Diffserv solution through the normal SLA process.

[BB] The benefits of removing say 20ms of tail latency easily apply in a 
country up to the size of the US (45ms RTT coast to coast in glass).

>
> [JM] The challenge with L4S is that it relies on the classifier not 
> only for low-latency service (a performance improvement, whose loss 
> would typically be tolerable) but to configure the type of congestion 
> signalling (a safety requirement, whose loss would be dangerous). This 
> is, I think, why they sought a classifier which would reliably 
> traverse AS boundaries. This design requirement ultimately stems from 
> their early design decision to perpetuate the DCTCP signalling method, 
> which changes the semantics of CE.
>
> Unfortunately this is not the only requirement exposed by that 
> decision. Not only must the classifier signal *survive* throughout the 
> path, but every potential bottleneck node must *understand* the 
> classifier and apply the different congestion signalling. This puts 
> L4S back in the position of having to limit deployment to controlled 
> environments, only now without the Diffserv "containment" mechanism to 
> help indicate when it strays outside of such an environment. This in 
> turn leads to the present attempt to work around the problem with 
> "classic detection" algorithms, which I believe will show significant 
> detection latency and will therefore be unable to switch modes as 
> seamlessly as SCE has already demonstrated.
>
> With SCE we have approached the problem from a different direction 
> from the start: Safety First. The spare ECN codepoint is used as an 
> additional output from the network, and CE retains its existing 
> meaning and usage. This means SCE can be deployed across any existing 
> network with congestion safety, competing normally with conventional 
> traffic - because on a conventional network an SCE flow is almost 
> indistinguishable from a conventional flow. The performance benefits 
> begin to appear, seamlessly, when the bottleneck on the path is SCE 
> enabled.
>
> And this means that if a Diffserv codepoint is used to request 
> low-latency service for an SCE flow, it can be given by simply 
> adjusting AQM parameters for that traffic, *without* impairing safety 
> should this classifier be lost en route, and with completely 
> equivalent performance. I think this is a very important point that 
> the L4S team has overlooked, in their continual attempts to discredit 
> the SCE project.

[BB] Yes, all this is known.
SCE is not the only way of doing safety though.

>
> [JM] We are working on a demonstration of this capability, which we 
> may be able to show at Vancouver (remotely), perhaps via the 
> Hackathon. Of course the fully-integrated code will also be publicly 
> available at the time of demonstration, so you can replicate and play 
> with the results.

[BB] As I said, we will publish results of another way of doing safety 
(that we also identified right from the start): Classic ECN AQM 
detection and fall-back. Probably early this week, but certainly also in 
Vancouver.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 16 Mar 2020 10:32:29 +0000

Sebastian,

There are two a very important misconceptions in this email that other 
people might be labouring under, so sorry for dredging up this week-old 
thread... inline...

> 	[SM] The bigger point is IMHO, is greedy and low-latency actually a desirable combination of features?

[BB] Not to have grocked this basic requirement is a serious disconnect.

Short answer:
Application-limited does not preclude needing to be capacity-seeking. 
"Application-limited" is decided when the application is written, 
whereas "capacity-seeking" (a.k.a. "greedy") copes with potential 
conditions encountered at run-time. So when a flow needs to be 
capacity-seeking, you don't want the onset of queuing to ruin the low 
latency.

Longer answer:
Presumably you're not disagreeing that being able to have high 
throughput at the same time as very low latency would enable many new 
applications (natural video-based interaction, video-based remote 
control, cloud-based AR/VR, online gaming, remote desktop, etc).

Even if the throughput of a video flow is usually application-limited, 
the higher the throughput, the more likely it will sometimes encounter 
scenarios where it is limited by available capacity. For instance,
a) when a path with a lower capacity bottleneck is used (e.g. I 
communicate with someone who has a lower capacity access link, or I move 
to a lower capacity access), or
b) because enough other traffic has arrived at the existing bottleneck.

Then you want the congestion control to become capacity-seeking (not 
unresponsive), without losing the low latency.

Every time L4S is presented or written about, the same high level 
requirement of high throughput and low latency is repeated. What did you 
think the whole L4S activity was about?

[I've snipped the next part of the conversation, because it was 
opinion-based and going nowhere.]

> 	[SM] But here is the rub, the L4S architecture makes in unduly hard to separate low-latency queueing from 1/p-type congestion control, as I fully agree these are ORTHOGONAL to each other, so artif…

[BB] Another rudimentary disconnect here...

Queuing delay is not caused by the queue, it's caused by the 
congestion-controllers that induce the queue.

1/p congestion controllers are what keeps the sawteeth small so that the 
ECN-marking threshold in the queue can be shallow without causing 
underutilization.

The CoDel config of 'target' (min queue depth) was recommended to be set 
quite low despite leading to underutilization for higher RTTs. However, 
if the congestion controllers are not scalable (1/p), even if they keep 
the sawteeth fairly small today, they will grow as link rate scales. 
Then we'll be back in the same hole in a few years time.

Unless we sort this out properly, the problem will always stay that we 
have to remain friendly to original Reno flows still likely to be 
running over the Internet. The L4S queue is a transition mechanism that
a) creates a clean-slate isolated from that slow-death problem.
b) gives immediate benefit today, which encourages adoption (it patently 
already has)

Again, every time L4S is presented or written about, this explanation is 
given up-front.

[Again I've snipped some opinion-based conversation, going nowhere...]

> 	[SM] That seems all irrelevant to L4S though, and especially why gauarding L4S use of ECT(1) with a bespoke DSCP would hinder EXPERIMENTAL roll-out of L4S in any meaningful way. As far as I can tell, you seem to think about how to deploy L4S as a standards RFC, but how about first making sure there is a viable path to it becoming a standard first?

[BB] If an experiment cannot transition to wider usage, potential 
adopters will notice and not even start. The limits to the experiment 
are best put in place by the experimenters, not by adding inherent 
limits to the technology.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 18 Mar 2020 17:33:46 +0000

Jonathan,

[Gorry asked us to hold this thread for a week or so - I've now asked 
him to allow it to proceed]
Inline...

On 12/03/2020 13:07, Jonathan Morton wrote:
>>>>> [LE] Is it at worst a no-op, or can it degrade performance for 
>>>>> non-participating flows? 
>>>> [JM] Non-SCE flows see only an RFC-3168 compliant AQM on the
>>>>   path, and ignore the SCE signals.  SCE flows at worst have a tendency
>>>> to defer to existing traffic - the opposite to what you fear - but with a
>>>>   well-designed bottleneck qdisc, this effect is limited or eliminated.
>>>> The SCE proposal presents several such qdiscs as examples.
[BB] As SCE AQMs are deployed, they SCE-mark any ECT0 IP header to ECT1, 
whether or not the source is taking part in the SCE experiment.

Currently nearly all packets on connections to & from Apple devices use 
ECT0. So an SCE AQM would routinely alter a high proportion of current 
Apple-customer-traffic to ECT1.

That may not be a problem. But it is not 'safety first'. If there are 
ECT1 black holes in the Internet, they will currently be benign to 
traffic traversing them, because no-one is using ECT1. However, if these 
black-holes do exist, deploying an SCE AQM will open up these holes 
beneath a potentially high proportion of any Apple-customer's traffic 
traversing them.

I have checked the four main large-scale ECN measurement studies (2011, 
2013, 2015, 2018) and none reported any experiments testing for ECT1 
black holes. Have any been conducted to your knowledge?

In the L4S approach, the source opts into the experiment by setting 
ECT1, so any L4S source that hits a black hole can opt back out of the 
experiment - for that path.

>>> [GF] ... Or rewrite ECT(1), as I understand for some existing 
>>> consideration was also be important, and hence we charted work to 
>>> try and improve ECN-marking in the IP layers. 
> [JM] I think we will need to have a focused discussion about tunnels at some point.  For the moment, I'll just note that if the SCE signal is erased (on either the forward or feedback path), then the flow will revert to conventional RFC-3168 behaviour, which is conservatively safe.
>
>>> [JM] SCE enabled middleboxes are easy to detect, by passing ECT(0) traffic through it and seeing if ECT(1) marks appear alongside (or before) CE marks. If CE marks or an AQM-like (or tail-drop-like) pattern of drops appears but ECT(1) does not, it's not an SCE enabled middlebox.  This is probably sufficient to reclaim the spare ECN codepoint.

[BB] Bear in mind that multiple parts need to deploy (sender, AQM, 
receiver) before there's any benefit.

Whether ECT1 is assigned to SCE or L4S, I think we should be honest with 
ourselves and plan for the codepoint not being reclaimable for at least 
20 years, even if deployment of all three parts never occurs (see nonce 
data point below). If there is widespread deployment but the experiment 
still fails despite this (e.g. an insurmountable technical flaw), I 
suspect at least double that (40 years) before use of the codepoint 
tails off enough to be reclaimed.

We should not understate the stakes.

>> [GF] Probably worth writing down. I've found it hard to test for ECN-marking on operational paths, but insight in this is always going to be welcome.
> [JM] Granted, there is the corner case where the middlebox in question is not a bottleneck on the path under test - so SCE or ECN functionality may be present but not visible.  That would appear to be the chief difficulty with achieving full probe coverage.

[BB] Even if the SCE AQM was the bottleneck, the measurement would have 
to be done just after the SCE AQM, because:

  * If measured at the receiver, absence of ECT(1) marks might still
    occur if an SCE AQM introduces ECT(1), but a subsequent tunnel decap
    eats them all up again.
  * If measured at the sender, absence of ECT(1) feedback could be due
    to the above tunnel problem, or due to the absence of a receiver (at
    least in TCP) that feeds back ECT(1).

    That's because an SCE TCP sender never knows whether it's paired up
    with a receiver that has ECT(1) feedback logic or not.
    (An SCE sender could test the receiver by sending fake ECT(1)
    markings to see if they cause feedback. But then that in turn would
    make in-network measurement think there was an SCE AQM when there
    wasn't.)

> But I think enough confidence could be obtained by such tests to permit reclaiming the codepoint for another experiment in a reasonable timeframe.  We can try paying more rigorous attention to this question if the WG deems it necessary.

[BB]  "Reasonable timeframe" needs to be understood in relation to the 
following data point. AFAICT there was one implementation of the ECN 
nonce on a private testbed, and possibly one instance deployed somewhere 
in Norway. But it still took 16 years of absence of measurements before 
we could reclaim the codepoint.

That said, when assigning a scarce codepoint in the IP header, designing 
an experiment for reversal is far less important than minimizing the 
chance the experiment will need to be reversed. That means a need for 
pre-existing strong (i.e. business, not just techie) interest in 
deploying /all/ the necessary parts of the experiment. That was what the 
ECN nonce failed on. With industry backing, even if there are teething 
problems, there will be energy (and money) to fix them.

I know the IETF doesn't normally look beyond 'techie', but it is capable 
of doing so.

Cheers

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 25 Mar 2020 09:48:50 +0100

Hi Bob,

Summary: more detailed discussion about semantics and disconnects, safe to ignore.

> On Mar 16, 2020, at 11:32, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> There are two a very important misconceptions in this email that other people might be labouring under, so sorry for dredging up this week-old thread... inline...
> 

	[SM] That is a tad impolite (given your own sensibilities about tone), no? Not accepting your premises is not the same as not understanding them.

> Application-limited does not preclude needing to be capacity-seeking.

	[SM] "capacity-seeking high bandwidth" I was mostly referring to the high bandwidth part, but I did not explicitly mention that, sorry if that caused confusion.

> "Application-limited" is decided when the application is written, whereas "capacity-seeking" (a.k.a. "greedy") copes with potential conditions encountered at run-time. So when a flow needs to be capacity-seeking, you don't want the onset of queuing to ruin the low latency.

	[SM] You are basically trying to sell me on the idea of  1/p-type congestion signaling, you can stop, that part I agree with (and both components, lower amplitude signaling and higher temporal fidelityas compared to rfc3168). Where we disagree is whether L4S is a sensible design to deploy such a congestion signaling into the wider internet.

> 
> Longer answer:
> Presumably you're not disagreeing that being able to have high throughput at the same time as very low latency would enable many new applications (natural video-based interaction, video-based remote control, cloud-based AR/VR, online gaming, remote desktop, etc).

	[SM] I am realist enough to understand that no low latency application will work robustly and reliably over the wider internet. But I also realize that you are arguing here for a "build it and they will come" approach, as you only have vage examples of use-cases that require low-latency-high-bandwidth, "would enable many new applications" is rather unspecific...

> 
> Even if the throughput of a video flow is usually application-limited, the higher the throughput, the more likely it will sometimes encounter scenarios where it is limited by available capacity. For instance,
> a) when a path with a lower capacity bottleneck is used (e.g. I communicate with someone who has a lower capacity access link, or I move to a lower capacity access), or
> Then you want the congestion control to become capacity-seeking (not unresponsive), without losing the low latency.

	[SM] Well adaptive streaming is an already deployed technique to deal with variable bandwidth paths, that works reasonably well, especially for pre-canned video where the real-time tolerance of users is relatively high... You really need an application where the end-user's feedback ends up in a tight control loop for the generation of the video before ultra-low queueing delay becomes essential. The roll-out of google stadia and nvidia gforce now with existing internet technology indicates that even without L4S the currently deployed tech allows sufficiently tight "real-time" control loops for interactive "video streaming". I am not saying that things can not be improved and that some of L4S components might not be great solutions, but I question the premise that L4S will allow anything qualitatively novel. Which IMHO is a big point when assessing L4S's side-effects.

> 
> 
> Every time L4S is presented or written about, the same high level requirement of high throughput and low latency is repeated.

	[SM] Well, repetition of a statement does not change its truth value, if at all it tells us about the persistence of the person repeating it.

> What did you think the whole L4S activity was about?

	[SM] This is a direct question I am not really sure you are interested in my answer, but here you go. As far as I see it, you are building a high priority high bandwidth lane onto customer access links that then can be monetized by ISPs by selling this high priority access to interested parties (on the content generating or consuming side). IMHO that is not a bad thing in itself, under the condition that it is fully opt-in and the end-user can decide whether they want that or not. A lot of users might be willing to spend money to improve their gaming quality (but that assumes that L4S will actually work well enough over arbitrarily long path between end-users and game-servers).

> 
> [I've snipped the next part of the conversation, because it was opinion-based and going nowhere.]

	[SM] ??? Most of our discussion is opinion-based including most of your arguments and I have repeated asked for hard data only to be told data is either irrelevant or that I should go measure myself if I want to see data.

> Queuing delay is not caused by the queue, it's caused by the congestion-controllers that induce the queue.

	[SM] Semantics, semantics semantics. Not terribly interested in that discussion. IMHO it is the queue that can, by say dropping immediately, shed the excess packets that cause the delay, not the end-points that can not really affect data once in flight. Sure it is the traffic (initiated by endpoints) that shows up in intermediary queues, but in the context of an AQM misbehaving (like dualQ is) pointing to the end-points is IMHO not helpful.

> 
> 1/p congestion controllers are what keeps the sawteeth small so that the ECN-marking threshold in the queue can be shallow without causing underutilization.
> 
> The CoDel config of 'target' (min queue depth) was recommended to be set quite low despite leading to underutilization for higher RTTs.

	[SM] Could you please give a citation where you got that idea from? The "underutilization for higher RTTs" is what we called RTT-dependence in another thread and where we agreed that it also affects TCP Prague, aka 1/p type AQMs/signaling.

> However, if the congestion controllers are not scalable (1/p), even if they keep the sawteeth fairly small today, they will grow as link rate scales. Then we'll be back in the same hole in a few years time.

	[SM] IMHO the reason why L4S is (still) under consideration for standardization is because everybody agrees that 1/p signaling with high temporal fidelity is really desirable.

> 
> Unless we sort this out properly, the problem will always stay that we have to remain friendly to original Reno flows still likely to be running over the Internet. The L4S queue is a transition mechanism that

	[SM] That green-field assumption is IMHO wish-ful thinking, there is no clean slate you need to coexist peacefully with the eExxisting internet; but that mindset effortlessly explains why you accept all the side-effects of the L4S design.

> b) gives immediate benefit today, which encourages adoption (it patently already has)

	[SM] Only that the amount of benefit is arguably not commensurate to its side-effects.

> 
> Again, every time L4S is presented or written about, this explanation is given up-front.
> 
> [BB] If an experiment cannot transition to wider usage, potential adopters will notice and not even start.

	[SM] That is not wrong, except that dropping a DSCP requirement when transitioning from experimental to standards track status seems like a very minor modification, and one that most ISPs are already comfortable with. So no I do not buy the argument that requiring a concomitant DSCP with ECT(1) to be the straw that breaks the camel's back in regard to experimental roll-out. And the fact that all you offer are bland general principles makes me believe that you grudgingly agree. 

> The limits to the experiment are best put in place by the experimenters, not by adding inherent limits to the technology.

	[SM] That indicates, that you consider the actual experimentation to be somebody else's problem though; pretty much what I feared releasing a half-backed design into the wider internet without appropriate safety mechanisms and no plan what the experiment is supposed to achieve. I guess I am tainted by what experiment means in my line of research.

Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 25 Mar 2020 12:44:04 +0200

> On 25 Mar, 2020, at 10:48 am, Sebastian Moeller <[email-masked]> wrote:
> 
> Not accepting your premises is not the same as not understanding them.

I think this is a crucial point that bears emphasis, given how some recent discussions have proceeded.

There are some points on which the L4S and SCE teams agree, and some points on which we disagree.  Many of these opinions are strongly held by each side, so we should not expect to convince each other of their truth - if that in fact happens, it's a pleasant bonus.

The people to be convinced are the Working Group at large.  Both teams will of course champion their own project.

What we need to do is ensure that the arguments being made are correctly *understood*, even if disagreement exists.  Correctly identifying the difference between disagreement with the intended argument, versus failing to convey the argument accurately, will definitely help with that.  But I think some of us are better at it than others.

For my own part, I will continue to seek clarification when it is obvious that an argument I tried to respond to was not the one intended.  On these occasions, I'll include a rephrased summary of the argument as I understood it, so that the point of misunderstanding can efficiently be resolved.  When I'm sure I understand the argument, I can then respond with the extent to which I agree or disagree with it, together with reasons why.  And, crucially, people reading that discussion will also understand the correct argument and put my response in the correct context.

I hope that everyone else in this debate will take a similar approach.  It will certainly make things more civilised.

 - Jonathan Morton