Subject: Re: [tsvwg] Comments on L4S drafts

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 8 Jun 2019 19:46:26 +0000

Hi

+1 on Bob's comment
"The aim here is to use the last unicorn in the world (ECT(1)) to the full.
If we don't make delay extremely low and extremely consistent we'll have
wasted it. So we must focus on 99th percentile delay"
I (and others) have tried to get ECN support in 3GPP access earlier. This is
in some cases made complicated by how the 3GPP protocol layers are defined,
the IP header is for instance ciphered or authenticated below the PDCP
layer, which makes I poses additional challenges if packets are queued up on
the RLC layer.
Up until now it has been quite a challenge to make ECN happen, I believe
that part of the reason has been that ECN is not judged to give a large
enough gain. 
L4S changes all this (I hope and believe), as it makes it possible to reach
very low queue delay, especially for NR. Besides this, L4S has the nice
property that it has potential to allow for faster rate increase when link
capacity increases. The latter is a welcome component when 5G access is to
be used to its full extent as the combined use of carrier aggregation, dual
connectivity, mix of low and high frequency band transmission etc., can make
the link throughput vary more than what is seen today with e.g. LTE. 
I see many applications that can benefit greatly from L4S, besides AR/VR,
there is also an increased interest in the deployment of remote control
capabilities for vehicles such as cars, trucks and drones, all of which
require low latency video streaming.
My bottomline is that I believe L4S provides with a clear benefit that is
large enough to be more widely accepted in 3GPP. SCE is as I see it more
like something that is just a minor enhancement to ECN and is therefore much
harder to sell in to 3GPP.   

Regards
Ingemar

> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 7 Jun 2019 19:07:53 +0100
> From: Bob Briscoe <[email-masked]>
> To: "Holland, Jake" <[email-masked]>, "[email-masked]"
> 	<[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Subject: Re: [tsvwg] Comments on L4S drafts
> Message-ID: <[email-masked]>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
> 
> Thanks Jake,
> 
> I'll address each of your questions inline. But I notice that I need to
lay down
> some context first.
> 
> The problem boils down to deployment incentives. The introduction of fine-
> grained congestion control requires changes to sender, receiver and at
least the
> bottleneck link before it is effective. ECN deployment faced the same
3-part
> deployment problem. So we tried hard to learn from it.
> 
> Faced with a 3-part deployment, no single party makes a move unless they
judge
> that the potential gain is worth the effort and that /all/ the other parts
(server,
> client, network) are strongly likely to make the same judgement {Note 1}.
> 
> The effort isn't just the coding, it's all the hassle dealing with
unexpected
> consequences of making the change, e.g. the risk of people's Internet
service
> being taken out by a middlebox black-holing the new protocol. High risk of
high
> cost/effort needs very high gain.
> 
> So the improvement has to be remarkable. Not just incremental, but
stunning
> enough to enable applications that are not even possible otherwise.
> 
> The aim here is to use the last unicorn in the world (ECT(1)) to the full.
If we
> don't make delay extremely low and extremely consistent we'll have wasted
it.
> So we must focus on 99th percentile delay (and more 9s if you want to take
> longer to measure it). Now, inline...
> 
> 
> On 05/06/2019 01:01, Holland, Jake wrote:
> > Hi Bob,
> >
> > I have a few comments and questions on draft-ietf-tsvwg-ecn-l4s-id-06
> > and draft-ietf-tsvwg-l4s-arch-03.
> >
> > I've been re-reading these with an eye toward whether it would be
> > feasible to make L4S compatible with SCE[1] by using ECN capability
> > alone as the dualq classifier (roughly as described in Appendix B.3 of
> > l4s-id), and using ECT(1) to indicate a sub-loss congestion signal,
> > assuming some reasonable mechanism for reflecting the ECT(1) signals
> > to sender (such as AccECN in TCP, or even just reflecting each SCE
> > signal in the NS bit from receiver, if AccECN is un-negotiated).
> >
> > I'm trying to understand the impact this approach would have on the
> > overall L4S architecture, and I thought I'd write out some of the
> > comments and questions that taking this angle on a review has left me
> > with.
> >
> > This approach of course would require some minor updates to DCTCP or
> > other CCs that hope to make use of the sub-loss signal, but the
> > changes seem relatively straightforward (I believe there's a
> > preliminary implementation that was able to achieve similarly reduced
> > RTT in lab) and the idea of course comes with some tradeoffs--I've
> > tried to articulate the key ones I noticed below, which I think are
> > mostly already stated in the l4s drafts, but I thought I'd ask your
> > opinion of whether you agree with this interpretation of what these
> > tradeoffs would look like, or there are other important points you'd
like to
> mention for consideration.
> May I give this proposal a name for brevity: ECN-DualQ-SCE (which sort-of
> represents ECN as the input classifier into 1 of 2 queues and SCE as the
output
> from that queue).
> 
> >
> >
> > 1.
> > Of course, I understand using SCE-style signaling with ECT capability as
> > the dualq classifier would come with a cost that where there's classic
ECT
> > behavior at endpoints, the low latency queue would routinely get some
> > queue-building, until there's pretty wide deployment of scalable
controllers
> > and feedback for the congestion signals at the endpoints.
> >
> > This is a downside for the proposal, but of course even under this
downside,
> > there's the gains described in Section 5.2 of l4s-arch:
> >     "State-of-the-art AQMs:  AQMs such as PIE and fq_CoDel give a
> >        significant reduction in queuing delay relative to no AQM at
all."
> Indeed, herein lies the problem. Imagine you are trying to convince a
> network operator to start a major project to tender for a new low
> latency technology then deploy it across their access network. You tell
> them it will also depend on:
> * servers/CDNs deploying new OS code.
> * and clients deploying new OS code.
> Then you tell them that, until /most/ servers deploy, and /most/ clients
> deploy (maybe a decade?), the low latency queue will routinely add as
> much queue delay as we can already get (without clients and servers
> changing)....
> 
> One day, you continue, if all the other servers and clients passing
> traffic through that box get upgraded, it will be cool. Until that day,
> a gamer in augmented reality gets stunningly low delay,... except every
> time her daughter in the bedroom looks at a mate's facebook page or
> watches a YouTube clip.
> 
> Is the network operator really going to take all those risks for jam
> tomorrow (=? maybe a decade)? I really don't think so.
> 
> Then we'll have burned the last unicorn to routinely get what we've
> already got.
> 
>   * Incremental deployment means, as you deploy the new capability, old
>     traffic continues to work, while new traffic gets the new service.
>   * As you say, with ECN-DualQ-SCE, new traffic only gets the new
>     service if there's no old traffic there. That's not only incremental
>     deployment; that's also ineffective deployment.
> 
> 
> >
> > On top of that, the same pressures that l4s-arch describes that should
> > cause rapid rollout of L4S should for the same reasons cause rapid
rollout
> > of the endpoint capabilities, especially if the network capability is
> > there.
> 
> I'm afraid there are not the same pressures to cause rapid roll-out at
> all, cos it's flakey now, jam tomorrow. (Actually ECN-DualQ-SCE has a
> much greater problem - complete starvation of SCE flows - but we'll come
> on to that in Q4.)
> 
> I want to say at this point, that I really appreciate all the effort
> you've been putting in, trying to find common ground.
> 
> In trying to find a compromise, you've taken the fire that is really
> aimed at the inadequacy of underlying SCE protocol - for anything other
> than FQ. If the primary SCE proponents had attempted to articulate a way
> to use SCE in a single queue or a dual queue, as you have, that would
> have taken my fire.
> 
> >
> > But regardless, the queue-building from classic ECN-capable endpoints
that
> > only get 1 congestion signal per RTT is what I understand as the main
> > downside of the tradeoff if we try to use ECN-capability as the dualq
> > classifier.  Does that match your understanding?
> This is indeed a major concern of mine (not as major as the starvation
> of SCE explained under Q4, but we'll come to that).
> 
> Fine-grained (DCTCP-like) and coarse-grained (Cubic-like) congestion
> controls need to be isolated, but I don't see how, unless their packets
> are tagged for separate queues. Without a specific fine/coarse
> identifier, we're left with having to re-use other identifiers:
> 
>   * You've tried to use ECN vs Not-ECN. But that still lumps two large
>     incompatible groups (fine ECN and coarse ECN) together.
>   * The only alternative that would serve this purpose is the flow
>     identifier at layer-4, because it isolates everything from
>     everything else. FQ is where SCE started, and that seems to be as
>     far as it can go.
> 
> Should we burn the last unicorn for a capability needed on
> "carrier-scale" boxes, but which requires FQ to work? Perhaps yes if
> there was no alternative. But there is: L4S.
> 
> That brings us neatly to the outstanding issues with L4S...
> 
> 
> >
> >
> > 2.
> > I ended up confused about how falling back works, and I didn't see it
> > spelled out anywhere.  I had assumed it was a persistent state-change
> > for the sender for the rest of the flow lifetime after detecting a
> > condition that required it, but then I saw some text that seemed to
> > indicate it might be temporary? From section 4.3 in l4s-id:
> >     "Note that a scalable congestion control is not expected to change
> >        to setting ECT(0) while it temporarily falls back to coexist with
> >        Reno ."
> >
> > Can you clarify whether the fall-back is meant to be temporary or not,
> > and whether I missed a more complete explanation of how it's supposed to
> > work?
> Firstly, as has been made clear in our latest talk/paper at Linux netdev
> and in my latest iccrg talk, currently TCP Prague only includes
> fall-back to Reno on loss. It does not do fall-back on classic ECN
> marking (yet). We're still working on RTT-independence and scaling to
> very low RTT (sub-MSS window) first.
> 
> Fall-back on loss is definitely very temporary: it does one large
> Reno-style window halving on a loss (ignoring any other losses in that
> RTT as Reno does), then immediately continues with DCTCP-style
> congestion avoidance driven by all the ECN marks (not just one per-RTT).
> 
> For classic ECN AQM detection, we only have initial design ideas.
> Olivier posted his design ideas here:
>  ??? https://protect2.fireeye.com/url?k=b73d693d-ebb7bdfc-b73d29a6-
> 863d9bcb726f-
> 062d85b471dea4c1&q=1&u=https%3A%2F%2Fgithub.com%2FL4STeam%2Ftcp-
> prague%2Fissues%2F2
> 
> I want to keep it simple (see response to Q4 about false negatives).
> Fall-back would be temporary, but last longer than for loss - until the
> flow next goes idle. Here's the simplest that I think might work:
>  ??? Starting X RTTs after first CE mark;??? // allows end of Slow Start
> to stabilize
>  ??? if (srtt > (min_rtt + Y) || rttvar > Z) {fallback()};
> Where X,Y&Z are TBD, dependent on experiments, but say X=5-6 RTT,
> Y=4-5ms & Z=dunno_without_measuring. The min_rtt could be taken only
> since the previous start-up or idle period (or perhaps the previous
> two). An idle would have to be defined as >3-4 RTT, to allow any
> self-induced queue to drain.
> 
> 
> The whole of L4S is experimental track. So others might take different
> approaches (e.g. BBRv2) and I'm sure our approach will evolve, which is
> why the requirement is worded liberally (it has to cover real-time, etc.
> not just TCP).
> 
> 
> >
> >
> > 3.
> > I also was a little confused on the implementation status of the
fallback
> > logic.  I was looking through some of the various links I could find,
and
> > I think these 2 are the main ones to consider? (from
> > https://protect2.fireeye.com/url?k=f4851fdd-a80fcb1c-f4855f46-
> 863d9bcb726f-
> db345c174f8a010a&q=1&u=https%3A%2F%2Friteproject.eu%2Fdctth%2F%23co
> de ):
> > - https://protect2.fireeye.com/url?k=58384351-04b29790-583803ca-
> 863d9bcb726f-
> 36bf3624b9b0808f&q=1&u=https%3A%2F%2Fgithub.com%2FL4STeam%2Fsch_d
> ualpi2_upstream
> > - https://protect2.fireeye.com/url?k=eeef7eb3-b265aa72-eeef3e28-
> 863d9bcb726f-
> 67813c5bf9b5fa59&q=1&u=https%3A%2F%2Fgithub.com%2FL4STeam%2Ftcp-
> prague
> >
> > It looks like the prague_fallback_to_ca case so far only happens when
> > AccECN is not negotiated, right?
> That's not the same sort of fall-back. That's fall-back because without
> AccECN there's only one ECN feedback signal per RTT, so it falls back to
> the configured classic congestion controller for the whole connection.
> Which controller depends on the parameter prague_ca_fallback which
> defaults to cubic.
> 
> As said above, fall-back on classic ECN has not yet been implemented in
> TCP Prague. Of the 3 things left on our list, it's the last 'cos we're
> waiting to see the results of measurements from a CDN, to see if there
> are any single queue classic ECN AQMs out there. If there aren't we
> would not plan to implement this requirement until there were. Whether
> others do is up to them of course.
> 
> 
> >
> > To me, the logic for when to do this (especially for rtt changes) seems
> > fairly complicated and easy to get wrong, especially if it's meant to be
> > temporary for the flow, or if needs to deal with things like network
path
> > changes unrelated to the bottleneck, or variations in rtt due an
endpoint
> > being a mobile device, or on wi-fi.
> >
> > Which brings me to:
> >
> >
> > *4.
> > (* I think this is the biggest point driving me to ask about this.)
> >
> > I'm pretty worried about mis-categorizing CE marking from classic AQM
> > algorithms as L4S-style markings, when using ECT(1) as the dualq
> > classifier.
> >
> > I did see this issue addressed in the l4s drafts, but reviewing it
> > left me a little confused, so I thought I'd ask about a point I
> > noticed for clarification:
> >
> >  From section 6.3.3 of l4s-arch:
> >     "an L4S sender will have to
> >     fall back to a classic ('TCP-Friendly') behaviour if it detects that
> >     ECN marking is accompanied by greater queuing delay or greater delay
> >     variation than would be expected with L4S"
> >
> >  From the abstract in l4s-arch:
> >     "In
> >     extensive testing the new L4S service keeps average queuing delay
> >     under a millisecond for _all_ applications even under very heavy
> >     load"
> >
> > My reading of these seems to suggest that if the sender can observe
> > a variance or increase of more than 1 millisecond of rtt, it should fall
> > back to classic ECN?
> >
> > I'm not sure yet how to square that with Section A.1.4 of l4s-id:
> >     "An increase in queuing delay or in delay variation would be
> >     a tell-tale sign, but it is not yet clear where a line would be
drawn
> >     between the two behaviours."
> >
> > Is the discrepancy here because the extensive testing (also mentioned in
> > the abstract of l4s-arch) was mainly in controlled environments, but the
> > internet is expected to introduce extra non-bottleneck delays even where
> > a dualq is present at the bottleneck, such as those from wi-fi, mobile
> > networks, and path changes?
> No, it's simply 'cos there is no implementation of this requirement yet.
> 
> >
> > Regardless, this seems to me like a worrisome gap in the spec, because
if
> > the claim that dualq will get deployed and enabled quickly and widely is
> > correct, it means this will be a common scenario in
deployment--basically
> > wherever there's existing classic AQMs deployed, especially since in CPE
> > devices the existing AQMs are generally configured to have a lower
> > bandwidth limit than the subscriber limit, so they'll (deliberately) be
> > the bottleneck whenever the upstream access network isn't overly
> > congested.
> I believe FQ-CoDel is the only AQM in CPE that I know of that supports
> classic ECN. In this case, an L4S-ECN congestion controller cannot
> starve a Cubic-ECN or Reno-ECN flow, cos the FQ scheduler controls their
> capacity shares.
> 
> The only other CPE AQM I am aware of is DOCSIS-PIE, which doesn't
> support ECN.
> 
> If the IETF assigns the ECT(1) codepoint to L4S, then it would be
> extremely easy to modify FQ-Codel to set a very shallow ECN threshold in
> any queue where at least one ECT(1) codepoint had been detected. This
> would work fine with highly transient flow queues.
> 
> >
> > I guess if it's really a 1-2 ms variance threshold to fall back, that
> > would probably address the safety concern, but it seems like it would
> > have a lot of false positives, and unnecessarily fall back on a lot of
> > flows.
> >
> > But worse, if there's some (not yet specified?) logic that tries to
reduce
> > those false positives by relaxing a simple very-few-ms threshold, it
seems
> > like there's a high likelihood of logic that produces false negatives
going
> > undetected.
> >
> > If that's the case, to me it seems like it will remain a significant
risk
> > even while TCP Prague has been deployed for quite a long time at a
sender,
> > as long as different endpoint and AQM implementations roll out randomly
> > behind different network conditions, for the various endpoints that end
> > up connected with the sender.
> I am less worried about this. I would be comfortable erring on the side
> of reducing false positives at the expense of false negatives.
> 
> Nonetheless, this position depends on what we find in measurement studies.
> * If we find no single-queue AQMs that do ECN-marking, it's a
> non-problem {Note 2}.
> * If such AQMs exist but are rare, they are likely to be in specific
> operator's networks, so there would be operator-specific ways to address
> such problems. E.g. if a CDN wanted to deploy the L4S experiment on its
> caches for that network, in collaboration with the network operator it
> could set a local-use DSCP instead of using ECT(1). That would still not
> deal with L4S traffic to/from the Internet, but the probability that
> different types of long-running flows coincide is low anyway, so the
> probability that different types of flows that are both long-running and
> non-CDN will coincide must surely be tiny.
> 
> >
> > It also seems to me there's a high likelihood of causing unsafe non-
> > responsive sender conditions in some of the cases where this kind of
false
> > negative happens in any kind of systematic way.
> This overstates the problem. There is no unresponsiveness. Even when two
> long-running flows coincide, an L4S flow does not actually starve a
> classic (e.g. Reno-ECN) TCP flow. They come to a balance that can be
> highly unequal in high BDP links, but never starvation or
> unresponsiveness. Indeed, as the link's BDP gets smaller, or the more
> flows there are, the more DCTCP & Reno-ECN tend to equality.
> 
> >
> > By contrast, as I understand it an SCE-based approach wouldn't need the
> > same kind of fallback state-change logic for the flow, since any CE
would
> > indicate a RFC 3168-style multiplicative decrease, and only ECT(1) would
> > indicate sub-loss congestion.
> I'm afraid you understand it wrong.
> 
> With the ECN-DualQ-SCE approach, any flows where the receiver does not
> feed back SCE (ECT(1)) markings starve any SCE (DCTCP-like) flows in the
> same bottleneck.
> 
> Similarly, any Reno-ECN or Cubic-ECN senders (i.e. without the logic to
> understand SCE) starve the SCE (DCTCP-like) flows in the same
> ECN-DualQ-SCE bottleneck.
> 
> And here, starve actually means starve. Not just come to a highly
> unbalanced equilibrium, but completely starve.
> 
> This is because a Cubic-ECN flow will keep pushing the queue up to the
> point where it emits CE markings, because it doesn't understand and
> therefore ignores the SCE markings. One queue can only have one length.
> So, because the Cubic flow(s) have pushed the queue past the shallower
> point where it starts to emit SCE markings, all packets not marked CE
> will be marked SCE.
> 
> For example, say Cubic flow(s) induce a fairly normal 0.5% CE marking
> (or 0.5% drop for non-ECN flows). Then there will be 99.5% SCE marking.
> 
> Then, the DCTCP-like flows designed to understand SCE will keep reducing
> in response to this saturated SCE marking and the Cubic flows will fill
> the space they leave and starve them.
> 
> We did experiments to try to minimize this starvation, with two AQMs in
> one queue where one type of CC ignores the signals from the lower
> threshold back in 2012. See:
>  ??? https://protect2.fireeye.com/url?k=47be1b4a-1b34cf8b-47be5bd1-
> 863d9bcb726f-
> 4c3e071522a975f6&q=1&u=http%3A%2F%2Fbobbriscoe.net%2Fpubs.html%23D
> CTCP-Internet
> This led us to realize we would have to use at least two queues.
> 
> >
> > This is one of the big advantages of the SCE-based approach in my mind,
> > since there's no chance of mis-classifying the meaning of a CE mark and
> > no need for a state change for how the sender handles the ECT backoff
logic
> > or sets the ECT markings.  (It just goes back to treating any CE as
RFC3168-
> > style loss equivalent, and SCE as a sub-loss signal.)
> >
> > Since an SCE-based approach would avoid this problem nicely, I consider
> > the reduced risk of false negatives (and unresponsive flows) here one of
the
> > important gains, to be weighed against the key downside mentioned in
> comment
> > #1.
> I hope you can see now that the ECN-DualQ-SCE approach suffers from the
> same problem as you are concerned about with L4S. Except the difference
> is it's not in 'legacy' non-SCE queues, but in the queue implementing
> SCE marking itself.
> 
> Unless one separates non-SCE traffic into a different queue, it starves
> SCE traffic.
> 
> >
> >
> > 5.
> > Something similar comes up again in some other places, for instance:
> >
> > from A.1.4 in l4s-id:
> (it's A.1.1.)
> >     "Description: A scalable congestion control needs to distinguish the
> >     packets it sends from those sent by classic congestion controls.
> >
> >     Motivation: It needs to be possible for a network node to classify
> >     L4S packets without flow state into a queue that applies an L4S ECN
> >     marking behaviour and isolates L4S packets from the queuing delay of
> >     classic packets."
> >
> > Listing this as a requirement seems to prioritize enabling the gains of
> > L4S ahead of avoiding the dangers of L4S flows failing to back off in
the
> > presence of possibly-miscategorized CE markings, if I'm reading it
right?
> > I guess Appendix A says these "requirements" are non-normative, but I'm
a
> > little concerned that framing it as a requirement instead of a design
> > choice with a tradeoff in its consequences is misleading here, and
> > pushes toward a less safe choice.
> As I hope you can now see from the last part of answer #4 that, if you
> try to classify ECN flows with fine-grained (DCTCP-like) and coarse
> (Cubic-like) congestion controls into the same queue (whether L4S or SCE
> marking), the Cubic-like congestion controls ruin it.
> 
> So I think this requirement stands. I've made a note-to-self to add the
> text: "To avoid having to use per-flow classification..." though.
> 
> >
> >
> > 6.
> > If queuing from classic ECN-capable flows is the main issue with using
> > ECT as the dualq classifier, do you think it would still be possible to
> > get the queuing delay down to a max of ~20-40ms right away for
ECN-capable
> > endpoints in networks that deploy this kind of dualq, and then hopefully
> > see it drop further to ~1-5ms as more endpoints get updated with AccECN
or
> > some kind of ECT(1) feedback and a scalable congestion controller that
> > can respond to SCE-style marking?
> Technically yes, but realistically no.
> 
> What I mean is, as I said from the start, if you remove the feature that
> deploying the L4S DualQ Coupled AQM gives very low and consistently very
> low latency straight away, then operators will lose interest in
> deploying it.
> 
> > Or is it your position that the additional gains from the ~1ms queueing
delay
> > that should be achievable from the beginning by using ECT(1) (in
connections
> > where enough of the key entities upgrade) are worth the risks?
> Well, I'd say "probably worth the risks", cos we're waiting for
> measurements to get a feel for whether any of the CE markings seen by
> the tests Apple reported in 2016-2017 are from single queue ECN AQMs.
> 
> See
> https://datatracker.ietf.org/meeting/104/materials/slides-104-iccrg-
> implementing-the-prague-requirements-in-tcp-for-l4s-01#page=11
> 
> >
> > (And if so, do you happen to have a pointer to any presentations or
papers
> > that made a quantitative comparison of the benefits from those 2
options?
> > I don't recall any offhand, but there's a lot of papers...)
> Latest results here (actually no different from results we reported in
> 2015 - all the changes to the code since have been non-performance
related):
> "DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM" Olga Albisser
> (Simula), Koen De Schepper (Nokia Bell-Labs), Bob Briscoe (Independent),
> Olivier Tilmans (Nokia Bell-Labs) and Henrik Steen (Simula), in Proc.
> Netdev 0x13
> <https://protect2.fireeye.com/url?k=ac0c7169-f086a5a8-ac0c31f2-
> 863d9bcb726f-
> 10f0079ea0780acc&q=1&u=https%3A%2F%2Fwww.netdevconf.org%2F0x13%2
> Fsession.html%3Ftalk-DUALPI2-AQM> (Mar 2019).
> 
> The paper via the netdev link shows qdelay, utilization, completion time
> efficiency, etc with the most extreme traffic load we use (2
> long-running flows plus 5X Web flows per sec, where X is each link rate
> in Mb/s, e.g. 600 flows/sec over the 120Mb/s link), for a full range of
> link rates, round trip times, etc.
> 
> The plots are pretty crammed, so if you'd prefer one example qdelay
> cumulative distribution function for the same extreme traffic load, see
> here:
> https://datatracker.ietf.org/meeting/104/materials/slides-104-iccrg-
> implementing-the-prague-requirements-in-tcp-for-l4s-01#page=22
> 
> If you want results from a range of less-extreme traffic models, just ask.
> 
> HTH
> 
> 
> 
> Bob
> 
> >
> >
> > Best regards,
> > Jake
> >
> >
> 
> {Note 1}: Or different server, client and network operators all agree to
> deploy, but let's assume that would be a bonus and not rely on it.
> 
> {Note 2}: Even where there are no single-queue AQMs now, there might be
> a concern that some could be enabled in future. Given study after study
> since ECN was first standardized (2001) have detected hardly any CE
> marks on the Internet until FQ-CoDel was deployed about 15 years later,
> the chance of those AQMs being turned on now is surely vanishing.
> 
> 
> 
> --
> ________________________________________________________________
> Bob Briscoe
https://protect2.fireeye.com/url?k=00e31d75-
> 5c69c9b4-00e35dee-863d9bcb726f-
> a188c700e74cbf8e&q=1&u=http%3A%2F%2Fbobbriscoe.net%2F
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://mailarchive.ietf.org/arch/browse/tsvwg/attachments/20190607/0fe50
> 0ab/attachment.html>
> 
> End of tsvwg Digest, Vol 182, Issue 6
> *************************************

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 14 Jun 2019 18:27:37 +0000

Hi Ingemar,
(bcc: ecn-sane, to keep them apprised on the discussion).

Thanks for chiming in on this.  A few comments inline:

> that part of the reason has been that ECN is not judged to give a large

Could you elaborate on this point?  

I haven't been sure how to think about the claims in the l4s drafts that
operators will deploy it rapidly because of performance.

Based on past analyses (e.g. the classic ECN rollout case study in RFC
8170 [1]), I thought network operators had a very "safety first" outlook on
these things, and that rapid deployment for performance benefits seemed
like wishful thinking.

But I'd be interested to know more about why that view might be mistaken.

> Besides this, L4S has the nice
> capacity increases.

I think section 3.4 of RFC 8257 says the rate increase would be the
same:
https://tools.ietf.org/html/rfc8257#section-3.4
   A DCTCP sender grows its congestion window in the same way as
   conventional TCP.

I guess this is referring to the paced chirping for rapid growth idea
presented last time?
https://datatracker.ietf.org/meeting/104/materials/slides-104-iccrg-implementing-the-prague-requirements-in-tcp-for-l4s-01#page=20

I'm a little unclear on how safe this can be made, but I agree it seems
useful if it can work well.

Do you think the L4S benefits will still be sufficient if this point
about faster growth doesn't hold up (and/or could be replicated regardless
of L4S), or is it critical to providing sufficient benefit in 3GPP?

(Note: I'm not taking a position on this point, just asking about how
much this point matters to the 3GPP support, as you see it.)

> require low latency video streaming.

Remote control over the internet instead of a direct radio link is an
interesting use case.  Do you happen to know the research about delay
parameters that make the difference between viable or not viable for
RC?

This touches on one of the reasons I've been skeptical that the benefits
will drive a rapid deployment--in most of the use cases I've come up with,
it seems like reducing delay from ~200-500ms down to ~15-30ms (as seems
achievable even for single queue with classic AQM) would give almost
all the same benefits as reducing from ~15-30ms down to 1ms.

Of course, there's a difference in that last 14-29ms, but for instance
for gaming reaction time it's well under the thresholds that make a
difference for humans (the low end of which is at 45ms, according to
[2]), so it seems like the value in that market would be captured by
classic ECN, and therefore since classic ECN deployment hasn't caught
on yet, I had to conclude that the performance gains to enable that
market aren't sufficient to drive wide adoption.

So I'm curious to know more about the use cases that get over that
hump from an operator's point of view, and what you've seen that leads
you to believe the additional gains of L4S from will make the difference
on those use cases where classic ECN wasn't adequate.

> harder to sell in to 3GPP.   

Thanks, this is good to know.

To me one benefit of SCE over L4S is that it seems safer to avoid
relying on an ambiguous signal (namely a CE that we don't know which
kind of AQM set it) in a control system, while still providing high-
fidelity info about the network device congestion, where available.

I agree that it's not completely clear exactly how the congestion
controllers can capitalize on that info, but to me it still seems worth
considering.

So although I'll support L4S if it really covers all the safety issues
and performs better, I'd be more comfortable with the signaling if
there's a way to make SCE do the same job, especially if the endpoint
implementation is simpler to get robustly deployed.

So really, I'm hoping for a bakeoff to decide this, because one of my
concerns is that L4S still doesn't have an implementation that does
all the things the drafts say are needed for safety on the internet,
even though the initial proof of concept demoing the performance
gains was presented 7 years ago.  It's good that it's getting closer,
but the long implementation cycle (which still doesn't have all the
features required by the drafts) is a concern for me from the
"running code" point of view.

On this point of view, it's possible that a parallel track might get
further faster, especially if it doesn't need the same special cases
to be safe, which is part of why I've been tentatively supportive.

And although I can see how the queue classification is a major issue
that could make the difference, especially with the very promising
dualq proposal, it also seems true that in addition to CPEs, there are
promising avenues for carrier-scale FQ systems (e.g [3], [4]) that could
solve that.  It makes me think that even if SCE only gets low-latency
with FQ and otherwise causes no harm, it's not clear it'll be a slower
path to ubiquitous deployment (and by the way, this approach also would
handle the opt-in access control problem).

Of course, this will presumably collapse to one answer at some point,
but I'll argue that it's worthwhile to give a good look to the alternate
proposal...

Anyway, thanks for the comments, I think it's good to see more
discussion on this.

Best regards,
Jake

[1] Appendix A.1 RFC 8170 https://tools.ietf.org/html/rfc8170#appendix-A.1
[2] https://ojs.bibsys.no/index.php/NIK/article/view/9 says 45ms
[3] http://ppv.elte.hu/
[4] https://ieeexplore.ieee.org/document/8419697

---

From: Dave Taht <[email-masked]>
Date: Sat, 15 Jun 2019 10:44:35 -0700

> > require low latency video streaming.

This is a poor example of a use case for l4s. The link rates in these
cases are *wildly variable*.

- in the case of a realtime command and control video stream on cars,
trucks, drones, uav or rockets, you *want* packet loss, you want the
most current frame or fragments thereof as closely tied to actual
event as possible. You want to drop from the head, not the tail of the
queue, when the rate changes, so the most
current data is available to the operator.

See any launch video from spacex for this.

You also want backpressure or preferably an out of band signal to the
encoder (if you have one) so it
drops the encoding resolution down to the data rate ASAP, if you
exceed the capacity, while holding the video frame rate steady.

And you want to be running well below the presently observed capacity
of the link at all times, because it's precisely when things are going
south that you want the most data, especially if you are also
multiplexing control traffic over the same link.

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Dave Taht <[email-masked]>
Date: Sat, 15 Jun 2019 10:54:25 -0700

> > > I see many applications that can benefit greatly from L4S, besides AR/VR,

I over reacted to the second part of this sentence. AR/VR - assuming
it's over a reliable short RTT isochronous transport, can benefit from
 L4S-style techniques.

> > > require low latency video streaming.

but not this.

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 16 Jun 2019 10:07:45 +0000

Hi Jake + all

Please see inline

/Ingemar

> -----Original Message-----
> From: Holland, Jake <[email-masked]>
> Sent: den 14 juni 2019 20:28
> To: Ingemar Johansson S <[email-masked]>; Bob Briscoe
> <[email-masked]>
> Cc: [email-masked]
> On 2019-06-08, 12:46, "Ingemar Johansson S"
> <[email-masked]> wrote:
> > Up until now it has been quite a challenge to make ECN happen, I
> > believe that part of the reason has been that ECN is not judged to
> > give a large enough gain.
> 
> I haven't been sure how to think about the claims in the l4s drafts that operators
> will deploy it rapidly because of performance.
> 
> 8170 [1]), I thought network operators had a very "safety first" outlook on these
> things, and that rapid deployment for performance benefits seemed like wishful
> thinking.
> 
[IJ] I believe that it is easy to end up in a lot of speculation. I don't believe that the safety first thinking makes much sense, yes it has sometimes been used as a counter argument. Part of the problem is perhaps that ECN was introduced into 3GPP for VoLTE. And then when ECN is proposed for its original use in 3GPP (=Generic transport protocol agnostic feature) it gets hard to make it stick. With that said, ECN is supported in both LTE and NR standards (TS36.300, TS38.300). It is however rarely deployed. One could speculate around the reasons, I believe that one big reason can be that traditional ECN does not show a large enough delta improvement to make it worthwhile. I can of course be wrong , I don't possess a crystal ball 😊

> > property that it has potential to allow for faster rate increase when
> > link capacity increases.
> 
> I guess this is referring to the paced chirping for rapid growth idea presented last
> time?
> implementing-the-prague-requirements-in-tcp-for-l4s-01#page=20
> 
> I'm a little unclear on how safe this can be made, but I agree it seems useful if it
> can work well.
[IJ] Yes DCTCP use traditional additive increase. I have personally done a few experiments in this area, nothing that is good enough to show as the experiments were very limited. One possible idea can be to make the bandwidth probing in BBR(v2) more aggressive. And there may also be possibilities with Paced chirping too

> 
> Do you think the L4S benefits will still be sufficient if this point about faster
> growth doesn't hold up (and/or could be replicated regardless of L4S), or is it
> critical to providing sufficient benefit in 3GPP?
[IJ] No, I don't believe that it is critical, it is definitely a welcome bonus if it is possible.

> 
> (Note: I'm not taking a position on this point, just asking about how much this
> point matters to the 3GPP support, as you see it.)
> 
> > I see many applications that can benefit greatly from L4S, besides
> > AR/VR, there is also an increased interest in the deployment of remote
> > control capabilities for vehicles such as cars, trucks and drones, all
> > of which require low latency video streaming.
> 
> Remote control over the internet instead of a direct radio link is an interesting
> use case.  Do you happen to know the research about delay parameters that
> make the difference between viable or not viable for RC?

> 
> This touches on one of the reasons I've been skeptical that the benefits will drive
> a rapid deployment--in most of the use cases I've come up with, it seems like
> reducing delay from ~200-500ms down to ~15-30ms (as seems achievable even
> for single queue with classic AQM) would give almost all the same benefits as
> reducing from ~15-30ms down to 1ms.
[IJ] The thing I like with L4S is that it reduces standing queues down to almost zero, which gives a very fast reaction time when throughput drops. In addition L4S gives frequent signals of congestion, which makes it easier for a congestion control algorithm to know when it is close to the congestion knee.

> 
> Of course, there's a difference in that last 14-29ms, but for instance for gaming
> reaction time it's well under the thresholds that make a difference for humans
> (the low end of which is at 45ms, according to [2]), so it seems like the value in
> that market would be captured by classic ECN, and therefore since classic ECN
> deployment hasn't caught on yet, I had to conclude that the performance gains
> to enable that market aren't sufficient to drive wide adoption.
> 
> So I'm curious to know more about the use cases that get over that hump from
> an operator's point of view, and what you've seen that leads you to believe the
> additional gains of L4S from will make the difference on those use cases where
> classic ECN wasn't adequate.
[IJ] I guess for this part, there need to be more input from operators

> 
> > My bottomline is that I believe L4S provides with a clear benefit that
> > is large enough to be more widely accepted in 3GPP. SCE is as I see it
> > more like something that is just a minor enhancement to ECN and is therefore
> much
> To me one benefit of SCE over L4S is that it seems safer to avoid relying on an
> ambiguous signal (namely a CE that we don't know which kind of AQM set it) in a
> control system, while still providing high- fidelity info about the network device
> congestion, where available.
> 
> I agree that it's not completely clear exactly how the congestion controllers can
> capitalize on that info, but to me it still seems worth considering.
> 
> So although I'll support L4S if it really covers all the safety issues and performs
> better, I'd be more comfortable with the signaling if there's a way to make SCE
> do the same job, especially if the endpoint implementation is simpler to get
> robustly deployed.
> 
> So really, I'm hoping for a bakeoff to decide this, because one of my concerns is
> that L4S still doesn't have an implementation that does all the things the drafts
> say are needed for safety on the internet, even though the initial proof of
> concept demoing the performance gains was presented 7 years ago.  It's good
> that it's getting closer, but the long implementation cycle (which still doesn't
> have all the features required by the drafts) is a concern for me from the
> On this point of view, it's possible that a parallel track might get further faster,
> especially if it doesn't need the same special cases to be safe, which is part of
> why I've been tentatively supportive.
> 
> And although I can see how the queue classification is a major issue that could
> make the difference, especially with the very promising dualq proposal, it also
> seems true that in addition to CPEs, there are promising avenues for carrier-
> scale FQ systems (e.g [3], [4]) that could solve that.  It makes me think that even
> if SCE only gets low-latency with FQ and otherwise causes no harm, it's not clear
> it'll be a slower path to ubiquitous deployment (and by the way, this approach
> also would handle the opt-in access control problem).
> 
> Of course, this will presumably collapse to one answer at some point, but I'll
> argue that it's worthwhile to give a good look to the alternate proposal...
> 
> Anyway, thanks for the comments, I think it's good to see more discussion on
> this.
> 
> Best regards,
> Jake
> 
> [2] https://protect2.fireeye.com/url?k=997ee527-c5f43093-997ea5bc-
> 866a015dd3d5-
> 1d25c70963170b1e&q=1&u=https%3A%2F%2Fojs.bibsys.no%2Findex.php%2FNI
> K%2Farticle%2Fview%2F9 says 45ms [3] http://ppv.elte.hu/ [4]
> https://ieeexplore.ieee.org/document/8419697
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 16 Jun 2019 12:18:59 +0000

Hi Dave + all

Please see inline [IJ]

/Ingemar

> From: Dave Taht <[email-masked]>
> Sent: den 15 juni 2019 19:54
> To: Holland, Jake <[email-masked]>
> Cc: Ingemar Johansson S <[email-masked]>; Bob Briscoe
> <[email-masked]>; [email-masked]
> Subject: Re: [Ecn-sane] [tsvwg] Comments on L4S drafts
> 
> > > > AR/VR,
> I over reacted to the second part of this sentence. AR/VR - assuming it's over a
> reliable short RTT isochronous transport, can benefit from  L4S-style techniques.
> 
> > > > there is also an increased interest in the deployment of remote
> > > > control capabilities for vehicles such as cars, trucks and drones,
> > > > all of which require low latency video streaming.
> 
> but not this.
> 
> > queue, when the rate changes, so the most current data is available to
> > the operator.
[IJ] Yes, of course. That is the whole intention with the congestion control algorithms devised within the charter of the RMCAT WG, SCReAM (RFC8298) is one of them. But simply head dropping is not good enough for video as this would distort the video.

> > encoder (if you have one) so it drops the encoding resolution down to
> > the data rate ASAP, if you exceed the capacity, while holding the
> > video frame rate steady.
[IJ] Yes, also in the scope of RMCAT. The main challenge here is that run of the mill video coders like omxh264enc (available in the NVIDIA Jetson Nano) take some 300-500ms to reduce the rate. For that reason SCReAM has an RTP buffer on the sender side that can swallow RTP packets temporarly if the encoder output rate is higher than the target rate. The whole RTP queue is discarded if the delay becomes too high, with omxh264enc one can enforce a new IDR frame at the same time. This gives a very short glitch in the video. With that said.. video encoders behave differently, some are better than others.

> > multiplexing control traffic over the same link.
[IJ] In the best of worlds you'd want to run exactly at or below the observed capacity but it is not always possible and perhaps not always needed, larger drops in link capacity can be solved with the RTP queue discard + new IDR outlined below. L4S can reduced network jitter which is welcome as network jitter leads to larger e2e delay as dejitterbuffers need to use a larger headroom to avoid choppy playout of the video. The link below tries to illustrate this a bit more 
https://github.com/EricssonResearch/scream#ecn-explicit-congestion-notification

> >
> > --
> >
> > Dave Täht
> > CTO, TekLibre, LLC
> > https://protect2.fireeye.com/url?k=f7d03840-ab0430e9-f7d078db-865b3b1e
> > 120b-b0476ab97f0a5e2a&q=1&u=http%3A%2F%2Fwww.teklibre.com%2F
> > Tel: 1-831-205-9740
> 
> 
> 
> --
> 
> Dave Täht
> CTO, TekLibre, LLC
> https://protect2.fireeye.com/url?k=8920c762-d5f4cfcb-892087f9-
> 865b3b1e120b-
> 71c0ea6118599683&q=1&u=http%3A%2F%2Fwww.teklibre.com%2F
> Tel: 1-831-205-9740

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 19 Jun 2019 13:59:07 +0100

Jake & Ingemar,

On 16/06/2019 11:07, Ingemar Johansson S wrote:
>> From: Holland, Jake<[email-masked]>
>> To: Ingemar Johansson S<[email-masked]>; Bob Briscoe
>> <[email-masked]>
>> Cc:[email-masked]
[BB] The ECN rollout case study in rfc8170 is not a useful example . It 
ends hoping there will be some client roll-out (written before Apple's 
decision) and doesn't even mention that network roll-out would still be 
needed subsequently. So it gives no insight into what causes network 
operator resistance
> [IJ] I believe that it is easy to end up in a lot of speculation. I don't believe that the safety first thinking makes much sense, yes it has sometimes been used as a counter argument. Part of the…
[BB] My experience on this comes from years inside BT. The last 15 were 
after ECN was standardized, and for the last few years I was in BT's 
tech strategy team, regularly making business cases for various 
improvements. And talking with folks from other operators, of course.

When I quantified the performance benefit of classic ECN, it was 
embarrassing. You only got significant benefits in an under-provisioned 
network, which most operators avoid for obvious other reasons. Classic 
ECN gave next-to-no benefit with long-running flows. The more 
significant benefit for short transactional flows was primarily due to 
avoiding the timeout when the last packet of a flow was dropped. I 
figured that could be solved e2e, and indeed in 2012 the tail loss probe 
was proposed to solve that problem. The remaining benefit was mostly due 
to not losing SYNs and to a lesser extent SYN/ACKs, but classic ECN 
couldn't be used on SYNs anyway. In comparison the potential risks on 
the cost side dominated.

Finally, for a large network improvement project it is nearly impossible 
to squeeze the cash needed out of the relatively small budgets assigned 
for regular network improvements. None of the access equipment supported 
a modern AQM or ECN, so we would have had to tender for new designs. To 
persuade vendors to spend that sort of money, you need a budget line in 
a project that is buying kit for a new service with a projected revenue 
stream (e.g. a new sports service, a VR product, etc). That means your 
performance improvement has to be necessary for that product.

An alternative would have beeen to show that the performance improvement 
would gain sales from competing ISPs for long enough to pay for the 
costs of the improvement, but that's much harder to argue convincingly.

> [IJ] Yes DCTCP use traditional additive increase. I have personally done a few experiments in this area, nothing that is good enough to show as the experiments were very limited. One possible idea…
[BB] Note that paced chirping is not the differentiator here. It doesn't 
depend on ECN, nor L4S-ECN, nor SCE-ECN for that matter. It is 
delay-based, and potentially applicable to any e2e technology.

The differentiator that L4S provides (and perhaps SCE if all the 
problems were fixed) is the introduction of scalable congestion control 
(like DCTCP), which induces a frequent amount of signalling per RTT that 
remains invariant as flow rate scales.

Support for a transition to scalable CC is as important as cutting 
latency. Aside from being able to scale flow rate indefinitely,...

...it also solves the problem of rapidly detecting when more capacity 
has become available. If you normally get 2 signals per RTT (like 
DCTCP), you can tell there's available capacity after 2 or 3 RTT. If you 
get 1 signal every few hundred RTTs (like Cubic), you cannot tell 
there's available capacity for a thousand or so RTTs. That in itself is 
useful.... You don't need to do the seeking with paced chirping, which 
is just one attempt to get up to capacity both with less overshoot and 
faster.

>> point matters to the 3GPP support, as you see it.)
[BB] Ingemar has described the New Radio meetings to me where he's tried 
to propose ECN in the RLC layer. Like the swathes of other proposals, he 
was given 2 minutes, to persuade primarily radio people, who have 
already seen all the work that went into ECN for VoLTE not being taken up.

5G has promised extremely low latency. It is currently planning to do 
that with 'old school' QoS - by limiting throughput into reserved 
capacity. But that doesn't scale to apps that want high bandwidth and 
low latency. That's when the NR working group will start listening more 
carefully to ECN-based solutions.

> [IJ] The thing I like with L4S is that it reduces standing queues down to almost zero, which gives a very fast reaction time when throughput drops. In addition L4S gives frequent signals of congest…
[BB] I did some research on motion-to-photon latency a while ago, with 
others. It was for VR/AR, but it translates to similar apps. Quoting:

    MTP Latency:  AR/VR developers generally agree that MTP latency
       becomes imperceptible below about 20 ms [Carmack13  <https://tools.ietf.org/html/draft-han-iccrg-arvr-transport-problem-01#ref-Carmack13>].  However,
       some research has concluded that MTP latency must be less than
       17ms for sensitive users [MTP-Latency-NASA  <https://tools.ietf.org/html/draft-han-iccrg-arvr-transport-problem-01#ref-MTP-Latency-NASA>].  Experience has shown
       that standards bodies tend to set demanding quality levels, while
       motivated humans often happily adapt to lower quality although
       they struggle with more demanding tasks.  Therefore, we must be
       clear that this 20 ms requirement is designed to enable immersive
       interaction for the same wide range of tasks that people are used
       to undertaking locally.
...
    For a summary of numerous references
    concerning the limit of human perception of delay see the thesis of
    Raaen [Raaen16  <https://tools.ietf.org/html/draft-han-iccrg-arvr-transport-problem-01#ref-Raaen16>].

Let's say 20ms is too pedantic and you've got 50ms round trip MTP budget 
(John Carmack says that 50ms feels responsive, but the slight lag is 
still subtly unnatural, and merely defers the onset of VR-sickness).

We projected [latency budget 
<https://tools.ietf.org/html/draft-han-iccrg-arvr-transport-problem-01#appendix-A.1.2>] 
that, with some expected advances, it should possible to get the total 
of all delays except propagation and queuing down to about 13ms.

If one subtracts the delays you just stated for queuing, you get the 
following left for propagation:

	target for 'responsive' 	non-network
	queuing
	left for 2-way propagation
	reach in fibre
2nd gen. AQM 	50ms
	- 13ms
	- 30ms
	= 7ms
	700km (440miles)
3rd gen. AQM 	50ms
	- 13ms
	- 1ms
	= 36ms
	3600km (2250miles)

5 times greater reach means responsive interaction between Los Angeles 
and Atlanta, rather than just Los Angeles and Phoenix.

For communicating with a data centre, 5 times greater reach means 
equivalent coverage from 25 times fewer sites (coverage area is the 
square of reach). Concentration of sites is surely a very important cost 
factor for a CDN.

Note that, for real-time comms you need to watch the 99 or 99.9 
percentile, not just median. See these percentiles on a log-scale at 
slide 24 here:
https://www.files.netdevconf.org/f/4ebdcdd6f94547ad8b77/?dl=1
This was under rather extreme load (600 web sessions per second - see 
slide for details).

Whatever, @Jake, I think you will agree that SCE's aim is to cut queuing 
to similarly low levels. So arguing that we don't need such low delay 
also argues against SCE.

> [IJ] I guess for this part, there need to be more input from operators
[BB] When Kjetil [2] says 45ms is good enough for today's games, I'd 
trust that. But you can't burn all that with queuing - if I had aimed 
for 45ms not 50 ms above, I'd have been left with 2ms for propagation.

When I showed Kjetil the demo of L4S using finger-gestures to pan and 
zoom cloud-rendered video, he agreed that humans are much more sensitive 
to the lag that the eye sees between their real hand controlling a 
movement and seeing the thing move under their hand. It depends how much 
freedom we want to give game developers to explore new user interfaces 
and delivery platforms (e.g. a Wii interacting with cloud-rendering).

>> do the same job, especially if the endpoint implementation is simpler to get
[BB] How much is this a case of, "There aren't any problems with the SCE 
endpoint because we haven't thought about the problems yet"?

As well as the straightforward engineering showstoppers that I have 
highlighted (which I'll repeat 1-by-1 in later emails), there's also 
algorithmic stuff that hasn't even been identified yet in SCE, let alone 
addressed theoretically, let alone implemented.

For instance, a shift to fine-grained signals also shifts the smoothing 
from the network to the sender. That means the sender has to smooth the 
SCE signal and not the CE. So you have to deal with the cases where the 
two controllers interact and one overtakes the other. I don't believe 
stability is understood in such a system (you can be pessimistic when 
slowing down, but you also have to ensure stability when speeding up).

It's not as if SCE can just ride on the back of the CC research we and 
others have already done - it also introduces its own new research 
problems.
>> concept demoing the performance gains was presented 7 years ago.
[BB] It was Jul 2015 (nearly 4 years ago, not 7).
>> It's good
[BB] The SCE endpoint will need all the features required by the drafts 
as well. Unless it is going to solely require FQ.

I should also add that we (the L4S proponents) never envisaged that we 
would have to do all the endpoint stuff. We were all from 
network-focused companies. Altho we all had background in congestion 
control for video, we weren't allowed/expected to do such work on 
company time.

What we didn't realize was that researchers aren't getting funding to do 
such work these days (those that haven't been collected by Google). So 
we eventually had to grasp the nettle and find ways to do the endpoint 
stuff ourselves.

For instance, on a personal note, CableLabs is only funding a fraction 
of my working week, and will only pay for time on tasks in my contract, 
which are nearly all about the network aspects. I am self-funding nearly 
all work I do on end-system stuff.

>> why I've been tentatively supportive.
[BB] Let me first see if I can get the SCE proponents to address the 
show-stoppers that I have highlighted. By remaining silent, they seem to 
have convinced everyone that these show-stoppers don't exist.

If necessary, it sounds like it would help to address the only 
outstanding concern with L4S (classic ECN fall-back), irrespective of 
whether we think the problem actually exists or will ever exist.

>> also would handle the opt-in access control problem).
[BB] You (@Jake) are right to point out that different people have 
different ideas of what they think might happen in the future. However, 
I think it is a bit of a stretch to imagine that ubiquitous deployment 
of FQ might happen...

FQ assumes L4 headers are accessible, which assumes the Internet is an 
unencrypted L3 network. In 4G and 5G the eNodeB or gNodeB where ECN 
would need to be marked is a L2 node. A node deeper into the network has 
already compressed, tunnelled and encapsulated the IP headers. So how 
would FQ here access L4 port numbers? it can't do the cake trick - 
creating an artificial bottleneck where the IP header is accessible, 
because this concerns radio capacity, which varies hugely and continually.

Not to mention... all my other unanswered points about where SCE doesn't 
work at all, e.g.

  * all tunnels will have to propagate the ECT(1) codepoint, when the
    spec saying this isn't even out of WGLC yet,
  * and the optional TCP option for AccECN will be needed to feed back
    ECT(1), when no major OS is going to implement the TCP option,
    because they don't want to handle all the pain of middlebox mangling,
  * and... <my other 3 points that I'll get to in later emails>

If the last unicorn goes to a solution that will rarely work, and 
becomes renowned as ineffective and unconvincing, we will have wasted 
the last unicorn.

>> Anyway, thanks for the comments, I think it's good to see more discussion on
[BB]  Having alternative(s) is v important, even if strawmen. Proper 
discussion is good too - I've been close enough to this that I can 
identify problems v quickly, but the wider community needs discussion 
time to get steeped in it all.

Thank you v much for all the time you're putting into this.
Cheers

Bob
>>
>> Best regards,
>> Jake
>>
>> [1] Appendix A.1 RFC 8170https://tools.ietf.org/html/rfc8170#appendix-A.1
>> [2]https://protect2.fireeye.com/url?k=997ee527-c5f43093-997ea5bc-
>> 866a015dd3d5-
>> K%2Farticle%2Fview%2F9 says 45ms [3]http://ppv.elte.hu/  [4]

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/