Subject: [tsvwg] Comments on L4S drafts

From: "Holland, Jake" <[email-masked]>
Date: Wed, 5 Jun 2019 00:01:27 +0000

Hi Bob,

I have a few comments and questions on draft-ietf-tsvwg-ecn-l4s-id-06
and draft-ietf-tsvwg-l4s-arch-03.

I've been re-reading these with an eye toward whether it would be
feasible to make L4S compatible with SCE[1] by using ECN capability alone
as the dualq classifier (roughly as described in Appendix B.3 of l4s-id),
and using ECT(1) to indicate a sub-loss congestion signal, assuming
some reasonable mechanism for reflecting the ECT(1) signals to sender
(such as AccECN in TCP, or even just reflecting each SCE signal in the
NS bit from receiver, if AccECN is un-negotiated).

I'm trying to understand the impact this approach would have on the
overall L4S architecture, and I thought I'd write out some of the
comments and questions that taking this angle on a review has left me
with.

This approach of course would require some minor updates to DCTCP or other
CCs that hope to make use of the sub-loss signal, but the changes seem
relatively straightforward (I believe there's a preliminary
implementation that was able to achieve similarly reduced RTT in lab) and
the idea of course comes with some tradeoffs--I've tried to articulate the
key ones I noticed below, which I think are mostly already stated in the
l4s drafts, but I thought I'd ask your opinion of whether you agree with
this interpretation of what these tradeoffs would look like, or there
are other important points you'd like to mention for consideration.

1.
Of course, I understand using SCE-style signaling with ECT capability as
the dualq classifier would come with a cost that where there's classic ECT
behavior at endpoints, the low latency queue would routinely get some
queue-building, until there's pretty wide deployment of scalable controllers
and feedback for the congestion signals at the endpoints.

This is a downside for the proposal, but of course even under this downside,
there's the gains described in Section 5.2 of l4s-arch:
   "State-of-the-art AQMs:  AQMs such as PIE and fq_CoDel give a
      significant reduction in queuing delay relative to no AQM at all."

On top of that, the same pressures that l4s-arch describes that should
cause rapid rollout of L4S should for the same reasons cause rapid rollout
of the endpoint capabilities, especially if the network capability is
there.

But regardless, the queue-building from classic ECN-capable endpoints that
only get 1 congestion signal per RTT is what I understand as the main
downside of the tradeoff if we try to use ECN-capability as the dualq
classifier.  Does that match your understanding?

2.
I ended up confused about how falling back works, and I didn't see it
spelled out anywhere.  I had assumed it was a persistent state-change
for the sender for the rest of the flow lifetime after detecting a
condition that required it, but then I saw some text that seemed to
indicate it might be temporary? From section 4.3 in l4s-id:
   "Note that a scalable congestion control is not expected to change
      to setting ECT(0) while it temporarily falls back to coexist with
      Reno ."

Can you clarify whether the fall-back is meant to be temporary or not,
and whether I missed a more complete explanation of how it's supposed to
work?

3.
I also was a little confused on the implementation status of the fallback
logic.  I was looking through some of the various links I could find, and
I think these 2 are the main ones to consider? (from
https://riteproject.eu/dctth/#code ):
- https://github.com/L4STeam/sch_dualpi2_upstream
- https://github.com/L4STeam/tcp-prague

It looks like the prague_fallback_to_ca case so far only happens when
AccECN is not negotiated, right?

To me, the logic for when to do this (especially for rtt changes) seems
fairly complicated and easy to get wrong, especially if it's meant to be
temporary for the flow, or if needs to deal with things like network path
changes unrelated to the bottleneck, or variations in rtt due an endpoint
being a mobile device, or on wi-fi.

Which brings me to:

*4.
(* I think this is the biggest point driving me to ask about this.)

I'm pretty worried about mis-categorizing CE marking from classic AQM
algorithms as L4S-style markings, when using ECT(1) as the dualq
classifier.

I did see this issue addressed in the l4s drafts, but reviewing it
left me a little confused, so I thought I'd ask about a point I
noticed for clarification:

From section 6.3.3 of l4s-arch:
   "an L4S sender will have to
   fall back to a classic ('TCP-Friendly') behaviour if it detects that
   ECN marking is accompanied by greater queuing delay or greater delay
   variation than would be expected with L4S"

From the abstract in l4s-arch:
   "In
   extensive testing the new L4S service keeps average queuing delay
   under a millisecond for _all_ applications even under very heavy
   load"

My reading of these seems to suggest that if the sender can observe
a variance or increase of more than 1 millisecond of rtt, it should fall
back to classic ECN?

I'm not sure yet how to square that with Section A.1.4 of l4s-id:
   "An increase in queuing delay or in delay variation would be
   a tell-tale sign, but it is not yet clear where a line would be drawn
   between the two behaviours."

Is the discrepancy here because the extensive testing (also mentioned in
the abstract of l4s-arch) was mainly in controlled environments, but the
internet is expected to introduce extra non-bottleneck delays even where
a dualq is present at the bottleneck, such as those from wi-fi, mobile
networks, and path changes?

Regardless, this seems to me like a worrisome gap in the spec, because if
the claim that dualq will get deployed and enabled quickly and widely is
correct, it means this will be a common scenario in deployment--basically
wherever there's existing classic AQMs deployed, especially since in CPE
devices the existing AQMs are generally configured to have a lower
bandwidth limit than the subscriber limit, so they'll (deliberately) be
the bottleneck whenever the upstream access network isn't overly
congested.

I guess if it's really a 1-2 ms variance threshold to fall back, that
would probably address the safety concern, but it seems like it would
have a lot of false positives, and unnecessarily fall back on a lot of
flows.

But worse, if there's some (not yet specified?) logic that tries to reduce
those false positives by relaxing a simple very-few-ms threshold, it seems
like there's a high likelihood of logic that produces false negatives going
undetected.

If that's the case, to me it seems like it will remain a significant risk
even while TCP Prague has been deployed for quite a long time at a sender,
as long as different endpoint and AQM implementations roll out randomly
behind different network conditions, for the various endpoints that end
up connected with the sender.

It also seems to me there's a high likelihood of causing unsafe non-
responsive sender conditions in some of the cases where this kind of false
negative happens in any kind of systematic way.

By contrast, as I understand it an SCE-based approach wouldn't need the
same kind of fallback state-change logic for the flow, since any CE would
indicate a RFC 3168-style multiplicative decrease, and only ECT(1) would
indicate sub-loss congestion.

This is one of the big advantages of the SCE-based approach in my mind,
since there's no chance of mis-classifying the meaning of a CE mark and
no need for a state change for how the sender handles the ECT backoff logic
or sets the ECT markings.  (It just goes back to treating any CE as RFC3168-
style loss equivalent, and SCE as a sub-loss signal.)

Since an SCE-based approach would avoid this problem nicely, I consider
the reduced risk of false negatives (and unresponsive flows) here one of the
important gains, to be weighed against the key downside mentioned in comment
#1.

5.
Something similar comes up again in some other places, for instance:

from A.1.4 in l4s-id:
   "Description: A scalable congestion control needs to distinguish the
   packets it sends from those sent by classic congestion controls.

   Motivation: It needs to be possible for a network node to classify
   L4S packets without flow state into a queue that applies an L4S ECN
   marking behaviour and isolates L4S packets from the queuing delay of
   classic packets."

Listing this as a requirement seems to prioritize enabling the gains of
L4S ahead of avoiding the dangers of L4S flows failing to back off in the
presence of possibly-miscategorized CE markings, if I'm reading it right?

I guess Appendix A says these "requirements" are non-normative, but I'm a
little concerned that framing it as a requirement instead of a design
choice with a tradeoff in its consequences is misleading here, and
pushes toward a less safe choice.

6.
If queuing from classic ECN-capable flows is the main issue with using
ECT as the dualq classifier, do you think it would still be possible to
get the queuing delay down to a max of ~20-40ms right away for ECN-capable
endpoints in networks that deploy this kind of dualq, and then hopefully
see it drop further to ~1-5ms as more endpoints get updated with AccECN or
some kind of ECT(1) feedback and a scalable congestion controller that
can respond to SCE-style marking?

Or is it your position that the additional gains from the ~1ms queueing delay
that should be achievable from the beginning by using ECT(1) (in connections
where enough of the key entities upgrade) are worth the risks?

(And if so, do you happen to have a pointer to any presentations or papers
that made a quantitative comparison of the benefits from those 2 options?
I don't recall any offhand, but there's a lot of papers...)

Best regards,
Jake

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 Jun 2019 19:07:53 +0100

Thanks Jake,

I'll address each of your questions inline. But I notice that I need to 
lay down some context first.

The problem boils down to deployment incentives. The introduction of 
fine-grained congestion control requires changes to sender, receiver and 
at least the bottleneck link before it is effective. ECN deployment 
faced the same 3-part deployment problem. So we tried hard to learn from 
it.

Faced with a 3-part deployment, no single party makes a move unless they 
judge that the potential gain is worth the effort and that /all/ the 
other parts (server, client, network) are strongly likely to make the 
same judgement {Note 1}.

The effort isn't just the coding, it's all the hassle dealing with 
unexpected consequences of making the change, e.g. the risk of people's 
Internet service being taken out by a middlebox black-holing the new 
protocol. High risk of high cost/effort needs very high gain.

So the improvement has to be remarkable. Not just incremental, but 
stunning enough to enable applications that are not even possible 
otherwise.

The aim here is to use the last unicorn in the world (ECT(1)) to the 
full. If we don't make delay extremely low and extremely consistent 
we'll have wasted it. So we must focus on 99th percentile delay (and 
more 9s if you want to take longer to measure it). Now, inline...

> are other important points you'd like to mention for consideration.
May I give this proposal a name for brevity: ECN-DualQ-SCE (which 
sort-of represents ECN as the input classifier into 1 of 2 queues and 
SCE as the output from that queue).

>        significant reduction in queuing delay relative to no AQM at all."
Indeed, herein lies the problem. Imagine you are trying to convince a 
network operator to start a major project to tender for a new low 
latency technology then deploy it across their access network. You tell 
them it will also depend on:
* servers/CDNs deploying new OS code.
* and clients deploying new OS code.
Then you tell them that, until /most/ servers deploy, and /most/ clients 
deploy (maybe a decade?), the low latency queue will routinely add as 
much queue delay as we can already get (without clients and servers 
changing)....

One day, you continue, if all the other servers and clients passing 
traffic through that box get upgraded, it will be cool. Until that day, 
a gamer in augmented reality gets stunningly low delay,... except every 
time her daughter in the bedroom looks at a mate's facebook page or 
watches a YouTube clip.

Is the network operator really going to take all those risks for jam 
tomorrow (=  maybe a decade)? I really don't think so.

Then we'll have burned the last unicorn to routinely get what we've 
already got.

  * Incremental deployment means, as you deploy the new capability, old
    traffic continues to work, while new traffic gets the new service.
  * As you say, with ECN-DualQ-SCE, new traffic only gets the new
    service if there's no old traffic there. That's not only incremental
    deployment; that's also ineffective deployment.

> of the endpoint capabilities, especially if the network capability is

I'm afraid there are not the same pressures to cause rapid roll-out at 
all, cos it's flakey now, jam tomorrow. (Actually ECN-DualQ-SCE has a 
much greater problem - complete starvation of SCE flows - but we'll come 
on to that in Q4.)

I want to say at this point, that I really appreciate all the effort 
you've been putting in, trying to find common ground.

In trying to find a compromise, you've taken the fire that is really 
aimed at the inadequacy of underlying SCE protocol - for anything other 
than FQ. If the primary SCE proponents had attempted to articulate a way 
to use SCE in a single queue or a dual queue, as you have, that would 
have taken my fire.

> classifier.  Does that match your understanding?
This is indeed a major concern of mine (not as major as the starvation 
of SCE explained under Q4, but we'll come to that).

Fine-grained (DCTCP-like) and coarse-grained (Cubic-like) congestion 
controls need to be isolated, but I don't see how, unless their packets 
are tagged for separate queues. Without a specific fine/coarse 
identifier, we're left with having to re-use other identifiers:

  * You've tried to use ECN vs Not-ECN. But that still lumps two large
    incompatible groups (fine ECN and coarse ECN) together.
  * The only alternative that would serve this purpose is the flow
    identifier at layer-4, because it isolates everything from
    everything else. FQ is where SCE started, and that seems to be as
    far as it can go.

Should we burn the last unicorn for a capability needed on 
"carrier-scale" boxes, but which requires FQ to work? Perhaps yes if 
there was no alternative. But there is: L4S.

That brings us neatly to the outstanding issues with L4S...

> and whether I missed a more complete explanation of how it's supposed to
Firstly, as has been made clear in our latest talk/paper at Linux netdev 
and in my latest iccrg talk, currently TCP Prague only includes 
fall-back to Reno on loss. It does not do fall-back on classic ECN 
marking (yet). We're still working on RTT-independence and scaling to 
very low RTT (sub-MSS window) first.

Fall-back on loss is definitely very temporary: it does one large 
Reno-style window halving on a loss (ignoring any other losses in that 
RTT as Reno does), then immediately continues with DCTCP-style 
congestion avoidance driven by all the ECN marks (not just one per-RTT).

For classic ECN AQM detection, we only have initial design ideas. 
Olivier posted his design ideas here:
     https://github.com/L4STeam/tcp-prague/issues/2

I want to keep it simple (see response to Q4 about false negatives). 
Fall-back would be temporary, but last longer than for loss - until the 
flow next goes idle. Here's the simplest that I think might work:
     Starting X RTTs after first CE mark;    // allows end of Slow Start 
to stabilize
     if (srtt > (min_rtt + Y) || rttvar > Z) {fallback()};
Where X,Y&Z are TBD, dependent on experiments, but say X=5-6 RTT, 
Y=4-5ms & Z=dunno_without_measuring. The min_rtt could be taken only 
since the previous start-up or idle period (or perhaps the previous 
two). An idle would have to be defined as >3-4 RTT, to allow any 
self-induced queue to drain.

The whole of L4S is experimental track. So others might take different 
approaches (e.g. BBRv2) and I'm sure our approach will evolve, which is 
why the requirement is worded liberally (it has to cover real-time, etc. 
not just TCP).

> AccECN is not negotiated, right?
That's not the same sort of fall-back. That's fall-back because without 
AccECN there's only one ECN feedback signal per RTT, so it falls back to 
the configured classic congestion controller for the whole connection. 
Which controller depends on the parameter prague_ca_fallback which 
defaults to cubic.

As said above, fall-back on classic ECN has not yet been implemented in 
TCP Prague. Of the 3 things left on our list, it's the last 'cos we're 
waiting to see the results of measurements from a CDN, to see if there 
are any single queue classic ECN AQMs out there. If there aren't we 
would not plan to implement this requirement until there were. Whether 
others do is up to them of course.

> networks, and path changes?
No, it's simply 'cos there is no implementation of this requirement yet.

> the bottleneck whenever the upstream access network isn't overly
I believe FQ-CoDel is the only AQM in CPE that I know of that supports 
classic ECN. In this case, an L4S-ECN congestion controller cannot 
starve a Cubic-ECN or Reno-ECN flow, cos the FQ scheduler controls their 
capacity shares.

The only other CPE AQM I am aware of is DOCSIS-PIE, which doesn't 
support ECN.

If the IETF assigns the ECT(1) codepoint to L4S, then it would be 
extremely easy to modify FQ-Codel to set a very shallow ECN threshold in 
any queue where at least one ECT(1) codepoint had been detected. This 
would work fine with highly transient flow queues.

> up connected with the sender.
I am less worried about this. I would be comfortable erring on the side 
of reducing false positives at the expense of false negatives.

Nonetheless, this position depends on what we find in measurement studies.
* If we find no single-queue AQMs that do ECN-marking, it's a 
non-problem {Note 2}.
* If such AQMs exist but are rare, they are likely to be in specific 
operator's networks, so there would be operator-specific ways to address 
such problems. E.g. if a CDN wanted to deploy the L4S experiment on its 
caches for that network, in collaboration with the network operator it 
could set a local-use DSCP instead of using ECT(1). That would still not 
deal with L4S traffic to/from the Internet, but the probability that 
different types of long-running flows coincide is low anyway, so the 
probability that different types of flows that are both long-running and 
non-CDN will coincide must surely be tiny.

> negative happens in any kind of systematic way.
This overstates the problem. There is no unresponsiveness. Even when two 
long-running flows coincide, an L4S flow does not actually starve a 
classic (e.g. Reno-ECN) TCP flow. They come to a balance that can be 
highly unequal in high BDP links, but never starvation or 
unresponsiveness. Indeed, as the link's BDP gets smaller, or the more 
flows there are, the more DCTCP & Reno-ECN tend to equality.

> indicate sub-loss congestion.
I'm afraid you understand it wrong.

With the ECN-DualQ-SCE approach, any flows where the receiver does not 
feed back SCE (ECT(1)) markings starve any SCE (DCTCP-like) flows in the 
same bottleneck.

Similarly, any Reno-ECN or Cubic-ECN senders (i.e. without the logic to 
understand SCE) starve the SCE (DCTCP-like) flows in the same 
ECN-DualQ-SCE bottleneck.

And here, starve actually means starve. Not just come to a highly 
unbalanced equilibrium, but completely starve.

This is because a Cubic-ECN flow will keep pushing the queue up to the 
point where it emits CE markings, because it doesn't understand and 
therefore ignores the SCE markings. One queue can only have one length. 
So, because the Cubic flow(s) have pushed the queue past the shallower 
point where it starts to emit SCE markings, all packets not marked CE 
will be marked SCE.

For example, say Cubic flow(s) induce a fairly normal 0.5% CE marking 
(or 0.5% drop for non-ECN flows). Then there will be 99.5% SCE marking.

Then, the DCTCP-like flows designed to understand SCE will keep reducing 
in response to this saturated SCE marking and the Cubic flows will fill 
the space they leave and starve them.

We did experiments to try to minimize this starvation, with two AQMs in 
one queue where one type of CC ignores the signals from the lower 
threshold back in 2012. See:
     http://bobbriscoe.net/pubs.html#DCTCP-Internet
This led us to realize we would have to use at least two queues.

> important gains, to be weighed against the key downside mentioned in comment
I hope you can see now that the ECN-DualQ-SCE approach suffers from the 
same problem as you are concerned about with L4S. Except the difference 
is it's not in 'legacy' non-SCE queues, but in the queue implementing 
SCE marking itself.

Unless one separates non-SCE traffic into a different queue, it starves 
SCE traffic.

> Something similar comes up again in some other places, for instance:
(it's A.1.1.)
> pushes toward a less safe choice.
As I hope you can now see from the last part of answer #4 that, if you 
try to classify ECN flows with fine-grained (DCTCP-like) and coarse 
(Cubic-like) congestion controls into the same queue (whether L4S or SCE 
marking), the Cubic-like congestion controls ruin it.

So I think this requirement stands. I've made a note-to-self to add the 
text: "To avoid having to use per-flow classification..." though.

> can respond to SCE-style marking?
Technically yes, but realistically no.

What I mean is, as I said from the start, if you remove the feature that 
deploying the L4S DualQ Coupled AQM gives very low and consistently very 
low latency straight away, then operators will lose interest in 
deploying it.

> where enough of the key entities upgrade) are worth the risks?
Well, I'd say "probably worth the risks", cos we're waiting for 
measurements to get a feel for whether any of the CE markings seen by 
the tests Apple reported in 2016-2017 are from single queue ECN AQMs.

See 
https://datatracker.ietf.org/meeting/104/materials/slides-104-iccrg-implementing-the-prague-requirements-in-tcp-for-l4s-01#page=11

> I don't recall any offhand, but there's a lot of papers...)
Latest results here (actually no different from results we reported in 
2015 - all the changes to the code since have been non-performance related):
"DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM" Olga Albisser 
(Simula), Koen De Schepper (Nokia Bell-Labs), Bob Briscoe (Independent), 
Olivier Tilmans (Nokia Bell-Labs) and Henrik Steen (Simula), in Proc. 
Netdev 0x13 
<https://www.netdevconf.org/0x13/session.html?talk-DUALPI2-AQM> (Mar 2019).

The paper via the netdev link shows qdelay, utilization, completion time 
efficiency, etc with the most extreme traffic load we use (2 
long-running flows plus 5X Web flows per sec, where X is each link rate 
in Mb/s, e.g. 600 flows/sec over the 120Mb/s link), for a full range of 
link rates, round trip times, etc.

The plots are pretty crammed, so if you'd prefer one example qdelay 
cumulative distribution function for the same extreme traffic load, see 
here:
https://datatracker.ietf.org/meeting/104/materials/slides-104-iccrg-implementing-the-prague-requirements-in-tcp-for-l4s-01#page=22

If you want results from a range of less-extreme traffic models, just ask.

HTH

Bob

> Jake

{Note 1}: Or different server, client and network operators all agree to 
deploy, but let's assume that would be a bonus and not rely on it.

{Note 2}: Even where there are no single-queue AQMs now, there might be 
a concern that some could be enabled in future. Given study after study 
since ECN was first standardized (2001) have detected hardly any CE 
marks on the Internet until FQ-CoDel was deployed about 15 years later, 
the chance of those AQMs being turned on now is surely vanishing.

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Holland, Jake" <[email-masked]>
Date: Fri, 14 Jun 2019 17:39:12 +0000

Hi Bob,

Thanks for your response, I think it helped clarify some important things

for me.

The point about starvation especially was a good one I hadn't fully

considered, and I agree if SCE-based implementations can’t demonstrate a

solution, that would be a major problem with the SCE approach for signaling.

And sorry for my slow response, I ended up restarting a few times to try to

dodge ratholes.  (Plus some day-job duties, apologies...)

I found it a bit challenging to avoid the ratholes effectively, so I'm

thinking maybe the right move is to set up a testbed.  Maybe playing with

that (very cool-looking!) L4SDemo tool can either ease my concerns, or

provide some more specific and detailed scenarios to address.

I see that the source code is published now at

https://github.com/L4STeam/l4sdemo (thanks Olivier!).  So I’ll try to

bring that up at some point, time permitting, in hopes it makes the

comments and questions more productive.

One meta-point I wanted to make:

  "In trying to find a compromise, you've taken the fire that is really

  aimed at the inadequacy of underlying SCE protocol - for anything

  other than FQ.  If the primary SCE proponents had attempted to

  articulate a way to use SCE in a single queue or a dual queue, as you

  have, that would have taken my fire."

I think "fire" here is a potentially harmful metaphor--I don't take your

comments as an attack or this discussion as a battle, but rather a

collaborative attempt to reach a common goal of a better internet.

I hope my comments on this are received the same way, even where we don't

see eye to eye yet.  While both ideas can't be the best use of ECT(1) at

the same time, I take this discussion as an effort to reach a common and

complete understanding of the issues at hand, so that we can hopefully

agree on the best approach in the end (or if we can't get there, maybe we

can at least agree on the underlying reasons we don't agree).

With that said, a few brief points I think really should be raised:

1. "non-problem" is an unreasonably strong conclusion to reach from a

snapshot failure to detect any single-queue marking AQMs.

We know that tc-pie exists in widely deployed systems, supports ECN, and

could be turned on at any moment by anybody, and we also know there's an

increased interest in ECN since Apple and Linux got it turned on on

endpoints.  Even if we measure everything today, it’s hard to be sure this

wouldn’t impact an in-progress rollout that someone has been working toward

for their network with proper due diligence, and following IETF advice

faithfully.

I think if the intent is really to deploy this experiment under the claim

that's a non-problem, it should be called out in the docs as a risk factor,

and consensus should probably be explicitly checked on that point.  It also

probably would be polite to update RFC 7567's advice in section 4, since it

seems like this position would invalidate (or at least add nuance) to

several of the SHOULDs given there, recommending the use of ECN.

2. “does not starve a classic flow, but can be highly unequal” is also

perhaps too low a bar to consider a non-problem, and also seems like maybe

it deserves to be called out as a risk factor.

3. One more meta-point: the sales-y language makes the drafts hard to

read for me, so please forgive some of my confusion.  I'm having a hard

time distinguishing the claims that are well-supported by test results in a

realistic experimental design from some of the claims that are more forward-

looking or speculative.

(4. There’s one other point I’ll mention in response to Ingemar’s comment,

about performance being sufficient to drive adoption, and the difference

between what’s achievable with classic ECN and what’s achievable with L4S,

but that thread is perhaps a better venue for discussing it.)

Best regards,

Jake

---

From: Luca Muscariello <[email-masked]>
Date: Fri, 14 Jun 2019 22:10:03 +0200

On Fri, Jun 7, 2019 at 8:10 PM Bob Briscoe <[email-masked]> wrote:

> all, cos it's flakey now, jam tomorrow. (Actually ECN-DualQ-SCE has a much
> greater problem - complete starvation of SCE flows - but we'll come on to
> that in Q4.)
>
> In trying to find a compromise, you've taken the fire that is really aimed
> at the inadequacy of underlying SCE protocol - for anything other than FQ.
> If the primary SCE proponents had attempted to articulate a way to use SCE
> in a single queue or a dual queue, as you have, that would have taken my
> fire.
>
> This is indeed a major concern of mine (not as major as the starvation of
> SCE explained under Q4, but we'll come to that).
>
> controls need to be isolated, but I don't see how, unless their packets are
> tagged for separate queues. Without a specific fine/coarse identifier,
> we're left with having to re-use other identifiers:
>
>    - You've tried to use ECN vs Not-ECN. But that still lumps two large
>    - The only alternative that would serve this purpose is the flow
>    identifier at layer-4, because it isolates everything from everything else.
>    FQ is where SCE started, and that seems to be as far as it can go.
>
> Should we burn the last unicorn for a capability needed on "carrier-scale"
> boxes, but which requires FQ to work? Perhaps yes if there was no
> alternative. But there is: L4S.
>
>
I have a problem to understand why all traffic ends up to be classified as
either Cubic-like or DCTCP-like.
If we know that this is not true today I fail to understand why this should
be the case in the future.
It is also difficult to predict now how applications will change in the
future in terms of the traffic mix they'll generate.
I feel like we'd be moving towards more customized transport services with
less predictable patterns.

I do not see for instance much discussion about the presence of RTC traffic
and how the dualQ system behaves when the
input traffic does not respond as expected by the 2-types of sources
assumed by dualQ.

If my application is using simulcast or multi-stream techniques I can have
several video streams in the same link,  that, as far as I understand,
will get significant latency in the classic queue. Unless my app starts
cheating by marking packets to get into the priority queue.

In both cases, i.e. my RTC app is cheating or not, I do not understand how
the parametrization of the dualQ scheduler
can cope with traffic that behaves in a different way to what is assumed
while tuning parameters.
For instance, in one instantiation of dualQ based on WRR the weights are
set to 1:16.  This has to necessarily
change when RTC traffic is present. How?

Is the assumption that a trusted marker is used as in typical diffserv
deployments
or that a policer identifies and punishes cheating applications?

BTW I'd love to understand how dualQ is supposed to work under more general
traffic assumptions.

Luca

---

From: Dave Taht <[email-masked]>
Date: Fri, 14 Jun 2019 14:44:31 -0700

This thread using unconventional markers for it is hard to follow.

Luca Muscariello <[email-masked]> writes:

>     I'm afraid there are not the same pressures to cause rapid
>     roll-out at all, cos it's flakey now, jam tomorrow. (Actually
>     ECN-DualQ-SCE has a much greater problem - complete starvation of
>     SCE flows - but we'll come on to that in Q4.)

Answering that statement is the only reason why I popped up here.
more below.

>     I want to say at this point, that I really appreciate all the
>     effort you've been putting in, trying to find common ground. 

I am happy to see this thread happen also, and I do plan to
stay out of it.

>     
>     In trying to find a compromise, you've taken the fire that is
>     really aimed at the inadequacy of underlying SCE protocol - for
>     anything other than FQ.

The SCE idea does, indeed work best with FQ, in a world of widely
varying congestion control ideas as explored in the recent paper, 50
shades of congestion control:

https://arxiv.org/pdf/1903.03852.pdf

>     If the primary SCE proponents had
>     attempted to articulate a way to use SCE in a single queue or a
>     dual queue, as you have, that would have taken my fire. 

I have no faith in single or dual queues with ECN either, due to
how anyone can scribble on the relevant bits, however...

>     This is indeed a major concern of mine (not as major as the
>     starvation of SCE explained under Q4, but we'll come to that).

I think I missed a portion of this thread. Starvation is impossible,
you are reduced to no less than cwnd 2 (non-bbr), or cwnd 4 (bbr).

Your own work points out a general problem with needing sub-packet
windows with too many flows that cause excessive marking using CE, which
so far as I know remains an unsolved problem.

https://arxiv.org/pdf/1904.07598.pdf

This is easily demonstrated via experiment, also, and the primary reason
why, even with FQ_codel in the field, we generally have turned off ecn
support at low bitrates until the first major release of sch_cake.

I had an open question outstanding about the 10% figure for converting
to drop sch_pie uses that remains unresolved.

As for what level of compatability with classic transports in a single
queue that is possible with a SCE capable receiver and sender, that
remains to be seen. Only the bits have been defined as yet. Two
approaches are being tried in public, so far.

>     
>     Fine-grained (DCTCP-like) and coarse-grained (Cubic-like)
>     congestion controls need to be isolated, but I don't see how,
>     unless their packets are tagged for separate queues. Without a
>     specific fine/coarse identifier, we're left with having to re-use
>     other identifiers:
>     
>     * You've tried to use ECN vs Not-ECN. But that still lumps two
>       large incompatible groups (fine ECN and coarse ECN) together. 
>       everything else. FQ is where SCE started, and that seems to be
>       as far as it can go.

Actually, I was seeking a solution (and had been, for going on 5 years)
to the "too many flows not getting out of slow start fast enough",
problem, which you can see from any congested airport, public space,
small office, or coffeeshop nowadays. The vast majority of traffic
there does not consist of long duration high rate flows.

Even if you eliminate the wireless retries and rate changes and put in a
good fq_codel aqm, the traffic in such a large shared environment is
mostly flows lacking a need for congestion control at all (dns, voip,
etc), or in slow start, hammering away at ever increasing delays in
those environments until the user stops hitting the reload button.

Others have different goals and outlooks in this project and I'm
not really part of that.

I would rather like to see both approaches tried in an environment
that had a normal mix of traffic in a shared environment like that.

Some good potential solutions include reducing the slower bits of the
internet back to IW4 and/or using things like initial spreading, both of
which are good ideas and interact well with SCE's more immediate
response curve, paced chirping also.

>     "carrier-scale" boxes, but which requires FQ to work? Perhaps yes
>     if there was no alternative. But there is: L4S.

The core of the internet is simply overprovisioned, with fairly short
queues. DCTCP itself did not deploy in very many places that I know of.

could you define exactly what carrier scale means?

>     
>     
>
> I have a problem to understand why all traffic ends up to be
> classified as either Cubic-like or DCTCP-like. 
> If we know that this is not true today I fail to understand why this
> should be the case in the future. 
> It is also difficult to predict now how applications will change in
> the future in terms of the traffic mix they'll generate.
> I feel like we'd be moving towards more customized transport services
> with less predictable patterns.
>
> I do not see for instance much discussion about the presence of RTC
> traffic and how the dualQ system behaves when the 
> If my application is using simulcast or multi-stream techniques I can
> have several video streams in the same link, that, as far as I
> understand,
> will get significant latency in the classic queue. Unless my app
> starts cheating by marking packets to get into the priority queue.
>
> In both cases, i.e. my RTC app is cheating or not, I do not understand
> how the parametrization of the dualQ scheduler 
> can cope with traffic that behaves in a different way to what is
> assumed while tuning parameters. 
> For instance, in one instantiation of dualQ based on WRR the weights
> are set to 1:16. This has to necessarily 
> BTW I'd love to understand how dualQ is supposed to work under more
> general traffic assumptions.
>
> Luca

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 19 Jun 2019 02:15:35 +0100

Luca,

I'm still preparing a (long) reply to Jake's earlier (long) response. 
But I'll take time out to quickly clear this point up inline...

On 14/06/2019 21:10, Luca Muscariello wrote:
>
> On Fri, Jun 7, 2019 at 8:10 PM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     anything other than FQ. If the primary SCE proponents had
I'm sorry for using "Cubic-like" and "DCTCP-like", but I was trying 
(obviously unsuccessfully) to be clearer than using 'Classic' and 
'Scalable'.

"Classic" means traffic driven by congestion controls designed to 
coexist in the same queue with Reno (TCP-friendly), which necessarily 
makes it unscalable, as explained below.

The definition of a scalable congestion control concerns the power b in 
the relationship between window, W and the fraction of congestion 
signals, p (ECN or drop) under stable conditions:
     W = k / p^b
where k is a constant (or in some cases a function of other parameters 
such as RTT).
     If b >= 1 the CC is scalable.
     If b < 1 it is not (i.e. Classic).

"Scalable" does not exclude RTC traffic. For instance the L4S variant of 
SCReAM that Ingemar just talked about is scalable ("DCTCP-like"), 
because it has b = 1.

I used "Cubic-like" 'cos there's more Cubic than Reno on the current 
Internet. Over Internet paths with typical BDP, Cubic is always in its 
Reno-friendly mode, and therefore also just as unscalable as Reno, with 
b = 1/2 (inversely proportional to the square-root). Even in its proper 
Cubic mode on high BDP paths, Cubic is still unscalable with b = 0.75.

As flow rate scales up, the increase-decrease sawteeth of unscalable CCs 
get very large and very infrequent, so the control becomes extremely 
slack during dynamics. Whereas the sawteeth of scalable CCs stay 
invariant and tiny at any scale, keeping control tight, queuing low and 
utilization high. See the example of Cubic & DCTCP at Slide 5 here:
https://www.files.netdevconf.org/f/4ebdcdd6f94547ad8b77/?dl=1

Also, there's a useful plot of when Cubic switches to Reno mode on the 
last slide.

> will get significant latency in the classic queue.

You are talking as if you think that queuing delay is caused by the 
buffer. You haven't said what your RTC congestion control is (gcc 
perhaps?). Whatever, assuming it's TCP-friendly, even in a queue on its 
own, it will need to induce about 1 additional base RTT of queuing delay 
to maintain full utilization.

In the coupled dualQ AQM, the classic queue runs a state-of-the-art 
classic AQM (PI2 in our implementation) with a target delay of 15ms. 
With any less, your classic congestion controlled streams would 
under-utilize the link.

> Unless my app starts cheating by marking packets to get into the 
> priority queue.
There's two misconceptions here about the DualQ Coupled AQM that I need 
to correct.

1/ As above, if a classic CC can't build ~1 base RTT of queue in the 
classic buffer, it badly underutiizes. So if you 'cheat' by directing 
traffic from a queue-building CC into the low latency queue with a 
shallow ECN threshold, you'll just massively under-utilize the capacity.

2/ Even if it were a strict priority scheduler it wouldn't determine the 
scheduling under all normal traffic conditions. The coupling between the 
AQMs dominates the scheduler. I'll explain next...

> change when RTC traffic is present. How?

The coupling simply applies congestion signals from the C queue across 
into the L queue, as if the C flows were L flows. So, the L flows leave 
sufficient space for however many C flows there are. Then, in all the 
gaps that the L traffic leaves, any work-conserving scheduler can be 
used to serve the C queue.

The WRR scheduler is only there in case of overload or unresponsive L 
traffic; to prevent the Classic queue starving.

> or that a policer identifies and punishes cheating applications?
As explained, if a classic flow cheats, it will get v low throughput. So 
it has no incentive to cheat.

There's still the possibility of bugs/accidents/malice. The need for 
general Internet flows to be responsive to congestion is also vulnerable 
to bugs/accidents/malice, but it hasn't needed policing.

Nonetheless, in Low Latency DOCSIS, we have implemented a queue 
protection function that maintains a queuing score per flow. Then, any 
packets from high-scoring flows that would cause the queue to exceed a 
threshold delay, are redirected to the classic queue instead. For 
well-behaved flows the state that holds the score ages out between 
packets, so only ill-behaved flows hold flow-state long term.

Queue protection might not be needed, but it's as well to have it in 
case. It can be disabled.

> general traffic assumptions.
Coexistence with Reno is a general requirement for long-running Internet 
traffic. That's really all we depend on. That also covers RTC flows in 
the C queue that average to similar throughput as Reno but react more 
smoothly.

The L traffic can be similarly heterogeneous - part of the L4S 
experiment is to see how broad that will stretch to. It can certainly 
accommodate other lighter traffic like VoIP, DNS, flow startups, 
transactional, etc, etc.

BBR (v1) is a good example of something different that wasn't designed 
to coexist with Reno. It sort-of avoided too many problems by being 
primarily used for app-limited flows. It does its RTT probing on much 
longer timescales than typical sawtoothing congestion controls, running 
on a model of the link between times, so it doesn't fit the formulae above.

For BBRv2 we're promised that the non-ECN side of it will coexist with 
existing Internet traffic, at least above a certain loss level. Without 
having seen it I can't be sure, but I assume that implies it will fit 
the formulae above in some way.

PS. I believe all the above is explained in the three L4S Internet 
drafts, which we've taken a lot of trouble over. I don't really want to 
have to keep explaining it longhand in response to each email. So I'd 
prefer questions to be of the form "In section X of draft Y, I don't 
understand Z". Then I can devote my time to improving the drafts.

Alternatively, there's useful papers of various lengths on the L4S 
landing page at:
https://riteproject.eu/dctth/#papers

Cheers

Bob

---

From: Dave Taht <[email-masked]>
Date: Tue, 18 Jun 2019 18:33:31 -0700

I simply have one question. Is the code for the modified dctcp and dualpi
in the l4steam repos on github ready for independent testing?

On Tue, Jun 18, 2019, 6:15 PM Bob Briscoe <[email-masked]> wrote:

> Luca,
>
> I'm still preparing a (long) reply to Jake's earlier (long) response. But
> I'll take time out to quickly clear this point up inline...
>
>> than FQ. If the primary SCE proponents had attempted to articulate a way to
>> use SCE in a single queue or a dual queue, as you have, that would have
>> taken my fire.
>>
>> "carrier-scale" boxes, but which requires FQ to work? Perhaps yes if there
>> was no alternative. But there is: L4S.
>>
>>
> "Classic" means traffic driven by congestion controls designed to coexist
> in the same queue with Reno (TCP-friendly), which necessarily makes it
> unscalable, as explained below.
>
> the relationship between window, W and the fraction of congestion signals,
> p (ECN or drop) under stable conditions:
>     W = k / p^b
> SCReAM that Ingemar just talked about is scalable ("DCTCP-like"), because
> it has b = 1.
>
> Reno-friendly mode, and therefore also just as unscalable as Reno, with b =
> 1/2 (inversely proportional to the square-root). Even in its proper Cubic
> mode on high BDP paths, Cubic is still unscalable with b = 0.75.
>
> get very large and very infrequent, so the control becomes extremely slack
> during dynamics. Whereas the sawteeth of scalable CCs stay invariant and
> tiny at any scale, keeping control tight, queuing low and utilization high.
> See the example of Cubic & DCTCP at Slide 5 here:
> own, it will need to induce about 1 additional base RTT of queuing delay to
> maintain full utilization.
>
> classic AQM (PI2 in our implementation) with a target delay of 15ms. With
> any less, your classic congestion controlled streams would under-utilize
> the link.
>
> Unless my app starts cheating by marking packets to get into the priority
> queue.
>
> There's two misconceptions here about the DualQ Coupled AQM that I need to
> correct.
>
> traffic from a queue-building CC into the low latency queue with a shallow
> ECN threshold, you'll just massively under-utilize the capacity.
>
> sufficient space for however many C flows there are. Then, in all the gaps
> that the L traffic leaves, any work-conserving scheduler can be used to
> serve the C queue.
>
> general Internet flows to be responsive to congestion is also vulnerable to
> bugs/accidents/malice, but it hasn't needed policing.
>
> Nonetheless, in Low Latency DOCSIS, we have implemented a queue protection
> function that maintains a queuing score per flow. Then, any packets from
> high-scoring flows that would cause the queue to exceed a threshold delay,
> are redirected to the classic queue instead. For well-behaved flows the
> state that holds the score ages out between packets, so only ill-behaved
> flows hold flow-state long term.
>
> Queue protection might not be needed, but it's as well to have it in case.
> It can be disabled.
>
>
> traffic. That's really all we depend on. That also covers RTC flows in the
> C queue that average to similar throughput as Reno but react more smoothly.
>
> The L traffic can be similarly heterogeneous - part of the L4S experiment
> is to see how broad that will stretch to. It can certainly accommodate
> other lighter traffic like VoIP, DNS, flow startups, transactional, etc,
> etc.
>
>
> BBR (v1) is a good example of something different that wasn't designed to
> coexist with Reno. It sort-of avoided too many problems by being primarily
> used for app-limited flows. It does its RTT probing on much longer
> timescales than typical sawtoothing congestion controls, running on a model
> of the link between times, so it doesn't fit the formulae above.
>
> having seen it I can't be sure, but I assume that implies it will fit the
> formulae above in some way.
>
>
> PS. I believe all the above is explained in the three L4S Internet drafts,
> which we've taken a lot of trouble over. I don't really want to have to
> keep explaining it longhand in response to each email. So I'd prefer
> questions to be of the form "In section X of draft Y, I don't understand
> Z". Then I can devote my time to improving the drafts.
>
> Alternatively, there's useful papers of various lengths on the L4S landing
> page at:
> _______________________________________________
> Ecn-sane mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/ecn-sane
>

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 19 Jun 2019 04:24:54 +0000

Hi Bob and Luca,

Thank you both for this discussion, I think it helped crystallize a
comment I hadn't figured out how to make yet, but was bothering me.

I’m reading Luca’s question as asking about fixed-rate traffic that does
something like a cutoff or downshift if loss gets bad enough for long
enough, but is otherwise unresponsive.

The dualq draft does discuss unresponsive traffic in 3 of the sub-
sections in section 4, but there's a point that seems sort of swept
aside without comment in the analysis to me.

The referenced paper[1] from that section does examine the question
of sharing a link with unresponsive traffic in some detail, but the
analysis seems to bake in an assumption that there's a fixed amount
of unresponsive traffic, when in fact for a lot of the real-life
scenarios for unresponsive traffic (games, voice, and some of the
video conferencing) there's some app-level backpressure, in that
when the quality of experience goes low enough, the user (or a qoe
trigger in the app) will often change the traffic demand at a higher
layer than a congestion controller (by shutting off video, for
instance).

The reason I mention it is because it seems like unresponsive
traffic has an incentive to mark L4S and get low latency.  It doesn't
hurt, since it's a fixed rate and not bandwidth-seeking, so it's
perfectly happy to massively underutilize the link. And until the
link gets overloaded it will no longer suffer delay when using the
low latency queue, whereas in the classic queue queuing delay provides
a noticeable degradation in the presence of competing traffic.

I didn't see anywhere in the paper that tried to check the quality
of experience for the UDP traffic as non-responsive traffic approached
saturation, except by inference that loss in the classic queue will
cause loss in the LL queue as well.

But letting unresponsive flows get away with pushing out more classic
traffic and removing the penalty that classic flows would give it seems
like a risk that would result in more use of this kind of unresponsive
traffic marking itself for the LL queue, since it just would get lower
latency almost up until overload.

Many of the apps that send unresponsive traffic would benefit from low
latency and isolation from the classic traffic, so it seems a mistake
to claim there's no benefit, and it furthermore seems like there's
systematic pressures that would often push unresponsive apps into this
domain.

If that line of reasoning holds up, the "rather specific" phrase in
section 4.1.1 of the dualq draft might not turn out to be so specific
after all, and could be seen as downplaying the risks.

Best regards,
Jake

[1] https://riteproject.files.wordpress.com/2018/07/thesis-henrste.pdf

PS: This seems like a consequence of the lack of access control on
setting ECT(1), and maybe the queue protection function would address
it, so that's interesting to hear about.

But I thought the whole point of dualq over fq was that fq state couldn't
scale properly in aggregating devices with enough expected flows sharing
a queue?  If this protection feature turns out to be necessary, would that
advantage be gone?  (Also: why would one want to turn this protection off
if it's available?)

---

From: Luca Muscariello <[email-masked]>
Date: Wed, 19 Jun 2019 06:02:13 -0700

Jake,

Yes, that is one scenario that I had in mind.
Your response comforts me that I my message was not totally unreadable.

My understanding was
- There are incentives to mark packets  if they get privileged treatment
because of that marking. This is similar to the diffserv model with all the
consequences in terms of trust.
- Unresponsive traffic in particular (gaming, voice, video etc.) has
incentives to mark. Assuming there is x% of unresponsive traffic in the
priority queue, it is non trivial to guess how the system works.
- in particular it is easy to see the extreme cases,
               (a) x is very small, assuming the system is stable, the
overall equilibrium will not change.
               (b) x is very large so the dctcp like sources fall back to
cubic like and the systems behave almost like a single FIFO.
               (c) in all other cases x varies according to the
unresponsive sources' rates.
                    Several different equilibria may exist, some of which
may include oscillations. Including oscillations of all fallback
mechanisms.
The reason I'm asking is that these cases are not discussed in the I-D
documents or in the references, despite these are very common use cases.

If we add the queue protection mechanism, all unresponsive  flows that are
caught cheating are registered in a blacklist and always scheduled in the
non-priority queue.
It that happens unresponsive flows will get a service quality that is worse
than if using a single FIFO for all flows.

Using a flow blacklist brings back the complexity that dualq is supposed to
remove compared to flow-isolation by flow-queueing.
It seems to me that the blacklist is actually necessary to make dualq work
under the assumption that x is small, because in the other cases the
behavior
of the dualq system is unspecified and likely subject to instabilities,
i.e. potentially different kind of oscillations.

Luca

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 19 Jun 2019 15:11:16 +0100

Jake,

> comments and questions more productive.
[BB] Cool.

> can at least agree on the underlying reasons we don't agree).
[BB] Understood. I was concerned that I was demolishing your idea in 
public, and I was trying to thank you for being willing to put up a 
strawman.

My quest is also solely to improve the Internet. I spend my 
half-sleeping hours thinking through all the possible side-effects and 
combinations of problems with different solutions. I hope I give due 
weight to problems with my own ideas vs. problems with those of others. 
However, recently I have had to counter some rather nasty slurs on our 
work and our motivations, that did require some over-compensation.

> several of the SHOULDs given there, recommending the use of ECN.
[BB] I understand this, and indeed I've been on the other side of it 
(where someone else's inconsiderate deployment screwed up something I 
had been working on for years - and screwed up other things others had 
been working on). Nonetheless, to a certain extent, it is the Wild West 
out there, and we cannot interminably walk on egg-shells to the extent 
that nothing gets done.

Indeed, FQ itself screwed up the work on background transport protocols, 
and many other plans for novel applications of unequal throughput (I'll 
start a separate thread on that).

Don't worry. Classic ECN fall-back is on the ToDo list. I just didn't 
want to do it unless we have to, cos I prefer simplicity.

> it deserves to be called out as a risk factor.
[BB] To be clear, I wasn't trying to say that a lesser problem was a 
non-problem.

I was pointing out that the word starvation has a specific meaning, that 
doesn't apply to Scalable vs Classic, but does apply to SCE vs Cubic 
(both in a single queue).

> realistic experimental design from some of the claims that are more 
[BB] if there are any you want changed, pls call them out.

Cheers

Bob