Subject: Re: [tsvwg] Requesting TSVWG adoption of SCE draft-morton-tsvwg-sce

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 17 Nov 2019 22:07:44 +0000

Hi

I don't believe that it is a good idea to add yet another ECN experiment at
this stage.  I tried to formulate my arguments below as well and
non-provocative as possible. If the language appears blunt, then blame it on
my 8 years old kid level English. 

I have for a while advocated the role of L4S as enabler for low latency
packet transport in 3GPP standardized access. Lately it seems like that
Sisyphus ball is perhaps a little bit easier to push up the hill, but
admittedly it is far too early to uncork the champagne bottle, attempts to
promote ECN have failed before. 
5G-AIS (Advanced Interactive Services)  is one work item in 3GPP where L4S
can be a useful tool to  reach the goal of low e2e latency and the big
benefit is that we get 5G network nodes to interact well with transport
protocols devised and standardized in the IETF (TCP and QUIC among others).
There are other use cases such as remote control where L4S will come in
handy. 

You may recall that there have been a lot of initiatives over the years with
protocol additions that are suggested to add interaction between network
nodes and endpoints with explicit rate recommendations, either inband in TCP
headers or by means of some other out of band signaling mechanism, none of
these initiatives have fared well AFAIK.  My argument is here that L4S can
provide with a generic signal that can solve most (if not all) of the
challenges that the more transport protocol specific mechanisms sought to
solve, and that with much less concerns as regards to challenges with e2e
encryption of transport headers etc. Additional work such as paced chirping
etc. can further help to enhance the solutions.
 
Now, say that the WG adopts SCE as well, this means that any SDO outside
IETF will have to choose between two alternatives and the likely outcome is
that none is chosen, simply because the solution space is now completely
fragmented, in the worst case ECN becomes a dead end.  And with that
situation it becomes necessary to find other technical solutions that enable
low latency, after all we still have the wish to enable support for low
latency.   

Before somebody starts to say that 3GPP, or any other SDO for that matter,
should not dictate how IETF should work, you are absolutely right!, that is
not the intention. 
I write this as an individual with a strong interest in e2e solutions that
scale well, don't cost an arm and a leg to implement, have a chance to
become widely used, and gives good end user experience.  I don't want to
change IETFs decision process, but want to point out the risk I see if the
WG decides to adopt another ECN experiment that is largely incompatible with
the existing L4S experiment. 

/Ingemar

> From: "Roni Even (A)" <[email-masked]>
> To: G Fairhurst <[email-masked]>, "[email-masked]"
> 	<[email-masked]>
> Subject: Re: [tsvwg] Requesting TSVWG adoption of SCE
> 	draft-morton-tsvwg-sce
> Message-ID:
> 	<6E58094ECC8D8344914996DAD28F1CCD23DBDCEB@DGGEMM506-
> MBS.china.huawei.com>
> 
> Content-Type: text/plain; charset="us-ascii"
> 
> Inline
> 
> > -----Original Message-----
> > From: tsvwg [mailto:[email-masked]] On Behalf Of G Fairhurst
> > Sent: Thursday, November 14, 2019 8:51 PM
> > To: [email-masked]
> > Subject: Re: [tsvwg] Requesting TSVWG adoption of SCE draft-morton-
> > tsvwg-sce
> >
> > I think we could do better to keep the discussion focussed on why the
IETF
> > would need an additional level of signal (aka
> > ECT(0)->SCE->CE->Loss) and whether this has implications on existing
RFCs
> > and the chartered work.
> [Roni] I think this is one  way to look at it but in my view I think that
the
> SCE/L4S usage is a hint that we need to look at a more general way to
extend
> the ECN/CC information to allow for more congestion information and
> learning the path characteristics to allow for other extensions in the
future
> >
> > Gorry
> >
> > On 15/11/2019, 02:39, Sebastian Moeller wrote:
> > > Hi Koen,
> > >
> > > given the list of open issues with L4S components adopting this draft
> seems
> > pre-mature:
> > >
> > > 1) peaceful coexistence with 1/sqrt(p) traffic at short RTTs with
L4S's
> > preferred dual queue AQM (dualQ), IMHO this is a show stopper unless the
> > aim is to give L4S style traffic an (unfair) bandwidth advantage.
> > >
> > > 2) peaceful coexistence with rfc3168-compliant ECN using AQMs (TCP
> > > Prague)
> > >
> > > Given the duration of the L4S project's development so far, it seems
> > optimistic that these issues will be solved and robustly stress tested
any
> time
> > soon.
> > >
> > > In that light in incompleteness and given that evolution works best
> > > under selective pressure and competition I think SCE should be allowed
> to
> > progress to RFC status on its own merits IMHO both proposals should
(once
> > show stoppers for safe operation on the internet are fixed) be given a
fair
> > chance to prove themselves as robust reliable and effective. Trying to
> > dispose of valid competition by appeal to the editors strikes me as an
> indirect
> > admission that one's own solution might be inferior...
> > >
> > >
> > > Best Regards
> > >
> > >
> > >
> > >
> > >> On Nov 14, 2019, at 18:04, De Schepper, Koen (Nokia -
> > BE/Antwerp)<[email-masked]>  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> I think it will be a very bad idea to start now an additional
experiment
> > which conflicts with the ongoing adopted drafts. We need to make sure
> that
> > the initial work can be finalized and that we don't self-undermine the
> > ongoing successful initiative with a second proposal which has a lesser
> scope
> > and no industry adoption.
> > >>
> > >> L4S work might not be very visible on the mailing list (because of
the
> > industry context), but there is traction with first product
availabilities to be
> > announced next year. CableLabs standardized L4S in LLD, and vendors are
> > working on this. Nokia has demonstrated L4S integration in its WiFi
Beacons
> > at BBWF19, which will be available Q1 next year, and other products will
> > follow. I'm sure these are not the only examples.
> > >>
> > >> On the other hand, the main concern about SCE is still not solved: I
see
> no
> > solution for the problem that SCE works only for FQ_x.
> > >> I also see no solution for SCE to work next to and without
interfering
> with
> > the ongoing L4S initiative.
> > >>
> > >> If we want to achieve low latency on the Internet, we need to bundle
> the
> > effort and maximize the chances for success of the most promising
> solution.
> > No solution is perfect by the way, and we should stop trying to expect
> > perfection too.
> > >>
> > >> I hope from a network vendor's perspective that we can finalize the
> > network drafts. The DualPI2 implementation is stable, seems we need to
> get
> > some agreement on the wording of the drafts and we need to find a
> > compromise on the TCP-Prague requirements between what application
> > developers think is feasible and safe enough/likely to happen. The
> > application/service provider's motivation will be higher than that of
> network
> > vendors to get a deployable TCP-Prague implementation (they for sure
> have
> > already solutions ready to try out and experiment with). Let's focus on
this,
> > instead of creating diversions...
> > >>
> > >> Koen.
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: tsvwg<[email-masked]>  On Behalf Of Rodney W.
> Grimes
> > >> Sent: Thursday, November 14, 2019 2:50 PM
> > >> To: [email-masked]
> > >> Subject: [tsvwg] Requesting TSVWG adoption of SCE
> > >> draft-morton-tsvwg-sce
> > >>
> > >> Hello tsvwg list members,
> > >>
> > >> It is our intent to ask for adoption by the TSVWG of
draft-morton-tsvwg-
> > sce (https://tools.ietf.org/html/draft-morton-tsvwg-sce-01) during the
> > IETF/106 Singapore TSVWG session.
> > >>
> > >>
> > >> The TSVWG chairs have provided the following guidelines for this
> > adoption request:
> > >>
> > >> (1) The WG chairs want to see interest in SCE technology beyond the
> draft
> > authors in order to adopt the SCE draft.   This will include surveying
the
> room
> > in Singapore (e.g., who has read this
> > >> draft?).
> > >>
> > >> (2) Coexistence of the L4S and SCE experiments is a concern that will
> need
> > to be addressed by the WG if the SCE draft is adopted, and hence is in
> scope
> > for discussion of this adoption request ..  In particular, absence of a
> > coexistence plan (e.g., to deal with the different uses of the ECT(1)
> > codepoint by L4S and SCE) is not an automatic barrier to WG adoption of
> the
> > SCE draft.
> > >>
> > >> (3) The TCPM WG chairs have indicated TCPM WG willingness to
> consider
> > complementary TCP work needed to complete SCE functionality.  In
> > particular, draft-grimes-tcpm-tcpsce is likely to be inc luded in the
TCPM
> > Singapore agenda for Friday morning.
> > >>
> > >>
> > >> Regards,
> > >> --
> > >> Rod Grimes
[email-masked]

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 18 Nov 2019 00:45:33 +0000

Hi Ingemar,

If fragmenting the space will prevent other SDOs from prematurely
adopting the unproven L4S technology, that seems like exactly the
right thing to do at this stage.

I think we've seen strong evidence that L4S may still contain
show-stopping problems.  Also that we have not yet seen strong
evidence that the problems stemming from the ambiguity in the
L4S signaling design can be fixed.

This carries a demonstrated potential for breaking existing
ECN deployments by under-responding to the already widely-deployed
congestion feedback systems.

Certainly L4S's implementation was demonstrated to contain an
issue that would have wrecked the latency of existing ECN
deployments, and it had not previously been detected, despite the
years of lab evaluation and repeated requests from reviewers to
test such scenarios earlier.

Although a fix was found for the specific initially-demonstrated
case, no fix has yet been demonstrated for what looks to be a very
similar issue occurring with staggered flow startup, which can't
be attributed to a wrong alpha starting value:
https://trac.ietf.org/trac/tsvwg/ticket/17#comment:8

The proposed pseudocode fix (with up to 5 tuning parameters, IIRC)
may or may not be able to address this for specific cases, and it
may or may not be possible to discover a set of tuning values that
can address a wide range of conditions, but it seems appropriate
to have some skepticism, at least until demonstration of successful
operation under a wide range of conditions, given the history of
such proposals.  This suggests that we do the opposite of
encouraging other SDOs to move broadly forward with L4S at this time.

I share your concern that we might lose the codepoint (and the low
latency functionality), and I acknowledge that a persistently
fragmented space introduces a risk that it never happens, or takes
an extra decade.

But the risk that concerns me even more is if L4S gets rolled out
and then these kinds of issues are discovered in production, after
other SDOs have prematurely standardized on this experiment, and it
therefore gets shut off with prejudice against future solutions.
That outcome also would lose the use of the codepoint, probably
even more permanently.

(Or even worse: if it does not get shut off in spite of the problems
it causes, which loses even the low-ish latency solutions we already
have, and adds to the congestion control aggression arms race.)

IMO, those would be even worse outcomes than a somewhat delayed
adoption of a fully vetted system (or at least one that can't break
existing deployed networks).

Best regards,
Jake

PS: I still don't understand why the gains available through the
use of regular AQM (especially with ECN) have not been more widely
adopted by the other SDOs that would want to make use of L4S.

It seems possible already to reduce the application-visible delay
spikes from ~200ms to ~20ms (provided that no overly aggressive
competing traffic improperly ignores the feedback, or that
flow-queuing or other queue protection mechanisms are more widely
deployed to prevent excessive damage from aggressive flows to less
aggressive competing flows).

I wonder if whatever would drive SDOs to start using L4S maybe
could instead be leveraged to drive adoption of the much more well-
proven existing ECN solutions, which at least already have a lot
of endpoint support deployed.

The endpoint support is a critical component to making this useful,
and I see no reason to believe it'll be any quicker than the existing
regular ECN was.  I'd even expect less so, since the behavior is much
more complicated and hard to test.

PPS: I agree it would be interesting to see paced chirping solutions
to help do better than slow start, and to quickly grow when new
capacity opens on-path.  But I'll point out that's not specific to L4S,
but rather should have application for any CC that can avoid pushing
the network until queue overflow, which to me likely includes regular
ECN-enabled Reno or Cubic, as well as BBR.

However, as yet another unproven TBD, I'll suggest it's not very
useful as a strong influence on this debate, in spite of the early
demos using L4S.  Regardless of the ultimate low latency solution,
that part will need further development and might not work.

---

From: Greg White <[email-masked]>
Date: Mon, 18 Nov 2019 01:46:41 +0000

Hi Jake,

A couple of comments.

Please don't confuse L4S with TCP Prague.  TCP Prague is just one congestion controller that is L4S compatible.  There are others, and there will likely be more in the future.  The bug you are referring to was not with L4S congestion signaling, it was with the TCP Prague implementation, and a part of that implementation (exit from slow start) where there is active research (paced chirping, etc.) and opportunity for continued evolution even after L4S gets deployed.   

The second issue you are referring to is not similar to the first.  It does not affect any flows other than the TCP Prague flow itself, and it seems to be restricted to these existing IQRouter/CAKE implementations. If we can consider improvements in the IQRouter/CAKE implementations (a not unreasonable fix for this sort of issue) a fairly minor change to fq_codel (low CE threshold marking for ECT(1) packets) would fix it.  Also (and this is speculation) a more responsive AQM such as PIE instead of CoDel probably would as well.

Greg

On 11/18/19, 8:45 AM, "tsvwg on behalf of Holland, Jake" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Ingemar,

    

    

    

    Best regards,
    Jake

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 18 Nov 2019 09:43:25 +0100

Dear Greg, Others

please see more below in-line.

On November 18, 2019 2:46:41 AM GMT+01:00, Greg White <[email-masked]> wrote:
> Hi Jake,
> 
> A couple of comments.
> 
> Please don't confuse L4S with TCP Prague.  TCP Prague is just one
> congestion controller that is L4S compatible.

	[SM] As far as I can tell TCP Prague is the reference implementation for an L4S compatible congestion controller. I do not believe it to be overly onerous to expect that it actually is demonstrated to work robustly and reliably even under adversarial conditions while maintaining.implementing the L4S arch I.D.'s requirements and claims. This is your chance to demonstrate that L4S works, please use it.

>  There are others, and
> there will likely be more in the future.  

	[SM]Once/If that is the case, I am certain,  these will be also tested against and compared with the declared L4S goals, but let's focus on what is available now, please.

> The bug you are referring to
> was not with L4S congestion signaling, it was with the TCP Prague
> implementation, and a part of that implementation (exit from slow
> start) where there is active research (paced chirping, etc.) and
> opportunity for continued evolution even after L4S gets deployed.   

	[SM] I consider a bug in a non-esoteric feature like slow-start to be at leat an indication that in the impressively fast implementation period of TCP Prague a few things can still be improved, like testing of core functionality. Paced chirping (PC) is a great concept, but currently not part of TCP Prague, so it seems orthogonal to this discussion. Unless your claim is that PC is the solution to the issue at hand, in which case I think it should be implemented in TCP Prague first and demonstrate its robustness and reliability in realistic test scenarios.

> 
> The second issue you are referring to is not similar to the first.  It
> does not affect any flows other than the TCP Prague flow itself, and it
> seems to be restricted to these existing IQRouter/CAKE implementations.

	[SM] It is a demonstration though, of TCP Prague not meeting its design goals...

> If we can consider improvements in the IQRouter/CAKE implementations (a
> not unreasonable fix for this sort of issue) a fairly minor change to
> fq_codel (low CE threshold marking for ECT(1) packets) would fix it. 
> Also (and this is speculation) a more responsive AQM such as PIE
> instead of CoDel probably would as well.

	[SM] The design goal needs to be compatibility withe the existing internet as changing all the internet seems like a tall order. So IMHO we do not get to pick and choose the internet L4S/SCE needs to be compatible with. I strongly believe that any new comer needs to play fairly with the existing traffic and if in doubt scale back itself instead of just steam-rolling over standards compliant traffic, but I am not an engineer, so maybe I am too naive in that regard.

Best Regards
	Sebastian

> 
> Greg
> 
> On 11/18/19, 8:45 AM, "tsvwg on behalf of Holland, Jake"
> <[email-masked] on behalf of [email-masked]> wrote:
> 
>   Hi Ingemar,
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 19 Nov 2019 08:11:44 +0000

Hi Jake
Please see inline

/Ingemar

> From: Holland, Jake <[email-masked]>
> Sent: den 18 november 2019 01:46
> To: Ingemar Johansson S <[email-masked]>;
> [email-masked]
> Cc: [email-masked]; Ingemar Johansson S
> <[email-masked]>
> If fragmenting the space will prevent other SDOs from prematurely adopting
> the unproven L4S technology, that seems like exactly the right thing to do 
> at
> this stage.

[IJ] One can see things differently, the work to address the need of latency 
sensitive applications that also drive high demands is happening now in 3GPP. 
If the IETF finds it-self in a situation where it offers two conflicting 
solutions , using the same bits, then I would expect that none is selected. 
Instead IETF will get future work with the standardization of some other 
technology to address the same issue, call it rate recommendation or whatever. 
These ideas have been proposed before and have so far not fared well in the 
IETF.

>
> I think we've seen strong evidence that L4S may still contain show-stopping
> problems.  Also that we have not yet seen strong evidence that the
> problems stemming from the ambiguity in the L4S signaling design can be
> fixed.
>
> This carries a demonstrated potential for breaking existing ECN deployments
> by under-responding to the already widely-deployed congestion feedback
> systems.
>
> Certainly L4S's implementation was demonstrated to contain an issue that
> would have wrecked the latency of existing ECN deployments, and it had not
> previously been detected, despite the years of lab evaluation and repeated
> requests from reviewers to test such scenarios earlier.
>
> Although a fix was found for the specific initially-demonstrated case, no 
> fix
> has yet been demonstrated for what looks to be a very similar issue 
> occurring
> with staggered flow startup, which can't be attributed to a wrong alpha
> starting value:
> The proposed pseudocode fix (with up to 5 tuning parameters, IIRC) may or
> may not be able to address this for specific cases, and it may or may not be
> possible to discover a set of tuning values that can address a wide range of
> conditions, but it seems appropriate to have some skepticism, at least until
> demonstration of successful operation under a wide range of conditions,
> given the history of such proposals.  This suggests that we do the opposite 
> of
> I share your concern that we might lose the codepoint (and the low latency
> functionality), and I acknowledge that a persistently fragmented space
> introduces a risk that it never happens, or takes an extra decade.
>
> But the risk that concerns me even more is if L4S gets rolled out and then
> these kinds of issues are discovered in production, after other SDOs have
> prematurely standardized on this experiment, and it therefore gets shut off
> with prejudice against future solutions.
> That outcome also would lose the use of the codepoint, probably even more
> permanently.
>
> (Or even worse: if it does not get shut off in spite of the problems it 
> causes,
> which loses even the low-ish latency solutions we already have, and adds to
> the congestion control aggression arms race.)
>
> adoption of a fully vetted system (or at least one that can't break existing
> deployed networks).

[IJ] Following the discussion, the only burning issue that persistently pops 
up is the 0-RTT issue and it should by now be quite obvious that this is a 
manifestation of the RTT unfairness in existing congestion control algorithms. 
This use case gives L4S flows a 1ms RTT and classic flows a 15ms RTT. This was 
addressed early on in the L4S work and there is an effort to fix this in TCP 
Prague. One can end up in discussions like the 15ms target gives Cubic CC 
flows a disadvantage and that this is a proof point that L4S is flawed. 
However..
FQ-CoDel is not problem free either see 
http://dl.ifip.org/db/conf/networking/networking2017/1570335770.pdf , which 
later on led to the alternate recommendations in RFC8511, in other words the 
unfairness issues are to be addressed in the endpoints.
It will never become perfect however. If you want 100% fair fairness, then you 
need QoS, and sometimes not even that will help you if one flows is 
occasionally application limited and the other is not.

And finally.. My observation is that there is a lot of concern that classic 
flows will get an unfair treatment and that L4S flows will do better, to a 
large extent I see it as that there are flaws in classic congestion control 
than L4S can fix. Perhaps then better to use L4S instead ?.

>
> Best regards,
> Jake
>
> PS: I still don't understand why the gains available through the use of 
> regular
> AQM (especially with ECN) have not been more widely adopted by the other
> SDOs that would want to make use of L4S.
>
> It seems possible already to reduce the application-visible delay spikes 
> from
> ~200ms to ~20ms (provided that no overly aggressive competing traffic
> improperly ignores the feedback, or that flow-queuing or other queue
> protection mechanisms are more widely deployed to prevent excessive
> damage from aggressive flows to less aggressive competing flows).
>
> I wonder if whatever would drive SDOs to start using L4S maybe could
> instead be leveraged to drive adoption of the much more well- proven
> existing ECN solutions, which at least already have a lot of endpoint 
> support
> deployed.
>
> The endpoint support is a critical component to making this useful, and I 
> see
> no reason to believe it'll be any quicker than the existing regular ECN was. 
> I'd
> even expect less so, since the behavior is much more complicated and hard
> to test.
>
> PPS: I agree it would be interesting to see paced chirping solutions to help 
> do
> better than slow start, and to quickly grow when new capacity opens on-
> path.  But I'll point out that's not specific to L4S, but rather should have
> application for any CC that can avoid pushing the network until queue
> overflow, which to me likely includes regular ECN-enabled Reno or Cubic, as
> well as BBR.
>
> However, as yet another unproven TBD, I'll suggest it's not very useful as a
> strong influence on this debate, in spite of the early demos using L4S.
> Regardless of the ultimate low latency solution, that part will need further
> development and might not work.
>

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 19 Nov 2019 10:00:38 +0100

Hi Ingemar,

> On Nov 19, 2019, at 09:11, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi Jake
> Please see inline
> 
> /Ingemar
> 

	[SM] What use would it be if the IETF would endorse a solution that is not proven to actually work?

> manifestation of the RTT unfairness in existing congestion control algorithms. 

	[SM] No, I disagree, assigning this failure of properly isolating the two queues from each other to TCP's RTT dependence is a red herring. It simply demonstrates that the dual queue AQM does not offer the required strict isolation between the two queues. I believe that this is a consequence of the "coupled" design and would argue that this is the wrong solution for the problem at hand. Even a strict priority scheduler with a hard cap of 50% bandwidth for high priority (L4S) traffic (and L4S type marking) combined with an un-coupled PIE instance on the lower priority queue (with RFC3168 marking or dropping) will solve the bandwidth sharing problem*. This is only appears to an insurmountable problem due to L4S competing requirements.

*) Yest this has the side-effect that L4S traffic will be bound to 50% of the bandwidth, but IMHO that is the right call to make, if the high priority traffic does not yield properly to normal traffic the high priority traffic needs to bear the cost.

> Prague.

	[SM] An "effort" simply is not enough, nor is just acknowledging an issue, we need a robust and reliable solution. And it seems counter-intuitive to try to fix a broken scheduler/AQM design by modifying CC algorithms (also, given that L4S plans to admit non TCP Prague into the L4S queue not a complete solution either).

> One can end up in discussions like the 15ms target gives Cubic CC 

	[SM] It is a fact that this demonstrates that the dual queue AQM and scheduler is not fit for its intended purpose, independent of whether less RTT-dependence in TCP Prague is a valid and worthy goal (IMHO it is).

> FQ-CoDel is not problem free either see 

	[SM] FQ seems orthogonal here, this is about L4S being ready and safe for deployment or not.

> unfairness issues are to be addressed in the endpoints.

	[SM] For RTT-unfairness inside each of L4S's traffic classes I agree, but the failure mode is between the traffic classes...

> occasionally application limited and the other is not.

	[SM] The goal here is not perfection, but simply that L4S' core components meet the requirements good enough to merit further testing and deployment, no?

> than L4S can fix. Perhaps then better to use L4S instead ?.

	[SM] Simple answer: No. If that logic is accepted it will simply lead to an arms race of techniques that gain an advantage by playing unfair with the existing internet, the L4S requirement to coexist with existing traffic is IMHO a realization that an arms race is not an evolutionary stable strategy.
Longer answer, why trust the future of the internet on a project that is not finished and where the core components do not (yet) meet the projects own requirements? If L4S is to be the future (I have my doubts, but will real data will convince me) it needs to proof its merits own its own, and not by disadvantaging "normal traffic.

Best Regards
	Sebastian

---

From: Michael Welzl <[email-masked]>
Date: Tue, 19 Nov 2019 18:20:14 +0800

> unfairness issues are to be addressed in the endpoints.

We only evaluated CoDel, not FQ-CoDel, and this didn’t focus on fairness.
FQ_* is quite fair AFAIK  :-)

So… the unfairness issue wasn’t the argument in this paper that led to RFC 8511.

Cheers,
Michael

---
(*) the use of capitals here means: I am NOT a participant in this discussion, OH MY GOD I am not, PLEASE don’t involve me!!!   … I’m merely correcting a statement about our own earlier work.  I do enjoy watching the debate from a safe distance, though.

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 19 Nov 2019 10:30:19 +0000

Hi Michael
Fair point, or perhaps in this discussion, I should use another word than "fair" to avoid confusion 😊

Yes. I know the paper discussed CoDel, what I wanted to point out however that CoDel has its own share of issues related to link underutilization at higher RTTs, meaning that the 5ms threshold is not good enough in all cases. My interpretation is that for classic traffic you need to set the thresholds differently depending on what RTT you expect your e2e traffic to experience.

/Ingemar

> From: Michael Welzl <[email-masked]>
> Sent: den 19 november 2019 11:20
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>
> Cc: Holland, Jake <[email-masked]>; [email-masked];
> <[email-masked]> Fairhurst <[email-masked]>; Ingemar
> Johansson S <[email-masked]>
> > https://protect2.fireeye.com/v1/url?k=e91d5acb-b58f57d4-e91d1a50-
> 0cc47
> > ad93db4-16a1170b93824c3b&q=1&e=74431335-0146-453f-a47c-
> fe14226bcc52&u=
> >
> http%3A%2F%2Fdl.ifip.org%2Fdb%2Fconf%2Fnetworking%2Fnetworking201
> 7%2F1570335770.pdf , which later on led to the alternate recommendations
> in RFC8511, in other words the unfairness issues are to be addressed in the
> endpoints.
> 
> (*) the use of capitals here means: I am NOT a participant in this discussion,
> OH MY GOD I am not, PLEASE don’t involve me!!!   … I’m merely correcting a
> statement about our own earlier work.  I do enjoy watching the debate from
> a safe distance, though.

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 19 Nov 2019 18:45:01 +0800

> On 19 Nov, 2019, at 6:30 pm, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

My reading of Codel literature is that 'interval' is intended to be set at an estimate of native path RTT, accurate to within about an order of magnitude, and that 'target' is to be set at nominally 1/20th of that.  The default settings of 100ms 'interval' and 5ms 'target' are appropriate for typical Internet paths, which I believe some literature quotes at 80ms average.  I say this as someone who has worked extensively with Codel implementations.

If you expect to handle a lot of traffic over GEO satellite paths, it would be sensible to increase these values about tenfold: 1000ms interval, 50ms target.  Some locations, especially in the southeastern corner of the world, may prefer settings optimised for intercontinental paths: 300ms interval, 15ms target.  On slow links it may be necessary to increase target to avoid capturing serialisation delay, without changing interval.

When congestion control exhibits a deep sawtooth (as it traditionally does), there's always a tradeoff between peak latency and overall utilisation; Codel broadly sacrifices the latter to optimise the former, but not recklessly IMHO.  A major goal of several recent proposals, including BBR, SCE, and L4S, is to remove that deep sawtooth, and thus allow operating closer to the ideal of maximum throughput with minimum latency.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 19 Nov 2019 11:45:40 +0100

Hi Ingemar,

> On Nov 19, 2019, at 11:30, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 
> Hi Michael

	[SM] This is made pretty explicit in the codel rfc (https://tools.ietf.org/html/rfc8289) in the sections:

"4.2.  Setting INTERVAL

   The INTERVAL value is chosen to give endpoints time to react to a
   drop without being so long that response times suffer.  CoDel's
   estimator, TARGET, and control loop all use INTERVAL.  Understanding
   their derivation shows that CoDel is the most sensitive to the value
   of INTERVAL for single long-lived TCPs with a decreased sensitivity
   for traffic mixes.  This is fortunate, as RTTs vary across
   connections and are not known a priori.  The best policy seems to be
   to use an INTERVAL value slightly larger than the RTT seen by most of
   the connections using a link, a value that can be determined as the
   largest RTT seen if the value is not an outlier (use of a 95-99th
   percentile value should work).  In practice, this value is not known
   or measured (however, see Appendix A for an application where
   INTERVAL is measured).  An INTERVAL setting of 100 ms works well
   across a range of RTTs from 10 ms to 1 second (excellent performance
   is achieved in the range from 10 ms to 300 ms).  For devices intended
   for the normal terrestrial Internet, INTERVAL SHOULD have a value of
   100 ms.  This will only cause overdropping where a long-lived TCP has
   an RTT longer than 100 ms and there is little or no mixing with other
   connections through the link.

4.3.  Setting TARGET

   TARGET is the maximum acceptable persistent queue delay above which
   CoDel is dropping or preparing to drop and below which CoDel will not
   drop.  TARGET SHOULD be set to 5 ms for normal Internet traffic.

   The calculations of 
Section 3.2
 show that the best TARGET value is
   5-10% of the RTT, with the low end of 5% preferred.  Extensive
   simulations exploring the impact of different TARGET values when used
   with mixed traffic flows with different RTTs and different bandwidths
   show that below a TARGET of 5 ms, utilization suffers for some
   conditions and traffic loads; above 5 ms showed very little or no
   improvement in utilization.

   Sojourn times must remain above the TARGET for INTERVAL amount of
   time in order to enter the drop state.  Any packet with a sojourn
   time less than TARGET will reset the time that the queue was last
   below TARGET.  Since Internet traffic has very dynamic
   characteristics, the actual sojourn delay experienced by packets
   varies greatly and is often less than TARGET unless the overload is
   excessive.  When a link is not overloaded, it is not a bottleneck,
   and packet sojourn times will be small or nonexistent.  In the usual
   case, there are only one or two places along a path where packets
   will encounter a bottleneck (usually at the edge), so the total
   amount of queueing delay experienced by a packet should be less than
   10 ms even under extremely congested conditions.  This net delay is
   substantially lower than common current queueing delays on the
   Internet that grow to orders of seconds [NETAL2010] [CHARB2007].

   Regarding the roles of TARGET and the minimum-tracking INTERVAL, note
   that TARGET SHOULD NOT be increased in response to lower layers that
   have a bursty nature, where packets are transmitted for short periods
   interspersed with idle periods where the link is waiting for
   permission to send.  CoDel's estimator will "see" the effective
   transmission rate over an INTERVAL amount of time, and increasing
   TARGET only leads to longer queue delays.  On the other hand, where a
   significant additional delay is added to the intrinsic RTT of most or
   all packets due to the waiting time for a transmission, it is
   necessary to increase INTERVAL by that extra delay.  TARGET SHOULD
   NOT be adjusted for such short-term bursts, but INTERVAL MAY need to
   be adjusted if the path's intrinsic RTT changes."

Yes, you are correct that INTERVAL should be roughly matched to the observed RTT, but note that roughly here means roughly, so being off by a factor of 2 is not too problematic. TARGET however is simply a function of interval, I note again, that the derivation of TARGET from INTERVAL dependes on TCPs standard response pattern and hence should not be specific to CODEL or FQ_CODEL (this is why I proposed that as a stop-gap measure the dual queue AQM is tested with a reference queue delay value of 5 ms, but I digress).
	The same applies for PIE and other AQMs, the root cause seems to be that longer RTTs imply looser control loops and hence a disadvantage at competing with nimbler short RTT flows, which probably might be partially ameliorated, but probably is unfixable gerically (short of giving all control loops the worst case delay artificially).
But see Høiland-Jørgensen T, Hurtig P, Brunstrom A (2015) The Good, the Bad and the WiFi: Modern AQMs in a residential setting. Computer Networks 89:90–106. (https://www.sciencedirect.com/science/article/pii/S1389128615002479) for a deeper exploration of how that interacts with different path lengths and fairness.

This is also orthogonal, to the failure of L4S reference scheduler to properly share bandwidth between the two traffic classes L4S recognizes. And I personally doubt that that is easily fixable inside the existing dual queue AQM/scheduler design, but am opn to be convinced otherwise by hard empirical data.

Best Regards
	Sebastian

---

From: Michael Welzl <[email-masked]>
Date: Tue, 19 Nov 2019 22:58:37 +0800

> On Nov 19, 2019, at 6:30 PM, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Michael

Right. Indeed, that’s why ABE (RFC8511) exists.

Cheers,
Michael

---

From: Roland Bless <[email-masked]>
Date: Wed, 20 Nov 2019 22:19:30 +0800

Hi Ingemar,

> Right. Indeed, that’s why ABE (RFC8511) exists.

Another approach in your proposed direction is AQM Steering [1] that had
the same goal of allowing to vary the thresholds automatically within
certain performance policy bounds. We used it with GSP, but it could
also be applied to other AQMs such as CoDel. We mainly found a
utilization dependency  with respect to the number of active flows at
the bottleneck: for few flows one needs higher thresholds to achieve
good utilization.

Regards,
 Roland

[1] https://ieeexplore.ieee.org/document/8696627
https://doc.tm.kit.edu/2018-kit-aqm-steering-authors-copy.pdf