Subject: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE (Evolvability))

From: Ingemar Johansson S <[email-masked]>
Date: Thu, 2 Jan 2020 10:13:12 +0000

Hi

I put my question in a separate thread to avoid messing up the original 
thread.

I read in https://tools.ietf.org/html/rfc4301#section-5.1.2.1  the following

      (6) If the ECN field in the inner header is set to ECT(0) or
          ECT(1), where ECT is ECN-Capable Transport (ECT), and if the
          ECN field in the outer header is set to Congestion Experienced
          (CE), then set the ECN field in the inner header to CE;
          otherwise, make no change to the ECN field in the inner
          header.  (The IPv4 checksum changes when the ECN changes.)

So, if we have an SCE flow that is tunneled:
1) At the tunnel ingress, the ECN bits are copied to the outer header.
2) Somewhere along the tunneled path an SCE compatible AQM remarks from ECT 
'10' to SCE '01'.
3) At tunnel egress, given rule #6 above, I understand that the inner header 
will still be ECT '10'.

In order words, the SCE congestion marks along the tunneled interface become 
ignored and the queue will grow until packets are either CE marked, or 
dropped.

Is this a correct interpretation ?

/Ingemar

> -----Original Message-----
> From: Black, David <[email-masked]>
> Sent: den 2 januari 2020 00:32
> To: Bob Briscoe <[email-masked]>; Roland Bless <[email-masked]>
> Cc: De Schepper, Koen (Nokia - BE/Antwerp) <koen.de_schepper@nokia-bell-
> labs.com>; Ingemar Johansson S <[email-masked]>; Kyle
> Rose <[email-masked]>; [email-masked]; [email-masked]; Black, David
> <[email-masked]>
> Subject: RE: [tsvwg] L4S vs SCE (Evolvability)
>
> Bob,
> (posting as an individual)
>
> The assertion that SCE traffic will "starve" is just plain wrong in the
> following ...
>
> > > The SCE markings can be used independently whether you have multiple
> > > queues or just a single queue.
> > [BB] This seems to be the opposite of reality. Perhaps you've used the
> > word 'can' because you think it might be possible. But there's good
> > reason to doubt that. As follows...
> >
> > Packets from non-SCE and from SCE senders are indistinguishable by
> > just looking at them. So, if there's a single queue, they have to be
> > mixed together in it {Note 1}. But a single queue can only have one
> > length.
> > Any transports that don't understand SCE drive the single queue to a
> > deeper operating point (defined by CE or drop).
>
> The reasoning appears to be sound up to this point, but the next sentence
> does
> not follow from the above:
>
> > This overruns the SCE
> > operating point, and causes SCE marking to approach 100% {Note 2}.
> > Then those transports that do understand SCE will starve.
>
> Actually, it causes both SCE and non-SCE senders to operate at that "deeper
> operating point (defined by CE or drop)" because SCE traffic has an
> RFC-3168-
> like response to CE marks, and the usual reaction to drops.  The result is a
> deeper queue than would have resulted in the absence of non-SCE traffic, but
> the result is definitely *not* starvation.
>
> Thanks, --David
>
> > -----Original Message-----
> > From: Bob Briscoe <[email-masked]>
> > Sent: Tuesday, December 31, 2019 6:11 PM
> > To: Roland Bless
> > Cc: De Schepper, Koen (Nokia - BE/Antwerp); Ingemar Johansson S; Kyle
> > Rose; [email-masked]; [email-masked]
> > Subject: Re: [tsvwg] L4S vs SCE (Evolvability)
> >
> >
> > [EXTERNAL EMAIL]
> >
> > Roland,
> >
> > As I just said, I noticed that Koen's useful response snipped some of
> > your responses about evolvability that I ought to have addressed.
> > Sorry, you'll need to reload state from November...
> >
> > On 22/11/2019 02:32, Roland Bless wrote:
> > > Hi Bob,
> > >
> > > see inline.
> > >
> > > On 21.11.19 at 15:44 Bob Briscoe wrote:
> > >> Roland,
> > >>
> > >> I'm not getting it....
> > >>
> > >> On 21/11/2019 09:32, Roland Bless wrote:
> > >>> Hi Bob,
> > >>>
> > >>> see inline.
> > >>>
> > >>> On 21.11.19 at 19:34 Bob Briscoe wrote:
> > >>>> On 20/11/2019 21:22, Roland Bless wrote:
> > >>>>> Yes, but as I also expressed my concerns w.r.t. the L4S
> > >>>>> codepoint earlier, at the cost of binding this to a quite fixed
> > >>>>> set of L4S behaviors and "burning" the last ECT codepoint.
> > >>>>> Personally, I like concepts with a little bit more potential to
> > >>>>> be useful for future development (evolvability) of congestion
> > >>>>> controls, e.g., BBRv2 and LoLa could also benefit from an SCE-like
> marking...
> > >>>>>
> > [BB] BBRv2 has not used SCE, but it has used L4S marking. And the
> > authors of the LoLa draft have not used SCE but they have merged with
> > the NQB draft, so that they can work with L4S as well. So perhaps your
> > belief that SCE is "more evolvable" is just that - a belief.
> >
> >
> > >> This last sentence... please really spell it out what you could
> > >> possibly be meaning here. What has made you suddenly think this
> > >> particular marking scheme has got magical properties that bestow
> > >> evolvability? I'm serious, if you can't explain why you've said a
> > >> sentence like this, it implies you are under the spell of some cult or
> > >> fad.
> > > Not sure what you are trying to indicate by your last sentence, but
> > > sure, I can elaborate a bit on my last sentence.
> > > I find it architecturally cleaner to have an additional ECN
> > > codepoint used for indicating "SCE" rather than (ab)using it as
> > > classifier for the dualQ AQM.
> > [BB] I just forked another thread 'ECN as a classifier' to address
> > that assertion.
> > > The SCE markings can be used independently whether you have multiple
> > > queues or just a single queue.
> > [BB] This seems to be the opposite of reality. Perhaps you've used the
> > word 'can' because you think it might be possible. But there's good
> > reason to doubt that. As follows...
> >
> > Packets from non-SCE and from SCE senders are indistinguishable by
> > just looking at them. So, if there's a single queue, they have to be
> > mixed together in it {Note 1}. But a single queue can only have one
> > length.
> > Any transports that don't understand SCE drive the single queue to a
> > deeper operating point (defined by CE or drop). This overruns the SCE
> > operating point, and causes SCE marking to approach 100% {Note 2}.
> > Then those transports that do understand SCE will starve.
> >
> > In contrast, L4S works in dual queues or multiple queues (where 'works'
> > means it gives very low latency without losing utilization).
> >
> > Also, there is considerable scope for evolvability with both:
> > * the dualQ is a framework that is intended to encompass different
> > AQMs in future.
> > * and there is considerable scope for developing the ways FQ uses L4S.
> >
> > {Note 1}: Unless L4 identifiers are used to separate out microflows,
> > and identify low latency flows by their behaviour. But then it's not
> > really a single queue, unless you make it look like a single queue,
> > but still treat it like multiple queues, as in LFQ. However, schemes
> > like LFQ are "single-queue" in name only. They still have all the
> > downsides of multiple queues.
> >
> > {Note 2}: Unless the SCE operating point is hardly any shallower than
> > that for CE, in which case SCE gives hardly any improvement.
> >
> > > If one leaves the coupling aside the algorithm for marking will
> > > probably not differ so much from what is proposed in L4S.
> > [BB] You still haven't explained (rather than asserted a belief) why
> > the choice between SCE and L4S has anything to do with evolvability.
> >
> > >>>> My whole purpose in solving the problem of deploying scalable CCs
> > over
> > >>>> the Internet was to re-juvenate evolution (to widen the range of
> > >>>> applications that could be supported by different transport
> > >>>> behaviours, particularly for real-time with low latency and high
> > >>>> throughput at the same time). One of the main things that has
> > >>>> stopped CCs evolving so
> > far
> > >>>> is the need for friendliness with the Reno behaviour that was not
> > >>>> scaling over the years.
> > >>> FWIW, our delay-based approach has deployment problems at the
> > >>> other end of the spectrum: it gets suppressed by loss-based CCs...
> > >>> (and we don't want to sacrifice our low delay goal).
> > >>>
> > >>>> If SCE is primarily supported in FQ AQMs, that will constrain
> > >>>> flows to be capped at the rate that FQ gives them. How is that
> > >>>> doing anything
> > >>> I was solely referring to the marking behavior of SCE, not a
> > >>> particular implementation based on FQ-AQMs.
> > >> This implies you believe all that silliness about a shallower SCE
> > >> threshold not starving an SCE flow in a single queue, because it
> > >> makes SCE flows not want to use the queue.
> > > I do not understand what you are saying here.
> > [BB] I've now explained this above (after "But there's good reason to
> > doubt that...").
> >
> > The silliness I'm referring to is the statement by Jonathan M in his
> > tsvwg talk at IETF-106 that CNQ is backward compatible with RFC3168
> > traffic, because its performance for SCE traffic when mixed with 3168
> > is bad enough that it discourages use of SCE alongside 3168.
> >
> > >>>> other than massively constraining future evolution of CCs,
> > >>>> especially real-time ones? See Per-Flow Scheduling and the
> > >>>> End-to-End
> > Argument
> > >>>> <https://protect2.fireeye.com/v1/url?k=10e7b9ba-4c6d6cbf-10e7f921
> > >>>> -861010bc36ff-2b3c729b7d8a378e&q=1&e=35be9991-3367-47d3-b023-
> ed85
> > >>>>
> 01dddd52&u=http%3A%2F%2Fbobbriscoe.net%2Fprojects%2Flatency%2Fper
> > >>>> -flow_tr.pdf>. I don't
> > need
> > >>>> to tell you that the e2e argument is all about giving end systems
> > >>>> the power to innovate without permission.
> > >>>> Anyway, what are you imagining would stop CCs evolving alongside
> > other
> > >>>> scalable CCs? In much the same way CCs have always evolved. With
> > >>>> L4S
> > you
> > >>>> have a clean slate that seems just like a FIFO with a shallow
> > >>>> ECN-only immediate AQM. And other flows are causing you hardly
> > >>>> any delay and
> > very
> > >>>> rarely any loss. Think of all the things you can do with that. Go
> > >>>> evolve, Roland!
> > >>> I already said that the Dual Queue AQM is nice, but comes with the
> > >>> problem due to its built-in dropping law. Other CCs may react
> > >>> differently and BBR was one example of newer CCs that did not work
> > >>> in the L4S
> > queue
> > >>> without further adaptation.
> > >> The built in dropping law is for the old traffic ('pre-evolved' if
> > >> you
> > >> like) that doesn't respond to anything else but loss. That's what
> > >> drop-based AQMs were for - to fool Reno etc into thinking they had
> > >> hit the buffers, so to speak.
> > > Yes, I understand that. But what happens if the classic traffic once
> > > vanishes and we only have low-delay congestion control and want to
> > > re-use the "second" queue for other purposes, e.g., using it for
> > > flows that do not use loss-based CC?
> > [BB] Let me repeat back the question, 'cos it seems rather odd, so
> > perhaps I've misunderstood.
> >
> > You want me to assume that the classic behaviour of repeatedly seeking
> > out more capacity until loss occurs doesn't exist on the Internet any
> > more. There's only low-delay CC, by which I think you mean CC based on
> > L4S ECN (but I'm not sure that's your meaning).
> >
> > Then, it sounds like everything would be working well. So why do you
> > want to re-use the "second" queue for other purposes? Nothing is
> > wasted if you don't use it. It has no capacity associated with it.
> > it's just some lines of code sitting there in case packets appear from a
> > classic
> CC.
> >
> >
> > >>>    > The key thing here is the wording of the Prague requirements
> > >>>> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08#section-4>.
> > >>>> We have a session in the 'Prague' side-meeting tomorrow to review
> > them
> > >>>> (and I encourage this on this list too).
> > >>>>
> > >>>> Later down the line, if the L4S experiment is successful, there
> > >>>> will be an opportunity to review these requirements if a
> > >>>> standards track doc replaces the experimental (it is easier to
> > >>>> relax CC requirements than tighten them). So, for the expt track,
> > >>>> the requirements are designed to protect competing flows from
> > >>>> harm in a tighter way than you might find in RFC5033 or similar.
> > >>>>
> > >>>>
> > >>>> Nonetheless, even the 1/p isn't tightly spec'd, quoting:
> > >>>>
> > >>>>      The inverse proportionality requirement above is worded
> > >>>>      as a 'SHOULD' rather than a 'MUST' to allow reasonable
> > >>>> flexibility
> > >>>>      when defining these specifications.
> > >>> Yes, but something has to be implemented (in hardware) and
> > >>> therefore
> > it
> > >>> is fixed for some time and it should be consistent along a path,
> > >>> so I don't see a viable path for evolvement of this coupling law...
> > >> Why does the coupling law need to evolve? It's a relationship
> > >> between something invariant (scalable) and the past (which is over,
> > >> by definition, so it's not going to change now).
> > > See above, probably we want to put something into queue which is
> > > using a non-loss-based congestion control and/or we need to change
> > > the (1/p,1/p^2) marking.
> > [BB] The queue doesn't induce loss, the CC does. The queue isn't 1/p
> > or
> > 1/p^2 or whatever, the different CCs are. So, if a CC is using the C
> > queue without inducing any loss (e.g. a delay-based CC might be able
> > to keep the queue under the AQM's target delay), then the coupling
> > won't couple any marking across to the L queue.
> >
> > But I can't imagine why a delay-based CC that can keep the queue below
> > the target of the AQM would want to classify itself into a queue
> > designed for CCs that need to induce a decent queue (in order to
> > maintain full utilization), given such queue-inducing traffic might
> > start up at any time.
> >
> > >> If you want to evolve, you select what's best for you - probably
> > >> the nice clean L4S ECN queue. I still don't get what sort of
> > >> evolution you can't do? Evolvability isn't about a researcher being
> > >> able to do what they'd like.
> > > I also don't understand it in this way, but investigating
> > > invariables and degrees of freedom prudently may enable or
> > > facilitate deployment of new stuff. If that new stuff cannot be
> > > deployed it never gets a chance of being weeded out later either.
> > [BB] If you have something specific in mind, please say it. This is
> > all getting rather abstract.
> >
> > This does make me want to question your notion of evolvability.
> > Usually, when we make sure the Internet supports evolvability, we mean
> > evolvability of new applications. We don't mean the ability for
> > researchers to come up with different ways to solve problems that are
> > already solved. There seems to be a hint of a complaint in your emails
> > that L4S doesn't leave room for researchers to solve the same problem
> > differently. That sort of evolvability is rather a luxury isn't it? If
> > one approach solves an enduring problem with the Internet, it would be
> > rather churlish to say "No, we can't use this solution, because it
> > might preclude some ideas that might lead to other solutions to the
> > same problem in the future, but we're not sure."
> >
> > Having said that, I hope I (and Koen) have shown that L4S still leaves
> > scope for complementary delay-based CCs, not to mention scope for
> > different AQMs and for different FQ-based solutions.
> >
> >
> > Regards
> >
> >
> >
> > Bob
> >
> >
> >
> > >
> > > Roland
> > >
> >
> > --
> > __________________________________________________________
> > ______
> > Bob
> > Briscoehttps://protect2.fireeye.com/v1/url?k=9834ac6d-c4be7968-9834ecf
> > 6-861010bc36ff-d256b2de1d12a128&q=1&e=35be9991-3367-47d3-b023-
> ed8501dd
> > dd52&u=http%3A%2F%2Fbobbriscoe.net%2F

---

From: "Black, David" <[email-masked]>
Date: Thu, 2 Jan 2020 16:25:41 +0000

See RFC 6040, which updates this area of RFC 4301.

In addition, see  draft-ietf-tsvwg-rfc6040update-shim and draft-ietf-tsvwg-ecn-encap-guidelines, both of which apply RFC 6040 to additional encapsulations/tunnels ... and both of which are on Bob Briscoe's "round tuit" list to write better fragmentation text (what's currently in both drafts does not comply with RFC 3168's fragmentation provisions, hence has to be changed).

Thanks, --David

> From: Ingemar Johansson S <[email-masked]>
> Sent: Thursday, January 2, 2020 5:13 AM
> To: Black, David; Bob Briscoe; Roland Bless
> Cc: De Schepper, Koen (Nokia - BE/Antwerp); Kyle Rose; [email-masked];
> [email-masked]; Ingemar Johansson S
> Subject: RFC 4301 on ECN codepoints (was RE: [tsvwg] L4S vs SCE
> (Evolvability))
> 
> Hi
>

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 3 Jan 2020 00:03:35 +0200

> On 2 Jan, 2020, at 12:13 pm, Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
> 

Yes.  It would be as if the SCE-aware AQM on the tunnel path was merely RFC-3168 compliant, if the tunnel exit node correctly implements the above language.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 3 Jan 2020 09:10:31 +0000

Hi
Yes I am aware that SCE works with RFC6040 tunnels, it does however seem to 
work well with RFC4301 ditto.
Now I have no idea as regards to how large extent RFC4301 is deployed on the 
internet and whether this is a large problem for SCE or not. I notice however 
that RFC4301 is no problem for L4S.

/Ingemar

> Sent: den 2 januari 2020 17:26
> To: Ingemar Johansson S <[email-masked]>; Bob Briscoe
> <[email-masked]>; Roland Bless <[email-masked]>
> labs.com>; Kyle Rose <[email-masked]>; [email-masked]; tsvwg-
> [email-masked]; Black, David <[email-masked]>
> Subject: RE: RFC 4301 on ECN codepoints (was RE: [tsvwg] L4S vs SCE
> (Evolvability))
>
> In addition, see  draft-ietf-tsvwg-rfc6040update-shim and 
> draft-ietf-tsvwg-ecn-
> encap-guidelines, both of which apply RFC 6040 to additional
> encapsulations/tunnels ... and both of which are on Bob Briscoe's "round 
> tuit"
> list to write better fragmentation text (what's currently in both drafts 
> does not
> comply with RFC 3168's fragmentation provisions, hence has to be changed).
>
> Thanks, --David
>
> > I put my question in a separate thread to avoid messing up the
> > original thread.
> >
> > 2) Somewhere along the tunneled path an SCE compatible AQM remarks
> > from ECT '10' to SCE '01'.
> > 3) At tunnel egress, given rule #6 above, I understand that the inner
> > header will still be ECT '10'.
> >
> > In order words, the SCE congestion marks along the tunneled interface
> > become ignored and the queue will grow until packets are either CE
> > marked, or dropped.
> >
> > > Rose <[email-masked]>; [email-masked]; [email-masked];
> > > Black,
> > David
> > > <[email-masked]>
> > > The assertion that SCE traffic will "starve" is just plain wrong in
> > > the following ...
> > >
> > > > [BB] This seems to be the opposite of reality. Perhaps you've used
> > > > the word 'can' because you think it might be possible. But there's
> > > > good reason to doubt that. As follows...
> > > >
> > > > just looking at them. So, if there's a single queue, they have to
> > > > be mixed together in it {Note 1}. But a single queue can only have
> > > > one length.
> > > > Any transports that don't understand SCE drive the single queue to
> > > > a deeper operating point (defined by CE or drop).
> > >
> > > The reasoning appears to be sound up to this point, but the next
> > > sentence does not follow from the above:
> > >
> > > > This overruns the SCE
> > > like response to CE marks, and the usual reaction to drops.  The
> > > result is a deeper queue than would have resulted in the absence of
> > > non-SCE traffic,
> > but
> > > > Cc: De Schepper, Koen (Nokia - BE/Antwerp); Ingemar Johansson S;
> > > > Kyle Rose; [email-masked]; [email-masked]
> > > > As I just said, I noticed that Koen's useful response snipped some
> > > > of your responses about evolvability that I ought to have addressed.
> > > > >>>>> codepoint earlier, at the cost of binding this to a quite
> > > > >>>>> fixed set of L4S behaviors and "burning" the last ECT codepoint.
> > > > >>>>> Personally, I like concepts with a little bit more potential
> > > > >>>>> to be useful for future development (evolvability) of
> > > > >>>>> congestion controls, e.g., BBRv2 and LoLa could also benefit
> > > > >>>>> from an SCE-like
> > > marking...
> > > > >>>>>
> > > > authors of the LoLa draft have not used SCE but they have merged
> > > > with the NQB draft, so that they can work with L4S as well. So
> > > > perhaps your belief that SCE is "more evolvable" is just that - a 
> > > > belief.
> > > >
> > > >
> > > > >> particular marking scheme has got magical properties that
> > > > >> bestow evolvability? I'm serious, if you can't explain why
> > > > >> you've said a sentence like this, it implies you are under the
> > > > >> spell of some cult or fad.
> > > > > Not sure what you are trying to indicate by your last sentence,
> > > > > but sure, I can elaborate a bit on my last sentence.
> > > > [BB] This seems to be the opposite of reality. Perhaps you've used
> > > > the word 'can' because you think it might be possible. But there's
> > > > good reason to doubt that. As follows...
> > > >
> > > > just looking at them. So, if there's a single queue, they have to
> > > > be mixed together in it {Note 1}. But a single queue can only have
> > > > one length.
> > > > Any transports that don't understand SCE drive the single queue to
> > > > a deeper operating point (defined by CE or drop). This overruns
> > > > the SCE operating point, and causes SCE marking to approach 100% {Note
> 2}.
> > > > {Note 1}: Unless L4 identifiers are used to separate out
> > > > microflows, and identify low latency flows by their behaviour. But
> > > > then it's not really a single queue, unless you make it look like
> > > > a single queue, but still treat it like multiple queues, as in
> > > > LFQ. However, schemes like LFQ are "single-queue" in name only.
> > > > They still have all the downsides of multiple queues.
> > > >
> > > > {Note 2}: Unless the SCE operating point is hardly any shallower
> > > > than that for CE, in which case SCE gives hardly any improvement.
> > > >
> > > > [BB] You still haven't explained (rather than asserted a belief)
> > > > why the choice between SCE and L4S has anything to do with 
> > > > evolvability.
> > > >
> > > > >>>> the Internet was to re-juvenate evolution (to widen the range
> > > > >>>> of applications that could be supported by different
> > > > >>>> transport behaviours, particularly for real-time with low
> > > > >>>> latency and high throughput at the same time). One of the
> > > > >>>> main things that has stopped CCs evolving so
> > > > far
> > > > >>>> is the need for friendliness with the Reno behaviour that was
> > > > >>>> not scaling over the years.
> > > > >>>> flows to be capped at the rate that FQ gives them. How is
> > > > >>>> that doing anything
> > > > >> This implies you believe all that silliness about a shallower
> > > > >> SCE threshold not starving an SCE flow in a single queue,
> > > > >> because it makes SCE flows not want to use the queue.
> > > > [BB] I've now explained this above (after "But there's good reason
> > > > to doubt that...").
> > > >
> > > > The silliness I'm referring to is the statement by Jonathan M in
> > > > his tsvwg talk at IETF-106 that CNQ is backward compatible with
> > > > RFC3168 traffic, because its performance for SCE traffic when
> > > > mixed with 3168 is bad enough that it discourages use of SCE alongside
> 3168.
> > > >
> > > > >>>> to tell you that the e2e argument is all about giving end
> > > > >>>> systems the power to innovate without permission.
> > > > >>>> scalable CCs? In much the same way CCs have always evolved.
> > > > >>>> With L4S
> > > > you
> > > > >>>> ECN-only immediate AQM. And other flows are causing you
> > > > >>>> hardly any delay and
> > > > very
> > > > >>>> rarely any loss. Think of all the things you can do with
> > > > >>>> that. Go evolve, Roland!
> > > > >>> I already said that the Dual Queue AQM is nice, but comes with
> > > > >>> the problem due to its built-in dropping law. Other CCs may
> > > > >>> react differently and BBR was one example of newer CCs that
> > > > >>> did not work in the L4S
> > > > queue
> > > > >> The built in dropping law is for the old traffic ('pre-evolved'
> > > > >> if you
> > > > >> like) that doesn't respond to anything else but loss. That's
> > > > >> what drop-based AQMs were for - to fool Reno etc into thinking
> > > > >> they had hit the buffers, so to speak.
> > > > > Yes, I understand that. But what happens if the classic traffic
> > > > > once vanishes and we only have low-delay congestion control and
> > > > > want to re-use the "second" queue for other purposes, e.g.,
> > > > > using it for flows that do not use loss-based CC?
> > > > You want me to assume that the classic behaviour of repeatedly
> > > > seeking out more capacity until loss occurs doesn't exist on the
> > > > Internet any more. There's only low-delay CC, by which I think you
> > > > mean CC based on L4S ECN (but I'm not sure that's your meaning).
> > > >
> > > > Then, it sounds like everything would be working well. So why do
> > > > you want to re-use the "second" queue for other purposes? Nothing
> > > > is wasted if you don't use it. It has no capacity associated with it.
> > > > it's just some lines of code sitting there in case packets appear
> > > > from a classic
> > > CC.
> > > >
> > > >
> > > > >>>> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08#s
> > > > >>>> ection-
> > 4>.
> > > > >>>> Later down the line, if the L4S experiment is successful,
> > > > >>>> there will be an opportunity to review these requirements if
> > > > >>>> a standards track doc replaces the experimental (it is easier
> > > > >>>> to relax CC requirements than tighten them). So, for the expt
> > > > >>>> track, the requirements are designed to protect competing
> > > > >>>> flows from harm in a tighter way than you might find in RFC5033 
> > > > >>>> or
> similar.
> > > > >>>>
> > > > >>>>
> > > > >>> is fixed for some time and it should be consistent along a
> > > > >>> path, so I don't see a viable path for evolvement of this coupling 
> > > > >>> law...
> > > > >> between something invariant (scalable) and the past (which is
> > > > >> over, by definition, so it's not going to change now).
> > > > > using a non-loss-based congestion control and/or we need to
> > > > > change the (1/p,1/p^2) marking.
> > > > [BB] The queue doesn't induce loss, the CC does. The queue isn't
> > > > 1/p or
> > > > 1/p^2 or whatever, the different CCs are. So, if a CC is using the
> > > > C queue without inducing any loss (e.g. a delay-based CC might be
> > > > able to keep the queue under the AQM's target delay), then the
> > > > coupling won't couple any marking across to the L queue.
> > > >
> > > > But I can't imagine why a delay-based CC that can keep the queue
> > > > below the target of the AQM would want to classify itself into a
> > > > queue designed for CCs that need to induce a decent queue (in
> > > > order to maintain full utilization), given such queue-inducing
> > > > traffic might start up at any time.
> > > >
> > > > >> If you want to evolve, you select what's best for you -
> > > > >> probably the nice clean L4S ECN queue. I still don't get what
> > > > >> sort of evolution you can't do? Evolvability isn't about a
> > > > >> researcher being able to do what they'd like.
> > > > [BB] If you have something specific in mind, please say it. This
> > > > is all getting rather abstract.
> > > >
> > > > Usually, when we make sure the Internet supports evolvability, we
> > > > mean evolvability of new applications. We don't mean the ability
> > > > for researchers to come up with different ways to solve problems
> > > > that are already solved. There seems to be a hint of a complaint
> > > > in your emails that L4S doesn't leave room for researchers to
> > > > solve the same problem differently. That sort of evolvability is
> > > > rather a luxury isn't it? If one approach solves an enduring
> > > > problem with the Internet, it would be rather churlish to say "No,
> > > > we can't use this solution, because it might preclude some ideas
> > > > that might lead to other solutions to the same problem in the future, 
> > > > but
> we're not sure."
> > > >
> > > > Having said that, I hope I (and Koen) have shown that L4S still
> > > > leaves scope for complementary delay-based CCs, not to mention
> > > > scope for different AQMs and for different FQ-based solutions.
> > > >
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > > Bob
> > > >
> > > >
> > > >
> > > > >
> > > > > Roland
> > > > >
> > > >
> > > > --
> > > >

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 3 Jan 2020 09:13:54 +0000

Hi Jonathan
Thanks for the response. I assume then that RFC4301 should be listed as a
possible issue for SCE, as already pointed out by Bob ?

/Ingemar

> From: Jonathan Morton <[email-masked]>
> Sent: den 2 januari 2020 23:04
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>
> Cc: Black, David <[email-masked]>; Bob Briscoe
<[email-masked]>;
> Roland Bless <[email-masked]>; Ingemar Johansson S
> <[email-masked]>; [email-masked]; [email-masked]
> Subject: Re: [tsvwg] RFC 4301 on ECN codepoints (was RE: L4S vs SCE
> (Evolvability))
> 
> > On 2 Jan, 2020, at 12:13 pm, Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]> wrote:
> >
> Yes.  It would be as if the SCE-aware AQM on the tunnel path was merely
RFC-
> 3168 compliant, if the tunnel exit node correctly implements the above
> language.
> 
>  - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 03 Jan 2020 10:28:18 +0100

Hi Ingmar,

>that RFC4301 is no problem for L4S.

        [SM] I beg to differ, L4S will not play nice with either a rfc3168 or SCE-capable AQM on a tunneled segment, as it will completely misinterpret the received CE marks and will not respond appropriately with a sufficiently large reduction of the congestion window*. The currently still preferred solution to this issue, if I understand Bob correctly, is still to reason that such ECN-enabled AQMs along tunneled path segments (which I understand to be always inside the core of an AS as towards the edges packets get decapsulated) are rare enough to allow completely ignoring them.

Best Regards
         Sebastian

*) I realize that there is finally some on-going research work on rfc3168-compatibility in TCP Prague, but I have seen no data yet indicating that the proposed scheme triggers fast enough (or even at all). I reserve judgement until I see the data.

>> > > Cc: De Schepper, Koen (Nokia - BE/Antwerp)
><koen.de_schepper@nokia-
>> > bell-
>> > > labs.com>; Ingemar Johansson S
><[email-masked]>;
>> > Kyle
>> > > Actually, it causes both SCE and non-SCE senders to operate at
>that
>> > "deeper
>> > > > In contrast, L4S works in dual queues or multiple queues (where
>
>> > > > 'works'
>> > > > >>>> My whole purpose in solving the problem of deploying
>scalable
>> > > > >>>> CCs
>> > > > over

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 3 Jan 2020 11:18:08 +0000

Hi
Please see inline
/Ingemar

> From: Sebastian Moeller <[email-masked]>
> Sent: den 3 januari 2020 10:28
> To: Ingemar Johansson S
> <ingemar.s.johansson=[email-masked]>; Black, David
> <[email-masked]>; Bob Briscoe <[email-masked]>; Roland Bless
> <[email-masked]>
> Cc: Ingemar Johansson S <[email-masked]>; tsvwg-
> [email-masked]; [email-masked]
> On January 3, 2020 10:10:31 AM GMT+01:00, Ingemar Johansson S
> >Yes I am aware that SCE works with RFC6040 tunnels, it does however
> >seem to work well with RFC4301 ditto.
> >Now I have no idea as regards to how large extent RFC4301 is deployed
> >on the internet and whether this is a large problem for SCE or not. I
> >notice however that RFC4301 is no problem for L4S.
>
>         [SM] I beg to differ, L4S will not play nice with either a rfc3168 
> or SCE-
> capable AQM on a tunneled segment, as it will completely misinterpret the
> received CE marks and will not respond appropriately with a sufficiently 
> large
> reduction of the congestion window*. The currently still preferred solution 
> to
> this issue, if I understand Bob correctly, is still to reason that such 
> ECN-enabled
> AQMs along tunneled path segments (which I understand to be always inside 
> the
> core of an AS as towards the edges packets get decapsulated) are rare enough
> to allow completely ignoring them.

[IJ] This is more of a concern with traditional loss/ECN based congestion 
control algorithms like Reno, Cubic (and DCTCP). If one look more into the 
future (and actually present) I see algorithms like BBRv2 that strive one BDP 
queue and will therefore not strive to bloat the queues as much as the 
forementioned CC algos.  It seems to me like there is incentive to go towards 
more of these. And as you point out below there is work ongoing work by Bob 
and others to add RFC3168 compatibility to TCP Prague.
Speaking of the SCE capable AQMs, how much does it take to make them L4S 
capable instead, an update of RFC8290 ?

> Best Regards
>          Sebastian
>
> *) I realize that there is finally some on-going research work on rfc3168-
> compatibility in TCP Prague, but I have seen no data yet indicating that the
> proposed scheme triggers fast enough (or even at all). I reserve judgement 
> until I
> see the data.
>
> >
> >/Ingemar
> >
> ><koen.de_schepper@nokia-bell-

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 3 Jan 2020 17:36:46 +0100

Hi Ingemar,

more below in-line.

> On Jan 3, 2020, at 12:18, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> Please see inline
> /Ingemar
> 
> control algorithms like Reno, Cubic (and DCTCP).

	[SM] How so? As far as I can tell the failure mode is, that L4S flows will not yield to the rfc3168 AQMs CE marks in the way that the AQM expects/is designed for, thereby crowding out the 1/sqrt(p) flows and securing themselves the lion's share of the bandwidth (assuming the hop does not need to dip deep into dropping-instead-of-marking territory)). BBR, might bully over the AQM's signals, but I am not sure whether that actually is an improvement over the status quo (unless purely viewed from BBR's perspective), but that is a different kettle of fish.

> If one look more into the 

	[SM] Which albeit quite belated is a welcome development. I am looking forward to the data demonstrating that approach working as intended and as required (I am especially interested to see how long this is going to take and whether it introduces a "latency-bolus" in the queue). But that still leaves all other transports the L4S folks envision classifying into the L4S-queue to worry about. Doing that for TCP Prague is only a first mile-stone, sure being proof of principle it might be an important mile stone, but if coexistence with the existing internet is of serious concern (which it should) it is only the beginning... 
	I consider this to be a side-effect/consequence of redefining what a CE-mark means, as the SCE proposal demonstrates that is not the only option to implement 1/p style signaling.

> capable instead, an update of RFC8290 ?

	[SM] Note that RFC8290 is not really SCE capable to begin with (how could it, it predates the SCE drafts considerably). Be that as it may, IMHO it would take a) as a precondition a robust and reliable indicator of how a flow will respond to congestion signaling, 1/p or 1/sqrt(p), unfortunately ECT(1) is neither robust nor reliable*, and b) a consensus that overloading the ECN CE-codepoint with two semantic different meanings is reliable, robust (and safe for the rest of the network). 
	The more relevant question for me is what it would take to remedy these two obvious issues in the L4S design... and what would it take in addition to start tackling the failure of the dual queue coupled AQM to do the one thing is was intended/designed for reliably and robustly. 

Best Regards
	Sebastian

*) Which flow does a CE-marked packet belong to a L4S-capable or an rfc3168-compatible one?

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 3 Jan 2020 23:05:10 +0200

> On 3 Jan, 2020, at 11:13 am, Ingemar Johansson S <[email-masked]> wrote:
> 

Only to the extent that it forces a reversion to backwards compatible behaviour.  It's a performance thing rather than a safety thing, and as such it's a situation we can live with.

It does not, for example, prevent the safe deployment of SCE endpoints and middleboxes, in the way that RFC-3168 compliant middleboxes presently do for L4S endpoints.

 - Jonathan Morton

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 4 Jan 2020 14:41:27 +0000

Hi
See inline
/Ingemar

> Sent: den 3 januari 2020 17:37
> To: Ingemar Johansson S <[email-masked]>
> Cc: Ingemar Johansson S
> <[email-masked]>; [email-masked]; [email-masked]
> > On Jan 3, 2020, at 12:18, Ingemar Johansson S
> <[email-masked]> wrote:
> >
> > Hi
> > Please see inline
> > /Ingemar
> >
> >> <[email-masked]>; Bob Briscoe <[email-masked]>; Roland
> >> Bless <[email-masked]>
> >>> Now I have no idea as regards to how large extent RFC4301 is
> >>> deployed on the internet and whether this is a large problem for SCE
> >>> or not. I notice however that RFC4301 is no problem for L4S.
> >>
> >>        [SM] I beg to differ, L4S will not play nice with either a
> >> rfc3168 or SCE- capable AQM on a tunneled segment, as it will
> >> completely misinterpret the received CE marks and will not respond
> >> appropriately with a sufficiently large reduction of the congestion
> >> window*. The currently still preferred solution to this issue, if I
> >> understand Bob correctly, is still to reason that such ECN-enabled
> >> AQMs along tunneled path segments (which I understand to be always
> >> inside the core of an AS as towards the edges packets get
> >> decapsulated) are rare enough to allow completely ignoring them.
> >
> > [IJ] This is more of a concern with traditional loss/ECN based
> > congestion control algorithms like Reno, Cubic (and DCTCP).
> 
> 	[SM] How so? As far as I can tell the failure mode is, that L4S
flows will
> not yield to the rfc3168 AQMs CE marks in the way that the AQM expects/is
> designed for, thereby crowding out the 1/sqrt(p) flows and securing
themselves
> the lion's share of the bandwidth (assuming the hop does not need to dip
deep
> into dropping-instead-of-marking territory)). BBR, might bully over the
AQM's
> signals, but I am not sure whether that actually is an improvement over
the
> status quo (unless purely viewed from BBR's perspective), but that is a
different
> kettle of fish.
> > future (and actually present) I see algorithms like BBRv2 that strive
> > one BDP queue and will therefore not strive to bloat the queues as
> > much as the forementioned CC algos.  It seems to me like there is
> > incentive to go towards more of these. And as you point out below
> > there is work ongoing work by Bob and others to add RFC3168
compatibility to
> TCP Prague.
> 
> 	[SM] Which albeit quite belated is a welcome development. I am
looking
> forward to the data demonstrating that approach working as intended and as
> required (I am especially interested to see how long this is going to take
and
> whether it introduces a "latency-bolus" in the queue). But that still
leaves all
> other transports the L4S folks envision classifying into the L4S-queue to
worry
> about. Doing that for TCP Prague is only a first mile-stone, sure being
proof of
> principle it might be an important mile stone, but if coexistence with the
existing
> internet is of serious concern (which it should) it is only the
beginning...
> 	I consider this to be a side-effect/consequence of redefining what a
CE-
> mark means, as the SCE proposal demonstrates that is not the only option
to
> implement 1/p style signaling.
> 
> 
> > Speaking of the SCE capable AQMs, how much does it take to make them
> > L4S capable instead, an update of RFC8290 ?
> 
> 	[SM] Note that RFC8290 is not really SCE capable to begin with (how
> could it, it predates the SCE drafts considerably). Be that as it may,
IMHO it
> would take a) as a precondition a robust and reliable indicator of how a
flow will
> respond to congestion signaling, 1/p or 1/sqrt(p), unfortunately ECT(1) is
neither
> robust nor reliable*, and b) a consensus that overloading the ECN
CE-codepoint
> with two semantic different meanings is reliable, robust (and safe for the
rest of
> the network).
> 	The more relevant question for me is what it would take to remedy
> these two obvious issues in the L4S design... and what would it take in
addition
> to start tackling the failure of the dual queue coupled AQM to do the one
thing is
> was intended/designed for reliably and robustly.

[IJ] I understand that 
  a) delves into the flow rate fairness that has been discussed at length in
other mail treads.
     I don't see the possible rate unfairness that DualQ gives as unsafe and
it is quite clear to me that 
     this is something that relates more to how endpoint congestion control
algorithms are deigned
     than on the specific design of DualQ. I understand that we disagree on
this and therefore it 
     is best to ask a wider group to comment this.
  b) Means that a RCF3168 ECN capable flow whose packets have been CE marked
upstreams are put 
    in the L4S queue in the DualQ downstreams and therefore causing packet
reordering for the RFC3168 flow. 
    The severity of this reordering assumes a scenario where 
     i) congestion occurs simultaneously both on the upstreams and
downstreams, 
        I have so far seen this as quite unlikely 
     ii) congestion occurs upstreams and the uncongested DualQ downstreams
deliberately builds a large queue, 
        I have not see that DualQ does this. 
    I have possibly missed something, please clarify

> 
> Best Regards
> 	Sebastian
> 
> 
> 
> *) Which flow does a CE-marked packet belong to a L4S-capable or an
rfc3168-
> compatible one?
> 
> 
> >
> >
> >> Best Regards
> >>         Sebastian
> >>
> >> *) I realize that there is finally some on-going research work on
> >> rfc3168- compatibility in TCP Prague, but I have seen no data yet
> >> indicating that the proposed scheme triggers fast enough (or even at
> >> all). I reserve judgement until I see the data.
> >>
> >>>
> >>> /Ingemar
> >>>
> >>>>>>>>>>> -861010bc36ff-2b3c729b7d8a378e&q=1&e=35be9991-3367-
> 47d3-
> >>>>> b023-
> >>>>>> ed85
> >>>>>>>>>>>
> >>>>>>
> >>>>>

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 4 Jan 2020 16:30:11 +0100

Hi Ingemar,

more below in-line.

> On Jan 4, 2020, at 15:41, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> See inline
> /Ingemar
> 

	[SM] I am puzzled, a) above is about how it mark/differentiate between flows that should be treated to 1/p versus those that should be treated to 1/sqrt(p) congestion signaling. IMHO this has exactly zero to do with flow rate fairness and all with how to make the two types of congestion responses coexist peacefully in the existing internet. But please elaborate what you referred to above with "flow rate fairness".
	Really, the constant attacks against FQ solution are a RED HERRING in the context whether L4S is ready to reach experimental stage. My issue is not that L4S might not be the theoretically best solution for the challenge it was designed to solve, my issue is that practically the current implementations ARE NO SOLUTION at all, as they do not work reliably and robustly, period.

>     I don't see the possible rate unfairness that DualQ gives as unsafe

	[SM] Try the shoe on the other foot then, if the IETF would command to change the dual queue coupled AQM so that it gives the same priority/bandwidth advantage to non-L$S flows instead, would you still be okay with that? If your answer is yes, you can predict which changes to dual queue coupled AQM I will propose next... I take it that you also consider 15 ms the ideal ref-delay for the non-L4S's queue PI2 instance without being able to tell what makes 15 so magically wonderful in that context?

>     than on the specific design of DualQ.

	[SM] Again I disagree, the dual queue coupled AQM's sole reason to exists is to equitably share between the two traffic classes the L4S approach categorizes the world into. Failure to do so reliably and robustly is NOT a function of the endpoints  but of your isolation mechanism (as an isolator that can be easily gamed by the endpoints does not even deserve that name). 
	I realize that this is an inconvenient truth and that both Bob and Koen always try to change this into an unrelated discussions about say RTT-independence, but that is simply a distraction strategy as I have alluded to multiple times, not least because so far the L4S team has utterly failed to present data demonstrating that they managed to make TCP Prague, their demonstration transport, RTT-independent enough to paper over the dual queue coupled AQM's design mistakes. But to be explicit, even if you manage to fix this for TCP Prague, unless you only admit TCP Prague flow the the L4S queue you have won very little. If it apprears that I repeat myself, then simply because part of my audience does not seem to be listening or engaging with my arguments at all.

> I understand that we disagree on
> this and therefore it 

	[SM] Exactly, by mis-classifying CE packets into the L4S queue one risks out-of-order delivery for non-L4S flows, as a direct consequence of abusing a single codepoint as binary classifier. If this single codepoint would be the only option this might be worth discussing/accepting, but since there are quite a number of different classifier schemes ECT(1) really should be taken off the table IMHO.
	Except, that for the reordering to happen, you do not even need packets in both directions, even in one direction reordering can be inflicted upon non-L4S flows, if a CE-marked packet hits the dual queue coupled AQM it will traverse the L4S queue and will be given priority over packets of the same flow in the non-L4S queue, so as long as the non-L4S queue is not empty reordering can happen. That is a side-effect most easily dealt with by employing a full independent bit for proper classification, or by defaulting all CE-marked packets to the non-L4S queues (that way reordering is restricted to the new L4S flows and there s a real incentive to carefully look again at the unfortunate choice of 15ms ref_delay for the non-L4S queue, a win-win situation form the perspective of non-L4S traffic).

>     i) congestion occurs simultaneously both on the upstreams and

	[SM] I am not sure that you need simultaneous congestion in both directions, and secondly, I believe this situation to be more probable/likely than you seem to assume (this is partly a consequence of the often highly asymmetric links to end-users). 

>    I have possibly missed something, please clarify

	As above for packet reordering you do not need simultaneous bi-directional congestion. My point however is that this again is one of the situations where the L4S team opted to make the non-L4S traffic pay all the cost of the "impedance mismatch" between the current network and the "brave new world" that L4S promises.

Best Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 4 Jan 2020 15:42:19 +0000

Sebastian,

On 03/01/2020 09:28, Sebastian Moeller wrote:
>> on the
>          [SM] I beg to differ, L4S will not play nice with either a rfc3168 or SCE-capable AQM on a tunneled segment, as it will completely misinterpret the received CE marks and will not respond appropriately with a sufficiently large reduction of the congestion window*.
What motivation did you have for intervening in a thread about a problem 
with tunnelling SCE codepoints to highlight a problem with L4S that you 
have already pointed out many times, that is (yet again) not relevant to 
the sentence, nor to the thread? If someone was motivated to spread fear 
uncertainty and doubt about L4S and to distract attention from any 
criticism of SCE, they would behave exactly like you are behaving.

Do you accept that "RFC4301 is not problem for L4S", which was the 
sentence you "begged to differ" about?
Do you accept (as Jonathan just has) that if an SCE AQM is within an 
RFC4301 tunnel, the tunnel endpoint will black-hole the SCE markings?

Please answer both questions without distracting the thread onto another 
issue. I am not trying to suppress discussion of other issues, I am 
trying to keep discussion of each issue on-topic.

Bob

> The currently still preferred solution to this issue, if I understand Bob correctly, is still to reason that such ECN-enabled AQMs along tunneled path segments (which I understand to be always inside the core of an AS as towards the edges packets get decapsulated) are rare enough to allow completely ignoring them.
>
> Best Regards
>           Sebastian
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 4 Jan 2020 17:16:11 +0100

Dear Bob,

more below in-line.

> On Jan 4, 2020, at 16:42, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> What motivation did you have for intervening in a thread about a problem with tunnelling SCE codepoints to highlight a problem with L4S that you have already pointed out many times, that is (yet again) not relevant to the sentence, nor to the thread? If someone was motivated to spread fear uncertainty and doubt about L4S and to distract attention from any criticism of SCE, they would behave exactly like you are behaving.

	[SM] The point I countered was "that RFC4301 is no problem for L4S" which  think is a bit disingenuous as the model discussed is a rfc3168 AQM on the tunneled segment, and L4S will, to cite myself, "not play nice with" that and no amount of rfc4301 compliance will fix that. My motivation was to remove the implicit argument expansion from "RFC4301 is no problem for L4S" to that the condition that will trigger that condition an SCE-compliant AQM on the tunneled/IPSEC path is no problem in the L4S design. Just because the L4S traffic itself does not suffer does not make that a situation that L4S properly deals with.

> 
> Do you accept that "RFC4301 is not problem for L4S", which was the sentence you "begged to differ" about?

	[SM] As above I was commenting upon the "RFC4301 is no problem for L4S" part of the sentence. SCE as far as I can see will only have issues if an AQM decides to SCE-mark an IPSEC packet AND the decapsulator strictly follows rfc4301 (which arguable it should). As Jonathan noted that leads to a lack of feed-back fidelity, but due to SCE-AQM still emitting CE will still be generally safe, unlike L4S behaviour in the same situation which will both ignore the SCE marks and will also mis-respond to the CE-marks.

> Do you accept (as Jonathan just has) that if an SCE AQM is within an RFC4301 tunnel, the tunnel endpoint will black-hole the SCE markings?

	[SM] Sure, I do.

> 
> Please answer both questions without distracting the thread onto another issue.

	[SM] Bob? That is rather rude, but fair enough I assume that we now share a common judgement on each other's styles.

> I am not trying to suppress discussion of other issues, I am trying to keep discussion of each issue on-topic.

	[SM] You might have noticed, that Ingemar did engage in the discussion already, so your admonition is a bit late, no?

Best Regards
	Sebastian

---

From: "[email-masked]" <[email-masked]>
Date: Sat, 4 Jan 2020 16:52:49 +0000 (UTC)

All the participants in these discussions (not just those who happen to be on the CC list of this thread)  would do well to remember that, no matter how irritated you are by the person you are replying to, most of the people reading your emails are not that person. Most of us here do our best to ignore personality clashes and extract any substantive technical points from these discussions, but indulging in high-school debating tactics makes this hard, and usually not in favour of the person indulging in them. 

Alex

---

From: Ingemar Johansson S <[email-masked]>
Date: Sat, 4 Jan 2020 19:10:23 +0000

Hi
Please see inline

/Ingemar

> Sent: den 4 januari 2020 16:30
> <ingemar.s.johansson=[email-masked]>; [email-masked]
> > On Jan 4, 2020, at 15:41, Ingemar Johansson S
> >> Bless <[email-masked]>; [email-masked]; [email-masked]
> >>>>> Yes I am aware that SCE works with RFC6040 tunnels, it does
> >>>>> however seem to work well with RFC4301 ditto.
> >>>>> deployed on the internet and whether this is a large problem for
> >>>>> SCE or not. I notice however that RFC4301 is no problem for L4S.
> >>>>
> >> not yield to the rfc3168 AQMs CE marks in the way that the AQM
> >> expects/is designed for, thereby crowding out the 1/sqrt(p) flows and
> >> securing
> > themselves
> >>> future (and actually present) I see algorithms like BBRv2 that
> >>> strive one BDP queue and will therefore not strive to bloat the
> >>> queues as much as the forementioned CC algos.  It seems to me like
> >>> there is incentive to go towards more of these. And as you point out
> >>> below there is work ongoing work by Bob and others to add RFC3168
> > compatibility to
> >> TCP Prague.
> >>
> >> forward to the data demonstrating that approach working as intended
> >> and as required (I am especially interested to see how long this is
> >> going to take
> > and
> >  a) delves into the flow rate fairness that has been discussed at
> > length in other mail treads.
> 
> 	[SM] I am puzzled, a) above is about how it mark/differentiate
> between flows that should be treated to 1/p versus those that should be
> treated to 1/sqrt(p) congestion signaling. IMHO this has exactly zero to
do
> with flow rate fairness and all with how to make the two types of
congestion
> responses coexist peacefully in the existing internet. But please
elaborate
> what you referred to above with "flow rate fairness".
> 	Really, the constant attacks against FQ solution are a RED HERRING
in
> the context whether L4S is ready to reach experimental stage. My issue is
> not that L4S might not be the theoretically best solution for the
challenge it
> was designed to solve, my issue is that practically the current
> implementations ARE NO SOLUTION at all, as they do not work reliably and
> robustly, period.
> 
> 
> 	[SM] Try the shoe on the other foot then, if the IETF would command
> to change the dual queue coupled AQM so that it gives the same
> priority/bandwidth advantage to non-L$S flows instead, would you still be
> okay with that? If your answer is yes, you can predict which changes to
dual
> queue coupled AQM I will propose next... I take it that you also consider
15
> ms the ideal ref-delay for the non-L4S's queue PI2 instance without being
> able to tell what makes 15 so magically wonderful in that context?
> 
> 
> > and
> >     this is something that relates more to how endpoint congestion
> > control algorithms are deigned
> 	[SM] Again I disagree, the dual queue coupled AQM's sole reason to
> exists is to equitably share between the two traffic classes the L4S
approach
> categorizes the world into. Failure to do so reliably and robustly is NOT
a
> function of the endpoints  but of your isolation mechanism (as an isolator
> that can be easily gamed by the endpoints does not even deserve that
> name).
> 	I realize that this is an inconvenient truth and that both Bob and
Koen
> always try to change this into an unrelated discussions about say RTT-
> independence, but that is simply a distraction strategy as I have alluded
to
> multiple times, not least because so far the L4S team has utterly failed
to
> present data demonstrating that they managed to make TCP Prague, their
> demonstration transport, RTT-independent enough to paper over the dual
> queue coupled AQM's design mistakes. But to be explicit, even if you
> manage to fix this for TCP Prague, unless you only admit TCP Prague flow
the
> the L4S queue you have won very little. If it apprears that I repeat
myself,
> then simply because part of my audience does not seem to be listening or
> engaging with my arguments at all.
> 
> 
> >  b) Means that a RCF3168 ECN capable flow whose packets have been CE
> > marked upstreams are put
> >    in the L4S queue in the DualQ downstreams and therefore causing
> > packet reordering for the RFC3168 flow.
> 
> 	[SM] Exactly, by mis-classifying CE packets into the L4S queue one
> risks out-of-order delivery for non-L4S flows, as a direct consequence of
> abusing a single codepoint as binary classifier. If this single codepoint
would
> be the only option this might be worth discussing/accepting, but since
there
> are quite a number of different classifier schemes ECT(1) really should be
> taken off the table IMHO.
> 	Except, that for the reordering to happen, you do not even need
> packets in both directions, even in one direction reordering can be
inflicted
> upon non-L4S flows, if a CE-marked packet hits the dual queue coupled AQM
> it will traverse the L4S queue and will be given priority over packets of
the
> same flow in the non-L4S queue, so as long as the non-L4S queue is not
> empty reordering can happen. That is a side-effect most easily dealt with
by
> employing a full independent bit for proper classification, or by
defaulting all
> CE-marked packets to the non-L4S queues (that way reordering is restricted
> to the new L4S flows and there s a real incentive to carefully look again
at the
> unfortunate choice of 15ms ref_delay for the non-L4S queue, a win-win
> situation form the perspective of non-L4S traffic).
> 
> 
> 	[SM] I am not sure that you need simultaneous congestion in both
> directions, and secondly, I believe this situation to be more
probable/likely
> than you seem to assume (this is partly a consequence of the often highly
> asymmetric links to end-users).
> 
> >     ii) congestion occurs upstreams and the uncongested DualQ
> > downstreams deliberately builds a large queue,
> 	As above for packet reordering you do not need simultaneous bi-
> directional congestion. My point however is that this again is one of the
> situations where the L4S team opted to make the non-L4S traffic pay all
the
> cost of the "impedance mismatch" between the current network and the
> "brave new world" that L4S promises.

[IJ] I believe that you misunderstood the scenario. It is not about
bidirectional congestion (or additional congestion in the reverse ACK path).
It is only about the congestion in the forward path. 
Consider a path between a sender and a receiver with two intermediate nodes
A and B, where node A is upstreams i.e closer to the sender. Both nodes
implement AQMs that can CE mark ECT packets. Node B implements a DualQ AQM
in which ECT(1) and CE marked packets are directed to the L4S queue. This
correctly means that CE marked packets for an RFC3168 flow will also go into
the L4S queue, which can cause packet reordering for the RFC3168 flow. 
For this to actually happen you need any of (repeat of my last email)
     i) congestion occurs simultaneously both on node A and B
          at least it should happen within the same RTT 
           I have so far seen this as quite unlikely 
     ii) congestion occurs in node A and the 
         uncongested DualQ in node B builds a (large) queue 
         I have not seen that DualQ does this.
How likely is it that any of these scenario happens ?

> >>>> indicating that the proposed scheme triggers fast enough (or even
> >>>> at all). I reserve judgement until I see the data.
> >>>>
> >>>>>
> >>>>> /Ingemar
> >>>>>
> >>>>>> To: Ingemar Johansson S <[email-masked]>;
> Bob
> >>>>> Briscoe
> >>>>>>> 2) Somewhere along the tunneled path an SCE compatible AQM
> >>>>>>> remarks from ECT '10' to SCE '01'.
> >>>>>>> become ignored and the queue will grow until packets are either
> >>>>>>> CE marked, or dropped.
> >>>>>>>
> >>>>>>>> The assertion that SCE traffic will "starve" is just plain
> >>>>>>>> wrong
> >>>>> in
> >>>>>>>> the following ...
> >>>>>>>>
> >>>>>>>>>> The SCE markings can be used independently whether you
> have
> >>>>>>> multiple
> >>>>>>>>> Packets from non-SCE and from SCE senders are
> >>>>>>>>> indistinguishable
> >>>>> by
> >>>>>>>>> Any transports that don't understand SCE drive the single
> >>>>>>>>> queue
> >>>>> to
> >>>>>>>> The reasoning appears to be sound up to this point, but the
> >>>>>>>> next sentence does not follow from the above:
> >>>>>>>>
> >>>>>>>>> This overruns the SCE
> >>>>>>>> operating point (defined by CE or drop)" because SCE traffic
> >>>>>>>> has
> >>>>> an
> >>>>>>>> RFC-3168-
> >>>>>>>> like response to CE marks, and the usual reaction to drops.
> >>>>>>>> The result is a deeper queue than would have resulted in the
> >>>>>>>> absence
> >>>>> of
> >>>>>>>> non-SCE traffic,
> >>>>>>> but
> >>>>>>>>> Cc: De Schepper, Koen (Nokia - BE/Antwerp); Ingemar
> Johansson
> >>>>> S;
> >>>>>>>>> perhaps your belief that SCE is "more evolvable" is just that
> >>>>>>>>> -
> >>>>> a
> >>>>>>>>> belief.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> The SCE markings can be used independently whether you
> have
> >>>>>>> multiple
> >>>>>>>>> Packets from non-SCE and from SCE senders are
> >>>>>>>>> indistinguishable
> >>>>> by
> >>>>>>>>> Any transports that don't understand SCE drive the single
> >>>>>>>>> queue
> >>>>> to
> >>>>>>>>> a deeper operating point (defined by CE or drop). This
> >>>>>>>>> overruns the SCE operating point, and causes SCE marking to
> >>>>>>>>> approach
> >>>>> 100% {Note
> >>>>>> 2}.
> >>>>>>>>>>>> FWIW, our delay-based approach has deployment problems
> at
> >>>>> the
> >>>>>>>>>>>>> <https://protect2.fireeye.com/v1/url?k=10e7b9ba-
> 4c6d6cbf-
> >>>>>>> 10e7f921
> >>>>>>>>> [BB] Let me repeat back the question, 'cos it seems rather
> >>>>>>>>> odd,
> >>>>> so
> >>>>>>>>>>>>> We have a session in the 'Prague' side-meeting tomorrow
> to
> >>>>>>>>>>>>> review
> >>>>>>>>> them
> >>>>>>>>>> See above, probably we want to put something into queue
> which
> >>>>> is
> >>>>>>>>> Having said that, I hope I (and Koen) have shown that L4S
> >>>>>>>>> still leaves scope for complementary delay-based CCs, not to
> >>>>>>>>> mention scope for different AQMs and for different FQ-based
> solutions.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Regards
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Bob
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Roland
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>
> >>>>>>>>> Briscoehttps://protect2.fireeye.com/v1/url?k=9834ac6d-
> c4be7968
> >>>>>>>>> -
> >>>>>>> 9834ecf
> >>>>>>>>> 6-861010bc36ff-d256b2de1d12a128&q=1&e=35be9991-3367-
> 47d3-
> >>>> b023-
> >>>>>>>> ed8501dd

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 5 Jan 2020 09:43:15 +0000

Hi
Please see inline

/Ingemar

> Sent: den 4 januari 2020 22:38
> more below in-line
> > On Jan 4, 2020, at 20:10, Ingemar Johansson S
> >>>>>> completely misinterpret the received CE marks and will not
> >>>>>> respond appropriately with a sufficiently large reduction of the
> >>>>>> congestion window*. The currently still preferred solution to
> >>>>>> this issue, if I understand Bob correctly, is still to reason
> >>>>>> that such ECN-enabled AQMs along tunneled path segments (which
> I
> >>>>>> understand to be always inside the core of an AS as towards the
> >>>>>> edges packets get
> >>>> expects/is designed for, thereby crowding out the 1/sqrt(p) flows
> >>>> and securing
> >>> themselves
> >>>>> there is incentive to go towards more of these. And as you point
> >>>>> out below there is work ongoing work by Bob and others to add
> >>>>> RFC3168
> >>> compatibility to
> >>>> TCP Prague.
> >>>>
> >>>> 	[SM] Note that RFC8290 is not really SCE capable to begin with
> >>>> (how could it, it predates the SCE drafts considerably). Be that as
> >>>> it may,
> >>> IMHO it
> >> between flows that should be treated to 1/p versus those that should
> >> be treated to 1/sqrt(p) congestion signaling. IMHO this has exactly
> >> zero to
> > do
> >> the context whether L4S is ready to reach experimental stage. My
> >> issue is not that L4S might not be the theoretically best solution
> >> for the
> > challenge it
> >> implementations ARE NO SOLUTION at all, as they do not work reliably
> >> and robustly, period.
> >>
> >>
> >> priority/bandwidth advantage to non-L$S flows instead, would you
> >> still be okay with that? If your answer is yes, you can predict which
> >> changes to
> > dual
> >> ms the ideal ref-delay for the non-L4S's queue PI2 instance without
> >> being able to tell what makes 15 so magically wonderful in that
context?
> >> function of the endpoints  but of your isolation mechanism (as an
> >> isolator that can be easily gamed by the endpoints does not even
> >> deserve that name).
> >> always try to change this into an unrelated discussions about say
> >> RTT- independence, but that is simply a distraction strategy as I
> >> have alluded
> > to
> >> present data demonstrating that they managed to make TCP Prague,
> >> their demonstration transport, RTT-independent enough to paper over
> >> the dual queue coupled AQM's design mistakes. But to be explicit,
> >> even if you manage to fix this for TCP Prague, unless you only admit
> >> TCP Prague flow
> > the
> >> then simply because part of my audience does not seem to be listening
> >> or engaging with my arguments at all.
> >>
> >>
> >> risks out-of-order delivery for non-L4S flows, as a direct
> >> consequence of abusing a single codepoint as binary classifier. If
> >> this single codepoint
> > would
> >> are quite a number of different classifier schemes ECT(1) really
> >> should be taken off the table IMHO.
> >> upon non-L4S flows, if a CE-marked packet hits the dual queue coupled
> >> AQM it will traverse the L4S queue and will be given priority over
> >> packets of
> > the
> >> same flow in the non-L4S queue, so as long as the non-L4S queue is
> >> not empty reordering can happen. That is a side-effect most easily
> >> dealt with
> > by
> >> CE-marked packets to the non-L4S queues (that way reordering is
> >> restricted to the new L4S flows and there s a real incentive to
> >> carefully look again
> > at the
> >> than you seem to assume (this is partly a consequence of the often
> >> highly asymmetric links to end-users).
> >>
> >> directional congestion. My point however is that this again is one of
> >> the situations where the L4S team opted to make the non-L4S traffic
> >> pay all
> > the
> 	[SM] Indeed I misread your upstream downstream references as
> denoting the general traffic direction and not simply the position in
relation
> to the L4S node. Thanks for clearing this up and persisting through my
block-
> headedness.
> 
> 
> > Consider a path between a sender and a receiver with two intermediate
> > nodes A and B, where node A is upstreams i.e closer to the sender.
> > Both nodes implement AQMs that can CE mark ECT packets. Node B
> > implements a DualQ AQM in which ECT(1) and CE marked packets are
> directed to the L4S queue.
> > This correctly means that CE marked packets for an RFC3168 flow will
> > also go into the L4S queue, which can cause packet reordering for the
> RFC3168 flow.
> 
> 	[SM] Yes, I agree.
> 
> 	[SM] Currently the likelihood is close to zero, by virtue of the
lack of
> L4S AQMs in the wild, so yes this is unlikely. But I for example use an
rfc-3168
> compliant AQM on ingress and egress of my network and hence ECN-
> enabled flows (on egress) will see CE-marks when leaving slow-start and
> later in their life time (when ever they exceed their share of the link).
If an
> L4S AQM is instantiated on say a peering link (as envisioned or at least
hinted
> at by some of the L4S design descriptions) congestion on that link can and
will
> be quite decoupled from congestion and ECN-marking on my own access
> link. Can I put a number to quantify this? No, I can not, at least not any
> believable number more precise than non-zero.
> 
> 
> 	[SM] If two (or more) packets of a single flow arrive back to back
at
> the L4S AQM with the last CE-marked, can you guarantee that that the dual
> queue coupled AQM will not reorder them even without saturating load,
> after all it is designed to give the L4S-queue (limited) priority over the
non-
> L4S queue?

[IJ] I cannot guarantee anything, my observation is however that possible
packet bursts are very likely ironed out in node A as it CE marks packets
for the RFC3168 and is therefore congested. This should mean that the queues
in node B should be empty and packets will be forwarded as they arrive.
Still there is a likelihood that packets arriving in the L4S buffer are
scheduled before packets arriving in the RFC3168 buffer and that can cause
packet reordering for the RFC3168 flow. 
The questions are :
 * How large will the reordering depth be?, my gut feeling is that this will
be in the order of sub milliseconds but I can only speculate. 
 * Is this worse than the reordering that we see already today ?
 * Is the reordering depth more than RACK can handle ?

> 	[SM] This is a very good question for which I am I hope to get an
> answer from real experimentally acquired data by the L4S proponents.
> Please note that since rfc3168 will probably stay in effect there is
simply no
> guarantee that this scenario might not become more probable in the future.

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 5 Jan 2020 12:11:13 +0100

Hi Ingemar,

more below in-line.

To anybody not interested in the detailed discussion below, please ignore this e-mail, thanks.

> On Jan 5, 2020, at 10:43, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
> Please see inline
> 
> /Ingemar
> 
> [IJ] I cannot guarantee anything,

	[SM] That is a problem, don't you agree? This is something that is empirically testable, the fact that is seems to have not been tested yet, shows that there are still avenues available to kick L4S's tires in the lab and that (experimental) roll-out into the wider internet seems premature.

> my observation is however that possible

	[SM] But as you conceded, we have no idea whether that is actually sufficient for the dual queue coupled AQM to not re-order packets.

> be in the order of sub milliseconds but I can only speculate. 

	[SM] The way TCP works the number of milliseconds seems less important than the maximum out of sequence-ness, I think about dupACKs here. Sure for the case where only a single CE-marked packet, jumps the queue nothing really will happen (as there will only be one dupACK and after that cumulative ACKs should increase again), except that the flow's tolerance against re-ordering along the rest of the path is now compromised... But the fact remains that this compromise for non-L4S flows only exists, because the ECT(1) as classifier breaks down for CE-marked packets; how about defaulting all CE-marked packets to the non-L4S queue instead, in that case L4S-traffic woulld pay the price for L4S' unfortunate choice of classifier (but due to recommending RACK L4S traffic should be more robust against re-ordering anyway)...

> * Is this worse than the reordering that we see already today ?

	[SM] In the limit yes, because we currently have this additional re-ordering "source" in the internet. 

> * Is the reordering depth more than RACK can handle ?

	[SM] How is that going to help standards-compliant TCPs not employing RACK? RACK, as far as I can tell recommends to keep the permissible re-ordering window <= the unloaded RTT*, so at short (and low bandwidth) paths even a few packets worth of re-ordering might cause issues even with RACK.

Best Regards
	Sebastian

*) https://tools.ietf.org/html/draft-ietf-tcpm-rack-06
"4.  The RACK reordering window MUST be bounded and this bound SHOULD
       be one round trip."

---

From: Ingemar Johansson S <[email-masked]>
Date: Sun, 5 Jan 2020 14:25:42 +0000

Hi Sebastian

I believe that I don't get any further so it is perhaps best to finalize
this tread?

There is a long list of things that can possibly go wrong with L4S. The
problem is that we don't know how serious they are if they are a problem at
all. 
L4S is chartered as an experiment and everyone is welcome to participate in
the work and verify its function in experiments. If you have concerns with
particular scenarios with L4S, please run experiments to verify these. And
if you find ways to improve things, then please suggest them to the working
group. 
The problem with yet a parallel experiment with the ECN bits (that I
understand that you want with SCE?)  is that it risks killing off the
usefuleness with ECN completely. Nobody wins in such a scenario as I see it,
except perhaps proponents of more complex and costly in-band or out-of-band
signaling to solve similar problems that L4S seek to solve.

I believe active work with L4S and contribution to it is much better use of
time and energy than the trench-war that we currently engage in. So my
suggestion is to first engage in the L4S work and see where it ends. There
is interest in L4S outside IETF, I will personally engage in L4S experiments
in 5G access technology where we see a need for it in it for services like
online gaming AR/VR and remote control.

Regards
/Ingemar

> Sent: den 5 januari 2020 12:11
> > On Jan 5, 2020, at 10:43, Ingemar Johansson S
> >>>>>>>>> deployed on the internet and whether this is a large problem
> >>>>>>>>> for SCE or not. I notice however that RFC4301 is no problem for
> L4S.
> >>>>>>>>
> >>>>>>>> respond appropriately with a sufficiently large reduction of
> >>>>>>>> the congestion window*. The currently still preferred solution
> >>>>>>>> to this issue, if I understand Bob correctly, is still to
> >>>>>>>> reason that such ECN-enabled AQMs along tunneled path
> segments
> >>>>>>>> (which
> >> I
> >>>>>>> queues as much as the forementioned CC algos.  It seems to me
> >>>>>>> like there is incentive to go towards more of these. And as you
> >>>>>>> point out below there is work ongoing work by Bob and others to
> >>>>>>> add
> >>>>>>> RFC3168
> >>>>> compatibility to
> >>>>>> TCP Prague.
> >>>>>>
> >>>>>> forward to the data demonstrating that approach working as
> >>>>>> intended and as required (I am especially interested to see how
> >>>>>> long this is going to take
> >>>>> and
> >>>>>> 	I consider this to be a side-effect/consequence of
redefining
> >>>>>> what a
> >>>>>> (how could it, it predates the SCE drafts considerably). Be that
> >>>>>> as it may,
> >>>>> IMHO it
> >>>>>> 	The more relevant question for me is what it would take to
> >>>>>> remedy these two obvious issues in the L4S design... and what
> >>>>>> would it take in
> >>>>> addition
> >>>> between flows that should be treated to 1/p versus those that
> >>>> should be treated to 1/sqrt(p) congestion signaling. IMHO this has
> >>>> exactly zero to
> >>> do
> >>>> implementations ARE NO SOLUTION at all, as they do not work
> >>>> reliably and robustly, period.
> >>>>
> >>>>
> >>>> 	[SM] Try the shoe on the other foot then, if the IETF would
> >>>> command to change the dual queue coupled AQM so that it gives the
> >>>> same priority/bandwidth advantage to non-L$S flows instead, would
> >>>> you still be okay with that? If your answer is yes, you can predict
> >>>> which changes to
> >>> dual
> >>>> even if you manage to fix this for TCP Prague, unless you only
> >>>> admit TCP Prague flow
> >>> the
> >>>> then simply because part of my audience does not seem to be
> >>>> listening or engaging with my arguments at all.
> >>>>
> >>>>
> >>>>> b) Means that a RCF3168 ECN capable flow whose packets have been
> >>>>> CE marked upstreams are put  in the L4S queue in the DualQ
> >>>>> downstreams and therefore causing packet reordering for the
> >>>>> RFC3168 flow.
> >>>>
> >>>> upon non-L4S flows, if a CE-marked packet hits the dual queue
> >>>> coupled AQM it will traverse the L4S queue and will be given
> >>>> priority over packets of
> >>> the
> >>>> unfortunate choice of 15ms ref_delay for the non-L4S queue, a
> >>>> win-win situation form the perspective of non-L4S traffic).
> >>>>
> >>>>
> >>>> directional congestion. My point however is that this again is one
> >>>> of the situations where the L4S team opted to make the non-L4S
> >>>> traffic pay all
> >>> the
> >>>> cost of the "impedance mismatch" between the current network and
> >>>> the "brave new world" that L4S promises.
> >>>
> >>> Consider a path between a sender and a receiver with two
> >>> intermediate nodes A and B, where node A is upstreams i.e closer to
the
> sender.
> >> enabled flows (on egress) will see CE-marks when leaving slow-start
> >> and later in their life time (when ever they exceed their share of the
link).
> >> link. Can I put a number to quantify this? No, I can not, at least
> >> not any believable number more precise than non-zero.
> >>
> >>
> >> the L4S AQM with the last CE-marked, can you guarantee that that the
> >> dual queue coupled AQM will not reorder them even without saturating
> >> load, after all it is designed to give the L4S-queue (limited)
> >> priority over the
> > non-
> >> L4S queue?
> >
> 	[SM] That is a problem, don't you agree? This is something that is
> empirically testable, the fact that is seems to have not been tested yet,
> shows that there are still avenues available to kick L4S's tires in the
lab and
> that (experimental) roll-out into the wider internet seems premature.
> 
> 
> > my observation is however that possible packet bursts are very likely
> > ironed out in node A as it CE marks packets for the RFC3168 and is
> > therefore congested. This should mean that the queues in node B should
> > be empty and packets will be forwarded as they arrive.
> 
> 	[SM] But as you conceded, we have no idea whether that is actually
> sufficient for the dual queue coupled AQM to not re-order packets.
> 
> 
> > Still there is a likelihood that packets arriving in the L4S buffer
> > are scheduled before packets arriving in the RFC3168 buffer and that
> > can cause packet reordering for the RFC3168 flow.
> > The questions are :
> > * How large will the reordering depth be?, my gut feeling is that this
> > will be in the order of sub milliseconds but I can only speculate.
> 
> 	[SM] The way TCP works the number of milliseconds seems less
> important than the maximum out of sequence-ness, I think about dupACKs
> here. Sure for the case where only a single CE-marked packet, jumps the
> queue nothing really will happen (as there will only be one dupACK and
after
> that cumulative ACKs should increase again), except that the flow's
tolerance
> against re-ordering along the rest of the path is now compromised... But
the
> fact remains that this compromise for non-L4S flows only exists, because
the
> ECT(1) as classifier breaks down for CE-marked packets; how about
> defaulting all CE-marked packets to the non-L4S queue instead, in that
case
> L4S-traffic woulld pay the price for L4S' unfortunate choice of classifier
(but
> due to recommending RACK L4S traffic should be more robust against re-
> ordering anyway)...
> 
> 	[SM] How is that going to help standards-compliant TCPs not
> employing RACK? RACK, as far as I can tell recommends to keep the
> permissible re-ordering window <= the unloaded RTT*, so at short (and low
> bandwidth) paths even a few packets worth of re-ordering might cause
> issues even with RACK.
> 
> Best Regards
> 	Sebastian
> 
> 
>

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 5 Jan 2020 16:55:40 +0100

Hi Ingemar,

please indulge one final response and then I let it rest, but feel free to give a final comment.

some parting words below in-line.

Everybody else, please ignore the rest.

> On Jan 5, 2020, at 15:25, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi Sebastian
> 

	[SM] And this is why the L4S team would do wonders for their cause if they would actually attempt to quantify these problems. So far I see more of a "probably insignificant so let's ignore this" kind of response.

> particular scenarios with L4S, please run experiments to verify these.

	[SM] I have been invited multiple times by different members of the L4S team to pick the hot chestnuts out of the fire already, as before I will pass.

> And

	[SM] As I have considering the ref_delay of 15ms in the non-L4S queue's PI2 instance, only to be ridiculed by the L4S members that did not even seemed to bothered to actually follow my recommendations. (Short note to the accidental reader: the 15ms reference delay specified for the non-L4S queue is 10ms too long for the targeted default 100ms RTT flow and severely compounds the systemic bias of the dual queue coupled AQM in favor of L4S over the non-L4S queue, to the tune of 1:8 to 1:15 ratios of non-L4S to L4S queue thoughput).

> understand that you want with SCE?)

	[SM] Ingemar, I mostly argue about L4S for a reason, I see discrepncies between L4S's declared goals and requirements and what the current implementation actually achieves. In my opinion L4S fall short to achieve its own goals andI try to judge it on its own merits. When I drag SCE into the discussion, then mainly as an example with working code that demonstrates that some of the problems L4S faces can be over-come quite elegantly. But my main argument is simply the "emperor has no cloths" aka the current implementation of the dual queue coupled AQM does not seem fit to merit experimental status (and more importanatly roll-out into the wider internet). I am not understanding this as a discussion of L4S versus L4S but rather L4S versus the real existing internet.

>  is that it risks killing off the

	[SM] ATM the clever ideas behind the L4S default AQM fall short, so rolling this out as is will exactly lead to such a nobody wins scenario you predict. 

> time and energy than the trench-war that we currently engage in.

	[SM] That certainly looks like an option, but given that Bob will even fight simple word substitutions in the I.D.s nail and tooth that have zero relevance for the actual L4S implementation, I fear that offering help will be a fool's errand.

> So my
> suggestion is to first engage in the L4S work and see where it ends.

	[SM] I tried to some degree, by openly talking about issues, only to find myself not in an open discussion about the issues but in a somewhat silly debate (mostly with Bob who is not willing to cede a single point, even when he is clearly out of his area of competence (I do not claim to be an expert in the theory behind CoDel, but Bob obviously has even less domain knowledge)). Since I consider debates entertaining but not really productive my attempt to engage in the L4S work is stalled.

> There
> is interest in L4S outside IETF,

	[SM] In the promises/goals of L4S or in the exact implementation?

> I will personally engage in L4S experiments

	[SM] And I will not attempt to stop you, But I will also not stop to argue to do my best what ever version of L4S that is finally rolled-out will have as little negative side-effects for the rest of the internet traffic out there. I am fully behind the idea to enable 1/p type congestion controllers over the wider internet and I am also a proponent for keeping latencies low, but I do not think that current L4S proposals are a viable way to get there. I understand and accept that it will be a long time until 1/p CC will "take over the internet" no matter how attractive and obvious they look to us, so the incremental deployment without side-effects component of that 1/p system is IMHO a very important issue, and so far L4S has too many side-effects designed in for my taste.

Best Regards
	Sebastian