Subject: Re: [tsvwg] ECN as a classifier (was: L4S vs SCE)

From: Sebastian Moeller <[email-masked]>
Date: Wed, 1 Jan 2020 10:00:54 +0100

Hi Bob,

more below in-line.

> On Dec 31, 2019, at 17:55, Bob Briscoe <[email-masked]> wrote:
> 
> Roland,
> 
> I just noticed that Koen's response (which I agreed with) snipped some of your responses that I ought to have addressed.
> For this one, I've forked a new thread...
> 
> On 22/11/2019 02:32, Roland Bless wrote:
>> Hi Bob,
>> 
>> see inline.
>> 
>> On 21.11.19 at 15:44 Bob Briscoe wrote:
>> 
>>> On 21/11/2019 09:32, Roland Bless wrote:
>>> 
>>>> On 21.11.19 at 19:34 Bob Briscoe wrote:
>>>> 
>>>>> On 20/11/2019 21:22, Roland Bless wrote:
>>>>> 
>> I find it architecturally cleaner to have an additional ECN codepoint
>> used for indicating "SCE" rather than (ab)using it as classifier for the
>> dualQ AQM.
>> 
> 
> The idea that it is abuse to use ECN as a classifier has never been justified, only asserted.

	[SM] Using ECT(1) as a classifier is sub-optimal for a number of reasons, that I will justify here for your entertainment:
1) For classifying between two classes one requires 2 states, or one full bit, ECT(1) however does not meet that criterion, it is a single code-point or colloquially speaking half a bit. Why the theory that ECT(1) can serve as a "classifier" is still maintained in spite of its objective insufficiencies boggles the mind. RFC3168 has an elucidating section giving rationale for the use of two full bits for ECNs two additional states instead of using just a single bit, that shines a pretty good light on this issue and why overloading too much information into a single bit has severe issues.
	Jusy in passing, in L4S classification breaks for packets carrying a CE mark, L4S proposes to carry all of these in the L4S-queue thereby risking increasing packet re-ordering for mis-classified non-L4S flows, fitting well in the L4S approach's tendency to move all the costs of dealing with the "impedance mismatch" between standards-compliant and L4S traffic onto standards-compliant traffic. This btw, is not a slur, this is an objective observation based on the L4S drafts.

2) In essence L4S defines a specific per-hop-behaviour for L4S-marked traffic, and for that purpose there are additional 6 bits reserved in the IPv4 and IPv6 headers, the diffserv bits. Let's have a look at rfc3168 on that topiic:
"One possibility might have been for the data sender to use the fourth
   ECN codepoint to indicate an alternate semantics for ECN.  However,
   this seems to us more appropriate to be signaled using a
   differentiated services codepoint in the DS field."
To which a number of reviewers already concurred.

So I justify the judgment that using ECT(1) as "classifier" is ABUSE, by two lines of argument:
a) does not really work well enough (it fails to act as a classifier under load)
b) it uses an ECN field for a purpose for which dedicated bits are already reserved.

@CHAIRS: For the potential experimental roll-out I propose a requirement/MUST that independent of the ECT(1) code point all L4S-senders also need to set a to-be-defined DSCP, so that the existing filtering and (de-)prioritization infrastructure in the field can be used to implement the desired PHB for L4S traffic.

Best Regards
	Sebastian

> 
> The actual position is pretty much the opposite. Below I explain why L4S follows the original way the architecture of the ECN field was defined. Whereas the way SCE uses the ECN field for 2-level marking is counter to the original ECN architecture. 
> 
> The ECN field is unlike any other field in the IP header, because it conveys a value from the sender down the layers to L3 or L2 and it conveys a 'return' value from any node in the network back up the stack to the receiving endpoint.
> 
> Originally, in RFC3168:
> 	• for the sender->network interface, the ECN field was defined as a way for the sender to classify a packet between different code paths in AQM algorithms (drop for 00 vs. marking for the other 3 codepoints). Certainly, it wasn't /envisaged/ that it would be used to classify between different queues. But /architecturally/ there is no difference. 
> 	• for the network->receiver interface, the two semantically identical ECT(X)->CE transitions were defined as the only way for the network to signal congestion.
> The two original standards track RFCs that originally defined ECN tunnelling [RFC3168] & [RFC4301] were both incompatible with 2-level marking like SCE. They required the decapsulation at the end of a tunnel to forward the inner header's ECN field unless the outer was CE. So they would strip any SCE signal applied to the outer header.
> 
> Around 2010, when I was rationalizing these two diverging tunnel definitions into one [RFC6040] (also standards track), I noticed that precluding 2-level marking was unnecessarily restrictive. So we defined the way ECN propagates to the forwarded header from the outer so that future tunnels would support either meaning of ECT(1): either equivalent to ECT(0) or a weaker congestion signal than CE, like SCE. 
> 
> However, the original RFC3168 ECN architecture (that is incompatible with SCE) remains and will remain more widely supported in the Internet, probably for decades. Indeed, even today, the RFCs to make that change to all encapsulations (not just IP-in-IP tunnels) are not past the WG stage yet.
> 
> Tunnels and encapsulations are everywhere. Jonathan M says it doesn't matter if the SCE signal gets black-holed in tunnels and encapsulations. Eventually new ones compatible with the SCE architecture will be deployed everywhere (which will take decades). But it does matter - because SCE wants to claim the last available ECN codepoint, which L4S has already developed to give consistently low latency - within the original ECN architecture. 
> 
> 
> Regards
> 
> 
> 
> Bob
> 
>> Roland
>> 
>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               
> http://bobbriscoe.net/

---

From: "Black, David" <[email-masked]>
Date: Wed, 1 Jan 2020 23:31:59 +0000

Bob,
(posting as an individual)

The assertion that SCE traffic will "starve" is just plain wrong in the following ...

> > The SCE markings can be used independently whether you have multiple
> > queues or just a single queue.
> [BB] This seems to be the opposite of reality. Perhaps you've used the
> word 'can' because you think it might be possible. But there's good
> reason to doubt that. As follows...
> 
> Packets from non-SCE and from SCE senders are indistinguishable by just
> looking at them. So, if there's a single queue, they have to be mixed
> together in it {Note 1}. But a single queue can only have one length.
> Any transports that don't understand SCE drive the single queue to a
> deeper operating point (defined by CE or drop).

The reasoning appears to be sound up to this point, but the next sentence does not follow from the above:

> This overruns the SCE
> operating point, and causes SCE marking to approach 100% {Note 2}. Then
> those transports that do understand SCE will starve.

Actually, it causes both SCE and non-SCE senders to operate at that "deeper operating point (defined by CE or drop)" because SCE traffic has an RFC-3168-like response to CE marks, and the usual reaction to drops.  The result is a deeper queue than would have resulted in the absence of non-SCE traffic, but the result is definitely *not* starvation.

Thanks, --David

> -----Original Message-----
> From: Bob Briscoe <[email-masked]>
> Sent: Tuesday, December 31, 2019 6:11 PM
> To: Roland Bless
> Cc: De Schepper, Koen (Nokia - BE/Antwerp); Ingemar Johansson S; Kyle
> Rose; [email-masked]; [email-masked]
> Subject: Re: [tsvwg] L4S vs SCE (Evolvability)
> 
> 
> [EXTERNAL EMAIL]
> 
> Roland,
> 
> As I just said, I noticed that Koen's useful response snipped some of
> your responses about evolvability that I ought to have addressed.
> Sorry, you'll need to reload state from November...
> 
> >> I'm not getting it....
> >>>>> Yes, but as I also expressed my concerns w.r.t. the L4S codepoint
> >>>>> earlier, at the cost of binding this to a quite fixed set of L4S
> >>>>> behaviors and "burning" the last ECT codepoint. Personally, I like
> >>>>> concepts with a little bit more potential to be useful for future
> >>>>> development (evolvability) of congestion controls, e.g., BBRv2 and
> >>>>> LoLa could also benefit from an SCE-like marking...
> >>>>>
> [BB] BBRv2 has not used SCE, but it has used L4S marking. And the
> authors of the LoLa draft have not used SCE but they have merged with
> the NQB draft, so that they can work with L4S as well. So perhaps your
> belief that SCE is "more evolvable" is just that - a belief.
> 
> 
> >> This last sentence... please really spell it out what you could possibly
> >> be meaning here. What has made you suddenly think this particular
> >> marking scheme has got magical properties that bestow evolvability? I'm
> >> serious, if you can't explain why you've said a sentence like this, it
> >> implies you are under the spell of some cult or fad.
> > Not sure what you are trying to indicate by your last sentence, but
> > sure, I can elaborate a bit on my last sentence.
> [BB] I just forked another thread 'ECN as a classifier' to address that
> assertion.
> > The SCE markings can be used independently whether you have multiple
> > queues or just a single queue.
> [BB] This seems to be the opposite of reality. Perhaps you've used the
> word 'can' because you think it might be possible. But there's good
> reason to doubt that. As follows...
> 
> Packets from non-SCE and from SCE senders are indistinguishable by just
> looking at them. So, if there's a single queue, they have to be mixed
> together in it {Note 1}. But a single queue can only have one length.
> Any transports that don't understand SCE drive the single queue to a
> deeper operating point (defined by CE or drop). This overruns the SCE
> operating point, and causes SCE marking to approach 100% {Note 2}. Then
> those transports that do understand SCE will starve.
> 
> In contrast, L4S works in dual queues or multiple queues (where 'works'
> means it gives very low latency without losing utilization).
> 
> Also, there is considerable scope for evolvability with both:
> * the dualQ is a framework that is intended to encompass different AQMs
> in future.
> * and there is considerable scope for developing the ways FQ uses L4S.
> 
> {Note 1}: Unless L4 identifiers are used to separate out microflows, and
> identify low latency flows by their behaviour. But then it's not really
> a single queue, unless you make it look like a single queue, but still
> treat it like multiple queues, as in LFQ. However, schemes like LFQ are
> "single-queue" in name only. They still have all the downsides of
> multiple queues.
> 
> {Note 2}: Unless the SCE operating point is hardly any shallower than
> that for CE, in which case SCE gives hardly any improvement.
> 
> > If one leaves the coupling aside the
> > algorithm for marking will probably not differ so much from what is
> > proposed in L4S.
> [BB] You still haven't explained (rather than asserted a belief) why the
> choice between SCE and L4S has anything to do with evolvability.
> 
> >>>> My whole purpose in solving the problem of deploying scalable CCs
> over
> >>>> the Internet was to re-juvenate evolution (to widen the range of
> >>>> applications that could be supported by different transport behaviours,
> >>>> particularly for real-time with low latency and high throughput at the
> >>>> same time). One of the main things that has stopped CCs evolving so
> far
> >>>> is the need for friendliness with the Reno behaviour that was not
> >>>> scaling over the years.
> >>> FWIW, our delay-based approach has deployment problems at the
> >>> other end of the spectrum: it gets suppressed by loss-based CCs...
> >>> (and we don't want to sacrifice our low delay goal).
> >>>
> >>>> If SCE is primarily supported in FQ AQMs, that will constrain flows to
> >>>> be capped at the rate that FQ gives them. How is that doing anything
> >>> I was solely referring to the marking behavior of SCE, not a particular
> >>> implementation based on FQ-AQMs.
> >> This implies you believe all that silliness about a shallower SCE
> >> threshold not starving an SCE flow in a single queue, because it makes
> >> SCE flows not want to use the queue.
> > I do not understand what you are saying here.
> [BB] I've now explained this above (after "But there's good reason to
> doubt that...").
> 
> The silliness I'm referring to is the statement by Jonathan M in his
> tsvwg talk at IETF-106 that CNQ is backward compatible with RFC3168
> traffic, because its performance for SCE traffic when mixed with 3168 is
> bad enough that it discourages use of SCE alongside 3168.
> 
> >>>> other than massively constraining future evolution of CCs, especially
> >>>> real-time ones? See Per-Flow Scheduling and the End-to-End
> Argument
> >>>> <http://bobbriscoe.net/projects/latency/per-flow_tr.pdf>. I don't
> need
> >>>> to tell you that the e2e argument is all about giving end systems the
> >>>> power to innovate without permission.
> >>>> Anyway, what are you imagining would stop CCs evolving alongside
> other
> >>>> scalable CCs? In much the same way CCs have always evolved. With L4S
> you
> >>>> have a clean slate that seems just like a FIFO with a shallow ECN-only
> >>>> immediate AQM. And other flows are causing you hardly any delay and
> very
> >>>> rarely any loss. Think of all the things you can do with that. Go
> >>>> evolve, Roland!
> >>> I already said that the Dual Queue AQM is nice, but comes with the
> >>> problem due to its built-in dropping law. Other CCs may react differently
> >>> and BBR was one example of newer CCs that did not work in the L4S
> queue
> >>> without further adaptation.
> >> The built in dropping law is for the old traffic ('pre-evolved' if you
> >> like) that doesn't respond to anything else but loss. That's what
> >> drop-based AQMs were for - to fool Reno etc into thinking they had hit
> >> the buffers, so to speak.
> > Yes, I understand that. But what happens if the classic traffic
> > once vanishes and we only have low-delay congestion control and
> > want to re-use the "second" queue for other purposes, e.g., using
> > it for flows that do not use loss-based CC?
> [BB] Let me repeat back the question, 'cos it seems rather odd, so
> perhaps I've misunderstood.
> 
> You want me to assume that the classic behaviour of repeatedly seeking
> out more capacity until loss occurs doesn't exist on the Internet any
> more. There's only low-delay CC, by which I think you mean CC based on
> L4S ECN (but I'm not sure that's your meaning).
> 
> Then, it sounds like everything would be working well. So why do you
> want to re-use the "second" queue for other purposes? Nothing is wasted
> if you don't use it. It has no capacity associated with it. it's just
> some lines of code sitting there in case packets appear from a classic CC.
> 
> 
> >>>    > The key thing here is the wording of the Prague requirements
> >>>> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08#section-4>.
> >>>> We have a session in the 'Prague' side-meeting tomorrow to review
> them
> >>>> (and I encourage this on this list too).
> >>>>
> >>>> Later down the line, if the L4S experiment is successful, there will be
> >>>> an opportunity to review these requirements if a standards track doc
> >>>> replaces the experimental (it is easier to relax CC requirements than
> >>>> tighten them). So, for the expt track, the requirements are designed to
> >>>> protect competing flows from harm in a tighter way than you might find
> >>>> in RFC5033 or similar.
> >>>>
> >>>>
> >>>> Nonetheless, even the 1/p isn't tightly spec'd, quoting:
> >>>>
> >>>>      The inverse proportionality requirement above is worded
> >>>>      as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility
> >>>>      when defining these specifications.
> >>> Yes, but something has to be implemented (in hardware) and therefore
> it
> >>> is fixed for some time and it should be consistent along a path, so I
> >>> don't see a viable path for evolvement of this coupling law...
> >> Why does the coupling law need to evolve? It's a relationship between
> >> something invariant (scalable) and the past (which is over, by
> >> definition, so it's not going to change now).
> > See above, probably we want to put something into queue which is
> > using a non-loss-based congestion control and/or we need to change
> > the (1/p,1/p^2) marking.
> [BB] The queue doesn't induce loss, the CC does. The queue isn't 1/p or
> 1/p^2 or whatever, the different CCs are. So, if a CC is using the C
> queue without inducing any loss (e.g. a delay-based CC might be able to
> keep the queue under the AQM's target delay), then the coupling won't
> couple any marking across to the L queue.
> 
> But I can't imagine why a delay-based CC that can keep the queue below
> the target of the AQM would want to classify itself into a queue
> designed for CCs that need to induce a decent queue (in order to
> maintain full utilization), given such queue-inducing traffic might
> start up at any time.
> 
> >> If you want to evolve, you select what's best for you - probably the
> >> nice clean L4S ECN queue. I still don't get what sort of evolution you
> >> can't do? Evolvability isn't about a researcher being able to do what
> >> they'd like.
> > I also don't understand it in this way, but investigating
> > invariables and degrees of freedom prudently may enable or facilitate
> > deployment of new stuff. If that new stuff cannot be deployed
> > it never gets a chance of being weeded out later either.
> [BB] If you have something specific in mind, please say it. This is all
> getting rather abstract.
> 
> This does make me want to question your notion of evolvability. Usually,
> when we make sure the Internet supports evolvability, we mean
> evolvability of new applications. We don't mean the ability for
> researchers to come up with different ways to solve problems that are
> already solved. There seems to be a hint of a complaint in your emails
> that L4S doesn't leave room for researchers to solve the same problem
> differently. That sort of evolvability is rather a luxury isn't it? If
> one approach solves an enduring problem with the Internet, it would be
> rather churlish to say "No, we can't use this solution, because it might
> preclude some ideas that might lead to other solutions to the same
> problem in the future, but we're not sure."
> 
> Having said that, I hope I (and Koen) have shown that L4S still leaves
> scope for complementary delay-based CCs, not to mention scope for
> different AQMs and for different FQ-based solutions.
> 
> 
> Regards
> 
> 
> 
> Bob
> 
> 
> 
> >
> > Roland
> >
> 
> --
> Bob Briscoehttp://bobbriscoe.net/

---

From: Greg White <[email-masked]>
Date: Tue, 7 Jan 2020 00:09:35 +0000

David,
See [GW] below.
Greg

On 1/1/20, 4:33 PM, "tsvwg on behalf of Black, David" <[email-masked] on behalf of [email-masked]> wrote:

    Bob,

    

[GW] Is that true?  Setting aside Bob's "Note 2" implementation, it seems your statement would only be true if the SCE sender had /no/ cwnd response to SCE-marks (i.e. ECT(1)) at all (which I guess would make it a non-SCE sender). I know the SCE congestion controller is still under development, but I seem to remember Jonathan indicating that it would aim to reduce its congestion window until the SCE-mark rate was 50% (don't quote me on that being the precise behavior currently).   If the non-SCE senders continue to drive the SCE-mark rate to 100% in order to experience occasional CE-marks, then the SCE senders would continue to reduce their congestion window on every RTT.   I suppose one way to (somewhat) get the behavior you are describing would be for an SCE sender to, upon receiving a single CE-mark, revert to an RFC3168 compliant mode and stop responding to SCE-marks.  I don’t believe that is the behavior today.

    Thanks, --David

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 7 Jan 2020 08:49:47 +0100

Hi Greg,

please note that I am not a member of the SCE development team, but I have looked at the L4S/SCE comparisons in detail and hence believe I can add (other people's) data to this discussion thread.

More below in-line.

> On Jan 7, 2020, at 01:09, Greg White <[email-masked]> wrote:
> 
> David,
> See [GW] below.
> Greg
> 

	[SM] I believe we could just look at the data collected for scenario 1 at https://github.com/heistp/sce-l4s-bakeoff/blob/master/README.md#scenario-1, especially https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/sce-s1-2/batch-sce-s1-2-cubic-vs-reno-sce-50Mbit-80ms_fixed.png, https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/sce-s1-2/batch-sce-s1-2-cubic-vs-reno-sce-50Mbit-10ms_fixed.png, and https://www.heistp.net/downloads/sce-l4s-bakeoff/bakeoff-2019-11-11T090559-r2/sce-s1-2/batch-sce-s1-2-cubic-vs-reno-sce-50Mbit-0ms_fixed.png, which show the "degenerate" single queue SCE-AQM case. 

	And your intuition seems to be right, the tested naive SCE AQM will, in single queue configuration, unduly slow down SCE-enabled flows (just like a single queue L4S AQM would unduly slow down flows with a 1/sqrt(p) response).
IMHO this seems to indicate that for single queue operation (or operation at similar computational cost to a single queue?) the SCE AQM needs refinements, and it is not clear yet how well this can be made to work but there is work on-going in that regard. It certainly seems safe behavior from the perspective of throughput of the standard non-SCE traffic though, and it also seems relatively easy to simply avoid instantiating single queue SCE AQMs until this issue is remedied.

	This issue is also explicitly acknowledged by SCE's developers at https://github.com/heistp/sce-l4s-bakeoff/blob/master/README.md#list-of-sce-issues:

"SCE Single Queue Non-SCE Unfairness

Because SCE is marked at lower queue depths than CE, non-SCE flows will outcompete SCE flows in a single queue without a change to the SCE marking ramp, as presented at IETF 105. This is effective, however it trades off some of the benefits of SCE, and requires tuning, which may be dependent on the link or traffic conditions.

Another solution is fair queueing. One of the arguments against this is the resource requirements for fair queueing, which led to the proposed Lightweight Fair Queueing and Cheap Nasty Queueing. Another argument is the perception that FQ can do harm to periodic bursty flows, however we have not yet shown this to be the case with hard evidence, and the SCE team is in general in favor of FQ.

We continue to explore alternative solutions to this challenge, and will seek to quantify any potential side effects of FQ, however this is a decades old debate that may not be resolved any time soon."

https://datatracker.ietf.org/meeting/105/materials/slides-105-tsvwg-sessa-61-some-congestion-experienced-00 (slide 13) shows that e.g. modification to the marking ramp-function can help ameliorate that issue.