Subject: Re: [tsvwg] New Version Notification for draft-morton-taht-tsvwg-sce-00.txt

From: Dave Taht <[email-masked]>
Date: Sun, 10 Mar 2019 18:16:09 -0700

I think this gets the procedure right for submittal here.

[email-masked] writes:

> A new version of I-D, draft-morton-taht-tsvwg-sce-00.txt
> has been successfully submitted by David M. Täht and posted to the
> IETF repository.
>
> Name:		draft-morton-taht-tsvwg-sce
> Revision:	00
> Title:		The Some Congestion Experienced ECN Codepoint
> Document date:	2019-03-10
> Group:		Individual Submission
> Pages:		7
> URL:            https://www.ietf.org/internet-drafts/draft-morton-taht-tsvwg-sce-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-morton-taht-tsvwg-sce/
> Htmlized:       https://tools.ietf.org/html/draft-morton-taht-tsvwg-sce-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-morton-taht-tsvwg-sce
>
>
> Abstract:
>    This memo reclassifies ECT(1) to be an early notification of
>    congestion on ECT(0) marked packets, which can be used by AQM
>    algorithms and transports as an earlier signal of congestion than CE.
>    It is a simple, transparent, and backward compatible upgrade to
>    existing IETF-approved AQMs, RFC3168, and nearly all congestion
>    control algorithms.
>
>                                                                                   
>
>
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat

---

From: Michael Welzl <[email-masked]>
Date: Mon, 11 Mar 2019 10:39:40 +0100

Hi,

Regarding this paragraph:

***
   The underlying cause is the very sharp "multiplicative decrease"
   reaction required of transport protocols to congestion signalling
   (whether that be packet loss or CE marks), which tends to leave the
   congestion window significantly smaller than the ideal BDP when
   triggered at only slightly above the ideal value.  The availability
   of this sharp response is required to assure network stability (AIMD
   principle), but there is presently no standardised and backwards-
   compatible means of providing a less drastic signal.
***

FWIW, TCP Alternative Backoff with ECN (ABE), RFC 8511, tries to address exactly that problem without requiring an extra code point.

The underlying logic is: when ECN is signalled, chances are that there's an AQM algorithm in place, and the queue is probably short - hence the very sharp response is probably not appropriate.

Cheers,
Michael

> On 11 Mar 2019, at 02:16, Dave Taht <[email-masked]> wrote:
> 
>

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 11 Mar 2019 16:29:06 +0000

Dave,

I'm pasting my comments on this from the bufferbloat list here. I guess 
discussion on assignment of an IETF codepoint ought to continue here.
=======================================================================================

L4S is far from dead. It's merely been working differently from how 
you're used to. Those working on an L4S AQM (at least those in the cable 
industry) had to have a private WG for the last ~18 months, but now 
we're allowed to publish and talk openly again. Similarly, there's work 
under the covers on an L4S AQM in switch hardware. And I see external 
signs of work under covers on DSL access equipment (covers that I am not 
under any longer).

Nonetheless, I think you will see updated Linux code for an L4S DualQ 
Coupled AQM built against the mainline tree appear on netdev list today.

==In summary==

The problem that the SCE draft identifies with TCP's sharp 
multiplicative decrease is also the primary problem that L4S identified.

Like L4S, SCE requires changes to network, sender and receiver (see 
comment later about the rcv-window approach hinted at in the SCE draft). 
But SCE is just starting on its journey. Having to change end systems 
and network together is really tough and takes many years.

It seems you're trying to do the same thing as L4S, but by slightly 
different means. Before splitting the people involved in this into two 
factions, can you say what you didn't like about the L4S approach in the 
first place? We've been very careful to specify L4S broadly enough so 
that it can encompass many different approaches within it.

The only thing stated against L4S I can find is that it's taking a long 
time. Starting an identically difficult approach now is going to restart 
the clock, and take a lot lot longer.

==2 output values vs. 2 input values.==

We considered schemes where the AQM can use a second marking as a lower 
strength /output/ (like VCP, my own QV and now SCE). But we worked out 
that there were a wider range of advantages and much more significant 
performance improvements from the sender using a second marking to 
distinguish how it will behave (i.e. a second /input/ to the classifier 
in front of the AQM).

Don't get me wrong. It's useful that you guys are putting the work in on 
SCE. Then everyone can compare the two approaches (again), as a check on 
whether that decision was correct. That's important, cos ECT(1) is the 
last useful codepoint in the IP header. See: "Notification of Less 
Severe Congestion than CE" at 
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-C.2 
where we've written:

    Before assigning ECT(1) as an identifer for L4S, we must carefully
    consider whether it might be better to hold ECT(1) in reserve for
    future standardisation of rapid flow acceleration, which is an
    important and enduring problem [RFC6077  <https://tools.ietf.org/html/rfc6077>].

==FQ-only vs. FQ or DualQ==

One of the problems with the 2 outputs approach (SCE etc.) is that it is 
only possible with per-flow queuing. I doubt you'll get the last useful 
codepoint in the IP header for just that. It's sort-of obvious that, if 
you try to implement SCE in a FIFO, you can only have one queue length 
for all the flows. Then legacy TCP flows that don't understand SCE would 
push the queue deeper to the CE threshold, ruining it for the flows that 
support SCE.

We worked out that the 2 inputs approach (L4S) is more generic - ie. it 
can be used with FQ or DualQ (multiple or just 2 queues).

For instance, you can modify fq_CoDel for senders that uses ECT(1) to 
indicate that they support a small multiplicative decrease (L4S 
senders). You only need the following: Include the last bit of the ECN 
field with the flow ID when you do the classification for sfq. Then in 
the queues with ECN==X1, you instantiate a shallow threshold ECN AQM. 
This could be CoDel with a shallow 'target', but you also want it to 
respond immediately (zero 'interval'), so even a simple step at about 
1ms will work, but a random RED-like ramp on the /instantaneous/ queue 
is much better.

==Re-purposing the Receive Window?===

Receiver congestion control using the receive window may seem like a 
useful stop-gap, but it runs counter to how nearly all today's transport 
protocols are intended to work (except, I know of a LEDBAT-like example 
from Microsoft Research). So you will have your work cut out proving 
that it is stable and that the two ends don't fight, etc. if you think 
L4S is taking years, you will find that takes longer. There is current 
research on this that I can point you to, if you want.

That's why we chose an approach that had a pre-existing widely deployed 
existence proof (DCTCP) to start from.

IETF groups like rmcat explicitly decided early on to require the 
approach where the receiver is a dumb reflector, then new sender 
congestion control algorithms can be deployed unilaterally. The argument 
was that the feedback function can be thought of as a sub-layer below 
the congestion control function. The ongoing addition of accurate ECN 
feedback to TCP and to QUIC also take the dumb reflector approach. And 
RTCP already does it that way.

==ECN feedback problems===

Over the last decades, we've made sure that the ECN feedback schemes for 
TCP, QUIC, RTP (but not SCTP yet) can all feed back ECT(1) as well as 
CE, in case a scheme like SCE came along.

However, the solution in the TCP case [draft-ietf-tcpm-accurate-ecn] is 
still problematic for SCE if you're impatient. The base scheme overloads 
3 bits in the TCP header, which it uses to feed back CE only. To feed 
back ECT(1) we had to add a TCP option. That's not going to get through 
middleboxes for many years. The TCP option is also optional to 
implement. Two of the main TCP developers are currently saying they will 
probably not implement it, at least not initially.

==Tunnels and lower layers==

Over the years I've maintained a fairly lonely activity to make sure 
that the ECN propagation behaviour of tunnels and layer 2 protocols will 
treat ECT(1) as either a stronger output signal (as in SCE) or as an 
alternative input signal to an AQM (as in L4S). Theoretically, this 
allows either the SCE or the L4S approach.

HOWEVER, you would probably not be surprised at how many people read the 
spec [RFC6040], and say "Ah, no router alters ECT(0) to ECT(1) today, so 
I'm not going to implement that unnecessary extra line of code in my 
tunnel decap."

==Wider benefit: Relaxing link ordering==

By overloading the ECT(1) marking to mean "the sender uses time for loss 
detection" a link can relax the reordering requirement on ECT(1) packets 
today. You can do that with L4S, cos the sender is selecting the 
marking. You can't do that when the AQM is selecting the marking (as 
with SCE).

If transport protocols detect loss in time units without tying it to any 
marking (as in RACK on its own), a link cannot use this to relax the 
ordering requirement until it is sure that all the legacy non-RACK 
transports have decayed out of the network. That would be measured in 
decades.

HTH

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Greg White <[email-masked]>
Date: Mon, 11 Mar 2019 17:15:54 +0000

On the item below, I am intending to finish up and submit a TSVWG informational draft today that describes the (new) mandatory support for L4S AQM and dual-queue requirement in DOCSIS 3.1 (CM & CMTS) equipment.  The draft will essentially be a reformat of this document:
https://cablela.bs/low-latency-docsis-technology-overview-february-2019

These requirements were added in December, and will roll out as firmware updates to existing DOCSIS 3.1 equipment (nearly 100% of CMTSs, and a growing fraction of CMs) over time.

-Greg

From: tsvwg <[email-masked]> on behalf of Bob Briscoe <[email-masked]>
Date: Monday, March 11, 2019 at 10:29 AM
To: Dave Taht <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: Jonathan Morton <[email-masked]>
Subject: Re: [tsvwg] New Version Notification for draft-morton-taht-tsvwg-sce-00.txt

L4S is far from dead. It's merely been working differently from how you're used to. Those working on an L4S AQM (at least those in the cable industry) had to have a private WG for the last ~18 months, but now we're allowed to publish and talk openly again.

---

From: Loganaden Velvindron <[email-masked]>
Date: Mon, 11 Mar 2019 23:31:42 +0400

On Mon, Mar 11, 2019 at 8:29 PM Bob Briscoe <[email-masked]> wrote:
>
> Dave,
>
> I'm pasting my comments on this from the bufferbloat list here. I guess discussion on assignment of an IETF codepoint ought to continue here.
> L4S is far from dead. It's merely been working differently from how you're used to. Those working on an L4S AQM (at least those in the cable industry) had to have a private WG for the last ~18 months, but now we're allowed to publish and talk openly again. Similarly, there's work under the covers on an L4S AQM in switch hardware. And I see external signs of work under covers on DSL access equipment (covers that I am not under any longer).
>
> Nonetheless, I think you will see updated Linux code for an L4S DualQ Coupled AQM built against the mainline tree appear on netdev list today.
>
> ==In summary==
>
> The problem that the SCE draft identifies with TCP's sharp multiplicative decrease is also the primary problem that L4S identified.
>

Hence why I think it's important to do research on both and see how
they fare when used by early adopters on routers at home :-)

---

From: Dave Taht <[email-masked]>
Date: Mon, 11 Mar 2019 16:33:19 -0700

Greg White <[email-masked]> writes:

> On the item below, I am intending to finish up and submit a TSVWG
> informational draft today that describes the (new) mandatory support
> for L4S AQM and dual-queue requirement in DOCSIS 3.1 (CM & CMTS)
> equipment. The draft will essentially be a reformat of this document:
>
> These requirements were added in December, and will roll out as
> firmware updates to existing DOCSIS 3.1 equipment (nearly 100% of
> CMTSs, and a growing fraction of CMs) over time.

I was unaware of this notice, and as I note, it's only march...

Using up the last ECN bit in this way, for the sole benefit of the cable
industry, strikes me as a poor standard for the internet.

> you're used to. Those working on an L4S AQM (at least those in the
> cable industry) had to have a private WG for the last ~18 months, but
> now we're allowed to publish and talk openly again.

---

From: Greg White <[email-masked]>
Date: Mon, 11 Mar 2019 23:49:59 +0000

I'm not sure how you misinterpreted L4S as being "for the sole benefit of the cable industry".  It is a standards-track draft here at IETF.  From what I understand, multiple network technologies are in the process of implementing it.  None have been public about it until now.

-Greg

On 3/11/19, 5:33 PM, "Dave Taht" <[email-masked]> wrote:

---

From: Dave Taht <[email-masked]>
Date: Mon, 11 Mar 2019 16:55:33 -0700

Greg White <[email-masked]> writes:

> I'm not sure how you misinterpreted L4S as being "for the sole benefit
> of the cable industry".  It is a standards-track draft here at IETF.
> From what I understand, multiple network technologies are in the
> process of implementing it.  None have been public about it until now.

Publically, judging from the traffic on the ietf mailing lists, it
appeared to have died. It wasn't until January or so that activity
picked up. 

we were public about our separate wg when we formed it back in
august. The first output of that we just submitted as this draft,
and there is code and data landing for public review now.

Our charter is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/

We looked at a lot more things than just tcp along the way.

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 12 Mar 2019 00:08:12 +0000

On 2019-03-11, 16:50, "Greg White" <[email-masked]> wrote:
> It is a standards-track draft here at IETF

I thought the L4S drafts were experimental, not standards-track:

https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-08
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-04

Was there one that's standards track?

Thanks and regards,
Jake

    -Greg

---

From: Luca Muscariello <[email-masked]>
Date: Mon, 11 Mar 2019 17:25:56 -0700

Hi Greg,

I'm curious about the queue protection function in the LLD document.
It seems to assume that a flow table is maintained to determine if a flow
has the right to enter the low latency queue.

Can you give more details about that component? Or point me to a reference?

Thanks
Luca

On Mon, Mar 11, 2019 at 4:50 PM Greg White <[email-masked]> wrote:

> I'm not sure how you misinterpreted L4S as being "for the sole benefit of
> the cable industry".  It is a standards-track draft here at IETF.  From
> what I understand, multiple network technologies are in the process of
> implementing it.  None have been public about it until now.
>
> -Greg
>
>

---

From: Greg White <[email-masked]>
Date: Tue, 12 Mar 2019 02:38:27 +0000

I stand corrected.  They are experimental. Nonetheless, they are not proprietary specs for the cable industry. 

The DOCSIS implementations of L4S are entirely in firmware and can be enabled/disabled by the operator via configuration settings.

-Greg

On 3/11/19, 6:08 PM, "Holland, Jake" <[email-masked]> wrote:

    

    Thanks and regards,
    Jake

        -Greg

---

From: "Holland, Jake" <[email-masked]>
Date: Tue, 12 Mar 2019 17:24:28 +0000

Hi Greg,

Thanks for clarifying, that's very interesting.

I applaud the release of a reference implementation of L4S suitable
for independent evaluation and real-world testing, so thanks and
nice work!

However, it's surprising to me to hear that support for a system
that was presented to the IETF as a proposed experiment is
mandatory--I would have thought support would only be applicable
to nodes explicitly participating in experiments?

If support for this is going to be very widely deployed, is the
scope of the experiment documented anywhere?  (Or has any thought
been given to anything like a pre-emptive sunset date, after which
the use of L4S and the L4S semantics for ECT(1) should be stopped
unless a later document extends the experiment or standardizes it?)

I couldn't find anything in those docs, so I thought I'd ask for
a pointer.

If there's no such thing, can I ask for more information about
other steps you've taken to address the responsibilities outlined
in Section 8 of RFC 8311 with regard to this experiment?
https://tools.ietf.org/html/rfc8311#section-8

(I've seen a lot of the lab work presented, with many thanks to
Bob, Koen, and others, and it looks promising. But I mean to ask
about mitigations that address potential issues that are so far
undiscovered that may crop up in deployment or in externally run
experiments?)

Thanks and regards,
Jake

On 2019-03-11, 19:38, "Greg White" <[email-masked]> wrote:

    
    -Greg

        

        
        Thanks and regards,
        Jake

            -Greg