Subject: [tsvwg] I-D Action: draft-ietf-tsvwg-ecn-l4s-id-15.txt

From: [email-masked]
Date: Fri, 07 May 2021 05:03:31 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Transport Area Working Group WG of the IETF.

        Title           : Explicit Congestion Notification (ECN) Protocol for Ultra-Low Queuing Delay (L4S)
        Authors         : Koen De Schepper
                          Bob Briscoe
	Filename        : draft-ietf-tsvwg-ecn-l4s-id-15.txt
	Pages           : 61
	Date            : 2021-05-07

Abstract:
   This specification defines the protocol to be used for a new network
   service called low latency, low loss and scalable throughput (L4S).
   L4S uses an Explicit Congestion Notification (ECN) scheme at the IP
   layer that is similar to the original (or 'Classic') ECN approach,
   except as specified within.  'Classic' ECN marking is required to be
   equivalent to a drop, both when applied in the network and when
   responded to by a transport.  Unlike 'Classic' ECN marking, for
   packets carrying the L4S identifier, the network applies marking more
   immediately and more aggressively than drop, and the transport
   response to each mark is reduced and smoothed relative to that for
   drop.  The two changes counterbalance each other so that the
   throughput of an L4S flow will be roughly the same as a non-L4S flow
   under the same conditions.  Nonetheless, the much more frequent
   control signals and the finer responses to them result in much more
   fine-grained adjustments, so that ultra-low and consistently low
   queuing delay (typically sub-millisecond on average) becomes possible
   for L4S traffic without compromising link utilization.  Thus even
   capacity-seeking (TCP-like) traffic can have high bandwidth and very
   low delay at the same time, even during periods of high traffic load.

   The L4S identifier defined in this document distinguishes L4S from
   'Classic' (e.g. TCP-Reno-friendly) traffic.  It gives an incremental
   migration path so that suitably modified network bottlenecks can
   distinguish and isolate existing traffic that still follows the
   Classic behaviour, to prevent it degrading the low queuing delay and
   low loss of L4S traffic.  This specification defines the rules that
   L4S transports and network elements need to follow to ensure they
   neither harm each other's performance nor that of Classic traffic.
   Examples of new active queue management (AQM) marking algorithms and
   examples of new transports (whether TCP-like or real-time) are
   specified separately.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-15
https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-15

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-15

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 May 2021 14:21:17 +0100

tsvwg folks,

This -15 revision addresses the discussions over the past couple of 
months since the last IETF meeting about the normative 'Prague' 
requirements text, in particular addressing points made by developers in 
the survey of how practical the requirements were.

We were about to post -15 for the tsvwg L4S interim on Monday, when 
Gorry sent his review yesterday. So we agreed that we would continue 
with this posting, and attempt to post a further revision before Monday, 
to try to address all Gorry's points where necessary. Hopefully we've 
captured most of the normative requirements in -15, and -16 will be more 
focussed on editorial.

I would particularly draw attention to the completely new requirement 
text for coexistence with Classic Congestion Control in a Classic ECN 
AQM. It's quoted below. The intent is to be consistent with the approach 
in l4sops. Greg posted the first WG draft of that about 12hrs ago: 
draft-ietf-tsvwg-l4sops-00

This requirement considerably longer than the other requirements, and 
I'll be surprised if we've got it completely right, 'cos it tries to 
constrain a multitude of possible approaches, allowing flexibility but 
also keeping the important constraints. So pls review closely.

    o  In uncontrolled environments, monitoring MUST be implemented to
       support detection of problems with an ECN-capable AQM at the path
       bottleneck that appears not to support L4S and might be in a
       shared queue.  Such monitoring SHOULD be applied to live traffic
       that is using Scalable congestion control.  Alternatively,
       monitoring need not be applied to live traffic, if monitoring has
       been arranged to cover the paths that live traffic takes through
       uncontrolled environments.

       The detection function SHOULD be capable of making the congestion
       control adapt its ECN-marking response to coexist safely with
       Classic congestion controls such as standard Reno [RFC5681], as
       required by [RFC5033].  Alternatively, if adaptation is not
       implemented and problems with such an AQM are detected, the
       scalable congestion control MUST be replaced by a Classic
       congestion control.

       Note that a scalable congestion control is not expected to change
       to setting ECT(0) while it transiently adapts to coexist with
       Reno.

       See Appendix A.1.5 and [I-D.ietf-tsvwg-l4sops] for rationale.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 7 May 2021 09:45:35 -0400

On Fri, May 7, 2021 at 9:22 AM Bob Briscoe <[email-masked]> wrote:

> tsvwg folks,
>
> This -15 revision addresses the discussions over the past couple of months
> since the last IETF meeting about the normative 'Prague' requirements text,
> in particular addressing points made by developers in the survey of how
> practical the requirements were.
>
> We were about to post -15 for the tsvwg L4S interim on Monday, when Gorry
> sent his review yesterday. So we agreed that we would continue with this
> posting, and attempt to post a further revision before Monday, to try to
> address all Gorry's points where necessary. Hopefully we've captured most
> of the normative requirements in -15, and -16 will be more focussed on
> editorial.
>
> I would particularly draw attention to the completely new requirement text
> for coexistence with Classic Congestion Control in a Classic ECN AQM. It's
> quoted below. The intent is to be consistent with the approach in l4sops.
> Greg posted the first WG draft of that about 12hrs ago:
> This requirement considerably longer than the other requirements, and I'll
> be surprised if we've got it completely right, 'cos it tries to constrain a
> multitude of possible approaches, allowing flexibility but also keeping the
> important constraints. So pls review closely.
>
> If a scalable congestion control transiently adapts to coexist with Reno,
and keeps marking its data segments with ECT(1), then if its estimation
that it is coexisting with Reno was incorrect, and instead it is actually
traveling through an L4S bottleneck, won't its Reno-style reactions to L4S
CE marks cause it to suffer very low throughput? To avoid this problem,
would it perhaps be advantageous to have a scalable congestion control
sender that is emulating Reno instead switch to using ECT(0), so that the
L4S bottleneck runs it through the classic queue?

best,
neal

> The IETF datatracker status page for this draft is:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>
> There are also htmlized versions available at:https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-15https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-15
>
> A diff from the previous version is available at:https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-15
>
>
> Internet-Drafts are also available by anonymous FTP at:ftp://ftp.ietf.org/internet-drafts/
>
>
> --

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 May 2021 16:56:21 +0100

Neal,

On 07/05/2021 14:45, Neal Cardwell wrote:
>
>
> On Fri, May 7, 2021 at 9:22 AM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     tsvwg folks,
>
>     This -15 revision addresses the discussions over the past couple
>     of months since the last IETF meeting about the normative 'Prague'
>     requirements text, in particular addressing points made by
>     developers in the survey of how practical the requirements were.
>
>     We were about to post -15 for the tsvwg L4S interim on Monday,
>     when Gorry sent his review yesterday. So we agreed that we would
>     continue with this posting, and attempt to post a further revision
>     before Monday, to try to address all Gorry's points where
>     necessary. Hopefully we've captured most of the normative
>     requirements in -15, and -16 will be more focussed on editorial.
>
>     I would particularly draw attention to the completely new
>     requirement text for coexistence with Classic Congestion Control
>     in a Classic ECN AQM. It's quoted below. The intent is to be
>     consistent with the approach in l4sops. Greg posted the first WG
>     draft of that about 12hrs ago: draft-ietf-tsvwg-l4sops-00
>
>     This requirement considerably longer than the other requirements,
>     and I'll be surprised if we've got it completely right, 'cos it
>     tries to constrain a multitude of possible approaches, allowing
>     flexibility but also keeping the important constraints. So pls
>     review closely.
>
> If a scalable congestion control transiently adapts to coexist with 
> Reno, and keeps marking its data segments with ECT(1), then if its 
> estimation that it is coexisting with Reno was incorrect, and instead 
> it is actually traveling through an L4S bottleneck, won't its 
> Reno-style reactions to L4S CE marks cause it to suffer very low 
> throughput? To avoid this problem, would it perhaps be advantageous to 
> have a scalable congestion control sender that is emulating Reno 
> instead switch to using ECT(0), so that the L4S bottleneck runs it 
> through the classic queue?

[BB] There are arguments both ways, so we didn't state this as a 
normative requirement, we just said it's 'not expected to'. But on 
balance we decided there were more points in favour of sticking with 
ECT(1) than switching:
* Avoids flipping to another path if there's load balancing or ECMP 
hashed on the ex-ToS byte, which is still unfortunately common.
* If it turns out to have been wrong to switch to Classic /behaviour/ 
because there is an L4S AQM at the bottleneck, if it switches to the 
ECT(0) /codepoint/ as well, it will definitely get classified into the 
Classic queue and never get the chance to measure that it should switch 
back again.
* But if it stays as ECT(1) but behaves as Classic:
     - a classic AQM will treat it as ECT(0), which is fine.
     - but, if it's actually still in an L4S queue, yes, it could suffer 
as you say, but it will also be able to continue measuring, including 
measuring that it's suffering, which could help it switch back to the 
correct (L4S) behaviour.

Let me know if you still think the text ought to say the opposite.

==Example plot==

For instance, in the Classic AQM detection algorithm that we implemented 
in TCP Prague, there were some cases at large base RTT where the algo 
switched to Classic behaviour then realized it's true identity and 
switched back. Here's an example:
https://l4s.net/ecn-fbk/results_v2.2/one_aqm/dualpi2_release/40mb_100ms/classic_ecn.html
it's over a DualQ bottleneck with Prague flows (rows) competing with 
Cubic (columns). The plots show the 'Classic ECN variable' of the 
detection algo in the Prague flows. And the green background shows where 
the variable should be if detection is correct (orange is when it 
incorrectly thinks it's in a Classic AQM):

* At (1 Prague : 9 Cubic) you can see the Prague flow starts to think 
it's in a Classic AQM, then when it gets there it finds it's probably 
not, so it retreats back and stays on the correct side.
* At (9:9) the red plot is a counter example that switches to Classic, 
starts to retreat back to L4S again, then bounces back into Classic as 
soon as it starts to behave as L4S. And the dark olive one is a 
counter-counter example; it does the two bounces like the red one, but 
then later, after a period of prevarication, it decides it really is 
L4S. Then the run ended, and we'll never know what it would have done 
next... ;)
* Note that it's hard to see the blue line on some of the plots that 
stay firmly at the bottom, where they should be.

The full explanation of the visualization is here: 
https://l4s.net/ecn-fbk/results_v2.2/full_heatmap/ and you can find the 
experiment set up off the 'Internals' menu.

Bob

>     On 07/05/2021 13:03, [email-masked]
>     <mailto:[email-masked]> wrote:
>>     https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/  <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/>
>>
>>     https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-15  <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-15>
>>     https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-15  <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-15>
>>
>>     https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-15  <https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-15>
>>
>>
>>     until the htmlized version and diff are available attools.ietf.org  <http://tools.ietf.org>.
>>
>>     ftp://ftp.ietf.org/internet-drafts/  <ftp://ftp.ietf.org/internet-drafts/>
>>
>>
>
>     -- 
>     Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>
>

---

From: Neal Cardwell <[email-masked]>
Date: Sat, 8 May 2021 10:44:07 -0400

On Fri, May 7, 2021 at 11:56 AM Bob Briscoe <[email-masked]> wrote:

>> text for coexistence with Classic Congestion Control in a Classic ECN AQM.
>> It's quoted below. The intent is to be consistent with the approach in
>> l4sops. Greg posted the first WG draft of that about 12hrs ago:
>> constrain a multitude of possible approaches, allowing flexibility but also
>> keeping the important constraints. So pls review closely.
>>
> [BB] There are arguments both ways, so we didn't state this as a normative
> requirement, we just said it's 'not expected to'. But on balance we decided
> there were more points in favour of sticking with ECT(1) than switching:
> * Avoids flipping to another path if there's load balancing or ECMP hashed
> on the ex-ToS byte, which is still unfortunately common.
> because there is an L4S AQM at the bottleneck, if it switches to the ECT(0)
> /codepoint/ as well, it will definitely get classified into the Classic
> queue and never get the chance to measure that it should switch back again.
>     - but, if it's actually still in an L4S queue, yes, it could suffer as
> you say, but it will also be able to continue measuring, including

Thanks, Bob. That's a very convincing argument for sticking with ECT(1).
IMHO it would be nice to lay out those considerations in much the way you
did here, in a brief paragraph in the draft (perhaps in an appendix, if it
interrupts the flow too much?). Of course if you think that is too much
detail, then I will defer to your judgement.

thanks,
neal

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 9 May 2021 21:36:14 +0100

Neal,
Done.

Here's the new text added at end of Appx A.1.5:

    Finally, we motivate the recommendation in Section 4.3 that a
    scalable congestion control is not expected to change to setting
    ECT(0) while it adapts its behaviour to coexist with Classic flows.
    This is because the sender needs to continue to check whether it made
    the right decision - and switch back if it was wrong, or if a
    different link becomes the bottleneck:

    o  If, as recommended, the sender changes only its behaviour but not
       its codepoint to Classic, its codepoint will still be compatible
       with either an L4S or a Classic AQM.  If the bottleneck does
       actually support both, it will still classify ECT(1) into the same
       L4S queue, where the sender can measure that switching to Classic
       behaviour was wrong, so that it can switch back.

    o  In contrast, if the sender changes both its behaviour and its
       codepoint to Classic, even if the bottleneck supports both, it
       will classify ECT(0) into the Classic queue, reinforcing the
       sender's incorrect decision so that it never switches back.

    o  Also, not changing codepoint avoids the risk of being flipped to a
       different path by a load balancer or multipath routing that hashes
       on the whole of the ex-ToS byte (unfortunately still a common
       pathology).

    Note that if a flow is configured to _only_ use a Classic congestion
    control, it is then entirely appropriate not to use ECT(1).

Bob

---

From: Neal Cardwell <[email-masked]>
Date: Mon, 10 May 2021 16:11:11 -0400

>    control, it is then entirely appropriate not to use ECT(1).
Thanks, Bob. This looks great to me.

best,
neal