Subject: Re: [tsvwg] [iccrg] IETF104 ICCRG TCP Prague presentation

From: Bob Briscoe <[email-masked]>
Date: Wed, 3 Apr 2019 18:40:34 +0100

Sebastien, (adding tsvwg, given this is all part of the decision over 
L4S/SCE),

On 02/04/2019 12:38, Sebastian Moeller wrote:
> I finally got around to see the TCP Prague presentation, nice, especially the part around slide 11 "Fallback to Reno-friendly on Classic ECN" caught my interest.
>
> The claim made there is that L4S's inability to properly deduce the type of CE signal (normal ECN response expected versus DCTCP-like ECN response expected) would only matter in the case of ECN-marking single queue AQMs and not for flow-queueung AQMs, and I would like to argue that that should be measured rather than solved by declaration.
[BB] I don't know whether you just looked at my slides without the voice 
track. But the voice track makes it clear that is exactly what I've been 
setting up.

To save those who weren't in the room from having to sit through the 
audio, I said that there are two sets of contradictory measurements:

  * those from 'academic' active measurement studies that have found
    hardly any CE
  * those from Apple's passive measurements on Apple devices that have
    found significant localized amounts of CE.

My hypothesis is that the active measurements aren't seeing much CE, cos 
they don't generate enough traffic themselves to congestion the link and 
they would have to coincide with other traffic to see it passively.

I said, in short, I believe the active measurements are more likely 
wrong, and passive more likely right.

Nonetheless the CE that Apple measured could be consistent with classic 
ECN marking from FQ-CoDel, e.g. in OpenWRT devices, which would protect 
competing flows from L4S flows. Apple saw CE "mainly in the uplink", 
which is likely to be FQ-CoDel. And the smaller amount seen in the 
downlink could have been peer-to-peer flows that had been marked on 
entry to the uplink by an FQ-CoDel home router.

With the help of others, I've devised a test for FQ vs FIFO, but we need 
to find where the CE-marking routers are before we apply the test (a 
scatter-gun approach would be infeasible given the need to congest the 
link). In discussions with Apple folks, we think it's best to run tests 
at the CDN end, to identify the network prefixes where CE is appearing 
(the client end used in the original tests usually doesn't know its 
public IP address).

If we find any CE in the downlink from the CDN, that will disprove my 
peer-peer FQ hypothesis. But it won't prove that it's from a FIFO. The 
test for FQ vs FIFO is not straightforward, and I haven't yet worked out 
how we run it in the relevant direction over the paths where we find CE 
(not least 'cos that would involve collecting and identifying client IP 
addresses, which raises privacy concerns). I suspect we'll need to 
co-opt some volunteers in the affected localities, somehow.

An alternative to inline testing will be to find out whether the 
networks where CE is being seen have supplied their customers with 
routers with FQ-CoDel enabled. That won't prove conclusively that all 
the CE is coming from FQ-CoDel, but we can only go so far in trying to 
prove a negative.

>
>
> Consider the following case, that get's more and more common: traffic shaping for both ingress and egress somewhere on the end-customer side of an internet access link. For egress I agree with the rationale, that flow-queueing/fair-queueung should deal with flows that appear unresponsive sooner or later (but I note that this still will increase the memory required for the queueing and should not be considered ideal behavior).
> But for ingress shaping downstream of the "true" bottleneck (I am simplifying over some details here) this rationale falls apart, the unexpected packets the L4S sender kept sending since it misinterpreted the CE mark, will still hog valuable bandwidth of the internet access link, and hence fair queuing on the home end of the true bottleneck will not isolate that homenet from the L4S misunderstanding.
[BB] For the avoidance of doubt, I think you're using 'egress' to mean 
[home -> Internet] (I use it for the other direction, but I can work 
with your terminology).

In the [Internet -> home] direction, you are right that FQ on the home 
end would not isolate the access link from a misunderstanding of any CE 
marking. But I'm not suggesting it could or would. The question is about 
the machine at the Internet side of the access link (BRAS, CMTS, etc) 
that is queuing traffic into the access link towards the home. We can be 
nearly certain it is not doing FQ (these machines serve thousands of 
customers, so I don't believe any use FQ). The question is whether we 
can find one of these non-FQ machines that has an AQM enabled that does 
CE marking.

That's what we're trying to find. It's of course hard to prove a 
negative - that none exist. All we can hope to do is measure a sample of 
the paths we find.

A nit: DCTCP is not /unresponsive/. It responds to congestion, but as if 
it is equivalent to a number of 'classic' (Reno-friendly) TCP flows.

>
> I would be delighted to learn where my assumptions are wrong...
Regards

Bob

>
> Regards
> 	Sebastian Moeller
>
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 3 Apr 2019 21:28:08 +0300

> On 3 Apr, 2019, at 8:40 pm, Bob Briscoe <[email-masked]> wrote:
> 
> To save those who weren't in the room from having to sit through the audio, I said that there are two sets of contradictory measurements: 
> 	• those from 'academic' active measurement studies that have found hardly any CE
> 	• those from Apple's passive measurements on Apple devices that have found significant localized amounts of CE.
> My hypothesis is that the active measurements aren't seeing much CE, cos they don't generate enough traffic themselves to congestion the link and they would have to coincide with other traffic to see it passively.
> 
> I said, in short, I believe the active measurements are more likely wrong, and passive more likely right.

This is actually reasonable.  However your slides were very unclear about this, which I think led to unnecessary confusion.

> Nonetheless the CE that Apple measured could be consistent with classic ECN marking from FQ-CoDel, e.g. in OpenWRT devices, which would protect competing flows from L4S flows. Apple saw CE "mainly in the uplink", which is likely to be FQ-CoDel. And the smaller amount seen in the downlink could have been peer-to-peer flows that had been marked on entry to the uplink by an FQ-CoDel home router.

It is actually common for OpenWRT devices to include "ingress shaping" to exert some control over bufferbloat on downloads.  The terminology, in fact, comes from this practice, as the port in question is facing the WAN link and has two qdiscs attached, one for egress traffic towards the Internet, and one for ingress traffic received from the Internet.

A lower rate of observed marking on downloads could be due to the greater difficulty of driving the download direction into saturation, especially given a backhaul network that may be oversubscribed in many ISPs during peak hours, and thus the greater difficulty of properly configuring an ingress shaper.  It is of course better for AQM to be installed on the upstream side of where the congestion actually occurs.

> In the [Internet -> home] direction, you are right that FQ on the home end would not isolate the access link from a misunderstanding of any CE marking. But I'm not suggesting it could or would. The question is about the machine at the Internet side of the access link (BRAS, CMTS, etc) that is queuing traffic into the access link towards the home. We can be nearly certain it is not doing FQ (these machines serve thousands of customers, so I don't believe any use FQ). The question is whether we can find one of these non-FQ machines that has an AQM enabled that does CE marking.

The motivating reason for ingress shaping is that, generally, the BRAS or equivalent device in the ISP does not perform AQM at all (though some do policing, which doesn't involve ECN).

In my experience there is often a "provisioning shaper" that is essentially a TBF with a dumb tail-dropping FIFO attached; I once suffered one on a rate-limited 3G service which had about 40 seconds of effective bloat.  If that had any sort of AQM installed, ECN or otherwise, flow-isolating or not, it would have resulted in vastly better service at essentially no cost to the ISP.  Indeed it may have saved them some money in reduced memory requirements on their provisioning shaper box.

On my present LTE connection, which varies wildly in available capacity on a minute-to-minute basis, I've set my ingress shaper to a flat 10Mbps rate.  It makes an enormous improvement to the quality of service I receive, even though it reduces my throughput to a fraction of what's available during off-hours.  Yes, it is a flow-isolating AQM shaper (Cake) which is broadly similar to fq_codel.

I did spend a few minutes today sketching out what happens if an L4S flow hits fq_codel on an ingress shaper.  The theoretical results are that, in congestion-avoidance mode, the L4S flow takes about twice as long to drain the dumb bottleneck queue as a Classic ECN flow after the onset of saturation.  But the real shocker is what happens in the terminal phase of slow-start, where a rapid correction is required after the minimum 2x overshoot in exponential growth (because it takes an RTT for the CE/ECE/CWR signal to circulate and take effect).

How many RTTs does it take for the cwnd to be halved in L4S, when the only CE signal is ramping up linearly at one-per-RTT-per-RTT?  Hopefully enough for L4S to detect the delay induced by the dumb FIFO and fall back to RFC-compliant behaviour.

(Yes, we have a real solution sketched out for SCE.  It will be tested soon.  We already have running code.)

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Wed, 3 Apr 2019 23:28:37 +0000

On 4/3/19, 7:28 PM, "iccrg on behalf of Jonathan Morton" <[email-masked] on behalf of [email-masked]> wrote:
[...]
    It is actually common for OpenWRT devices to include "ingress shaping" to exert some control over bufferbloat on downloads.  The terminology, in fact, comes from this practice, as the port in question is facing the WAN link and has two qdiscs attached, one for egress traffic towards the Internet, and one for ingress traffic received from the Internet.
[...]
  It is of course better for AQM to be installed on the upstream side of where the congestion actually occurs.
[...]    
    The motivating reason for ingress shaping is that, generally, the BRAS or equivalent device in the ISP does not perform AQM at all (though some do policing, which doesn't involve ECN).

[GW] It will be the case that many such "equivalent devices" can (and will) support a dual-queue AQM (not fq), which (where available) will eliminate the need for ingress shaping, and provide a better result.    If fq_codel (or CAKE) were updated to utilize the L4S-style of ECN (or even just treat ECT(1) as non-ECT), this would be a non-issue.   How feasible would that be? 

[...]

[GW] One of the optimizations being investigated for L4S (both TCP/QUIC Prague and (I think) BBRv2) is a revised startup process that avoids the 2x overshoot from slow-start.  Nonetheless, you are right that an issue to be understood (and addressed?) with TCP Prague / fq_codel compatibility is how to prevent TCP Prague seeing worse latency than classic TCP at flow startup.  As you indicated, once the Prague sender falls back to Reno behavior, things are fine.  In fq_codel/CAKE currently, does the AQM switch from CE mark to packet drop at some higher threshold of sojourn time?  If so, would this provide for a faster fall-back to Reno than you considered in your analysis?  Alternatively, if fq_codel/CAKE were to treat ECT(1) as non-ECT (or implement the L4S style ECN) this would again not be an issue, I believe.  
    
[...]

     - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 4 Apr 2019 03:28:13 +0300

> On 4 Apr, 2019, at 2:28 am, Greg White <[email-masked]> wrote:
> 
> It will be the case that many such "equivalent devices" can (and will) support a dual-queue AQM (not fq), which (where available) will eliminate the need for ingress shaping, and provide a better result.    If fq_codel (or CAKE) were updated to utilize the L4S-style of ECN (or even just treat ECT(1) as non-ECT), this would be a non-issue.   How feasible would that be? 

The present out-of-tree version of Cake supports an L4S-style high-fidelity ECN marking method, simultaneously with conventional ECN marking.  It does so using the SCE signalling method, with ECT(1) being an output rather than an input.  This is, in fact, the running code we had at the time of our TSVWG presentation.

Since then we have adapted a pair of FreeBSD endpoints to understand this new signal and respond appropriately to it (using a response function very similar to DCTCP).  Crucially, they continue to respond normally, with TCP-friendly semantics, to non-SCE middleboxes and/or when conversing with non-SCE endpoints.  These are early results from early code, but they prove the concept.  

This bears repeating: we have effectively *rehabilitated* DCTCP.

It took three developers one week to do this, amidst many distractions.

> In fq_codel/CAKE currently, does the AQM switch from CE mark to packet drop at some higher threshold of sojourn time?

Technically yes, there is a failsafe intended for handling flows unresponsive to ECN (eg. due to blackholes) or insufficiently responsive to Codel-induced drops on Non-ECT flows.  However, this only kicks in when the queue backlog limit is reached.  It is a simple implementation of the BLUE algorithm, which ignores ECN and just drops, and is self-resetting.

> If so, would this provide for a faster fall-back to Reno than you considered in your analysis?

Not normally; the backlog limit is typically set to many RTTs' worth of queuing delay.  The mechanism has been carefully tested and works well, but rarely activates in practice.

> Alternatively, if fq_codel/CAKE were to treat ECT(1) as non-ECT (or implement the L4S style ECN) this would again not be an issue, I believe.

Perhaps, but I thought FQ AQMs weren't much of a problem to begin with - aside from the ingress-queuing problem?  I'm also reluctant to interpret ECT(1) or CE incompatibly with RFC-3168, unless given very good reason - and I currently have a good counter-argument.  If I may quote from that document, it does specifically contemplate use of the fourth codepoint to signal "mild congestion":

>    One possibility might have been for the data sender to use the fourth
>    ECN codepoint to indicate an alternate semantics for ECN.  However,
>    this seems to us more appropriate to be signaled using a
>    differentiated services codepoint in the DS field.
> 
>    A second possible use for the fourth ECN codepoint would have been to
>    give the router two separate codepoints for the indication of
>    congestion, CE(0) and CE(1), for mild and severe congestion
>    respectively.

It goes on to suggest problems with incremental deployment of such a scheme, but I believe SCE overcomes those problems elegantly.  As I have previously pointed out, the remaining benefits of L4S can be realised through assigning a DSCP to a PHB with the desired characteristics, and having something like DualQ implement that PHB.

I can't help noticing, meanwhile, that DualQ seems to be designed for use wherever a single-queue AQM would normally fit (though I have my doubts that it will work on much hardware already deployed at crucial locations).  Yet at the same time, you claim such single-queue AQMs are presently (and will remain) so rare that you haven't even developed a method of reliably detecting them.  This seems like an important discrepancy.

Meanwhile, work continues.

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Thu, 4 Apr 2019 09:40:44 +0000

On 4/4/19, 1:28 AM, "Jonathan Morton" <[email-masked]> wrote:
[...]    
    The present out-of-tree version of Cake supports an L4S-style high-fidelity ECN marking method, simultaneously with conventional ECN marking.  It does so using the SCE signalling method, with ECT(1) being an output rather than an input.  This is, in fact, the running code we had at the time of our TSVWG presentation.
    
[GW] Has any progress been made on a variant of Cake that utilizes the L4S ECT(1) semantic?  It would be useful for comparison purposes.  

[...]    
    > Alternatively, if fq_codel/CAKE were to treat ECT(1) as non-ECT (or implement the L4S style ECN) this would again not be an issue, I believe.
    
    Perhaps, but I thought FQ AQMs weren't much of a problem to begin with - aside from the ingress-queuing problem?  

[GW] Well, a goal would be for FQ AQMs to provide appropriate high-fidelity congestion signals to L4S flows so that they can achieve all four Ls and the S.  Failing that, to not provide weaker congestion signals for L4S flows as compared to classic TCP.  It isn't a problem from a fairness perspective, but the goal is low latency after all.    This seems like a fairly simple change to FQ.

    >    congestion, CE(0) and CE(1), for mild and severe congestion
    
[GW] I don't believe that the RFCs are on your side here.   RFC 3168 contemplates, and rejects, that definition of ECT(1), so I'm not sure how you read it as supporting the use. RFC 3168 explicitly defines ECT(1) as being a transport sender signal that it is "ECN Capable Transport".  I don't see how you can claim adherence to RFC 3168, yet use ECT(1) to mean (S)CE.  Moreover, RFC 3168 has been updated by (Standards-Track) RFC 8311, which enables the L4S experiments, and which the SCE approach also violates.  

[...] 
    I can't help noticing, meanwhile, that DualQ seems to be designed for use wherever a single-queue AQM would normally fit (though I have my doubts that it will work on much hardware already deployed at crucial locations).  

[GW] We have worked very closely with all of the DOCSIS equipment vendors to ensure that Low Latency DOCSIS (including dual-queue) is implementable via a firmware update to ALL existing DOCSIS 3.1 CMs and CMTS that are in the field today.  In some markets that is more than 90% of deployed CMTSs. I obviously can't speak to other access network technologies, but to the extent that forwarding architectures are similar, I would presume that doubling the number of queues is significantly simpler than multiplying by 1000.

    Yet at the same time, you claim such single-queue AQMs are presently (and will remain) so rare that you haven't even developed a method of reliably detecting them.  This seems like an important discrepancy.

[GW] The concern would be single-queue *ECN* AQMs. Single-queue packet drop AQMs do exist.

[GW] Why?  It seems to me that the L4S approach can be implemented easily in fq_codel and CAKE, as well as in systems that can only realistically support dual-queue.  It has been demonstrated to provide scalable throughput and ultra-low loss and queuing delay (which, I think, is the goal of SCE as well), and has quite a bit of support and momentum behind it. The philosophical debate as to which is better, dualq-coupled-aqm or FQ, can continue and implementers can choose whichever they like.  I don't doubt that the SCE signaling method could be used in a variant of DCTCP, but the *huge* downside is that it can only work on bottleneck links that are able (or willing) to support FQ, which will be precious few.  In 2-3 years' time, 10s of millions of actual bottleneck links in broadband networks will be supporting dual-queue.  It seems that the only real argument in favor of the SCE signaling method is the fallback in presence of single-queue classic ECN (unless you think blocking implementers from supporting dual-queue is some kind of perverse benefit).  Help analyzing (and addressing if need be) that situation would be the most effective way of achieving the shared goal of low latency scalable congestion control.  Continuing to push for (or worse, deploy) an incompatible solution just increases the risk that no one will be able to achieve that goal.

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Apr 2019 12:05:49 +0200

Hi Bob,

thanks for your detailed response.

> On Apr 3, 2019, at 19:40, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastien, (adding tsvwg, given this is all part of the decision over L4S/SCE),
> 
> [BB] I don't know whether you just looked at my slides without the voice track. But the voice track makes it clear that is exactly what I've been setting up.

	I also listened to the video, but it seems I was confused (I am easily confused). So I whole heartedly agree with this approach.

> My hypothesis is that the active measurements aren't seeing much CE, cos they don't generate enough traffic themselves to congestion the link and they would have to coincide with other traffic to s…

	I also note, that SQM for example defaults to ECN off on the path from the router towards the internet (rationale, with slow uplinks often seen at internet access links the serialization delay from a single packet can be painful already, this rationale might up for re-evaluation in the light of the newer graded types of responses to ECN marks at the core of the L4S/SCE proposals, but I digress). The consequence of this decision is that the likelyhood of seeing ECN marks is reduced making it harder to measure. If we had a significant number of sqm-devices in the field we might help with switching ECN on on the uplink as well, but we have no reliable way to estimate and no "telemetrics" from deployed instances

> Nonetheless the CE that Apple measured could be consistent with classic ECN marking from FQ-CoDel, e.g. in OpenWRT devices, which would protect competing flows from L4S flows.
> Apple saw CE "mainly in the uplink", which is likely to be FQ-CoDel. And the smaller amount seen in the downlink could have been peer-to-peer flows that had been marked on entry to the uplink by an FQ-CoDel home router.
> 
> With the help of others, I've devised a test for FQ vs FIFO, but we need to find where the CE-marking routers are before we apply the test (a scatter-gun approach would be infeasible given the need to congest the link). In discussions with Apple folks, we think it's best to run tests at the CDN end, to identify the network prefixes where CE is appearing (the client end used in the original tests usually doesn't know its public IP address).

	Sounds reasonable, with the potential unhappyness of ISPs for the congestion, but then this is similar to what a speedtest does (or netalyzr).

> 
> If we find any CE in the downlink from the CDN, that will disprove my peer-peer FQ hypothesis. But it won't prove that it's from a FIFO. The test for FQ vs FIFO is not straightforward, and I haven't yet worked out how we run it in the relevant direction over the paths where we find CE (not least 'cos that would involve collecting and identifying client IP addresses, which raises privacy concerns). I suspect we'll need to co-opt some volunteers in the affected localities, somehow.
> 
> An alternative to inline testing will be to find out whether the networks where CE is being seen have supplied their customers with routers with FQ-CoDel enabled. That won't prove conclusively that all the CE is coming from FQ-CoDel, but we can only go so far in trying to prove a negative.

	I wonder how are reading out the CE marking at all? Is the idea to place measurement devices between CDN end-hosts and the CPE and look at the IP headers, or is the plan to use a specific application?

> [BB] For the avoidance of doubt, I think you're using 'egress' to mean [home -> Internet] (I use it for the other direction, but I can work with your terminology).

	Sorry, this really is not my field so my nomenclature is probably idiosyncratic, but yes, I see this from the vantage point of the home network, and upload traffic egress from my network, while download traffic ingresses, from the other side of the access links, these direction can be defines the other way around.

> 
> In the [Internet -> home] direction, you are right that FQ on the home end would not isolate the access link from a misunderstanding of any CE marking. But I'm not suggesting it could or would.

	But that is the use-case I am concerned about, as with all the promises (better) AQM at the head-end brings, it will IMHO not eradicate the use of "ingress"-shaping on the customer side, as that is required to allow meaning-full prioritization and to establish the bandwidth-sharing model for the home network (whre typically per-flow-fairness is more desirable than no fairness, but where per-internal-IP-address-fairness might be even more desirable, Jonathan's cake actually offers that capability, but since this absolutely requires access to the internal-IP-addresses will require the AQM ti live inside the NAT typical for IPv4 access links).

> The question is about the machine at the Internet side of the access link (BRAS, CMTS, etc) that is queuing traffic into the access link towards the home. We can be nearly certain it is not doing FQ (these machines serve thousands of customers, so I don't believe any use FQ). The question is whether we can find one of these non-FQ machines that has an AQM enabled that does CE marking.
> 
> That's what we're trying to find. It's of course hard to prove a negative - that none exist. All we can hope to do is measure a sample of the paths we find.

	Okay, but any idea what to do about my hobbyhorse, short of completely disabling L4S for all nodes in the home network (which seems heavy-handed, given that L4S's goal "low latency networking" is IMHO desirable for all end-users)? This kind of set-up is not as unique as it might seem, and typically users employing it are latency sensitive and hence certainly part of the audience which might like the L4S approach other-wise.

> 
> 
> A nit: DCTCP is not /unresponsive/. It responds to congestion, but as if it is equivalent to a number of 'classic' (Reno-friendly) TCP flows.

	I agree, hence the term "appear unresponsive", the problem isn't DCTCP here, but a misunderstanding between AQM and involved TCP.

>> I would be delighted to learn where my assumptions are wrong...

	Thanks for your kind and informative response.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Apr 2019 12:18:38 +0200

Hi Jonathan,

> On Apr 3, 2019, at 20:28, Jonathan Morton <[email-masked]> wrote:
> 

	+1

> A lower rate of observed marking on downloads could be due to the greater difficulty of driving the download direction into saturation, especially given a backhaul network that may be oversubscribe…

	It is a trade-off, since any upstream AQM will in all likely not be configurable by the end-user, even if one might dream of AQM as a service kind of thing... (there probably is no real market for that...)

> The motivating reason for ingress shaping is that, generally, the BRAS or equivalent device in the ISP does not perform AQM at all (though some do policing, which doesn't involve ECN).

	I happen to believe that even if the upstream end would/will offer AQM there still is going to be a desire for detailed configuration, especially in light of on-line gaming (of the reaction-time bound variety; given that many games do not even offer IPv6, I see not flag-day change to L4S or alternatives, unless Microsoft/Sony are on board and will update their console OSs in a timely fashion).

> In my experience there is often a "provisioning shaper" that is essentially a TBF with a dumb tail-dropping FIFO attached; I once suffered one on a rate-limited 3G service which had about 40 second…

	Current ISP, offers up to 300ms delay during a simple speedtest, better than 4000ms, but certainly only useful as a worst-case limiter...

> I did spend a few minutes today sketching out what happens if an L4S flow hits fq_codel on an ingress shaper.  The theoretical results are that, in congestion-avoidance mode, the L4S flow takes about twice as long to drain the dumb bottleneck queue as a Classic ECN flow after the onset of saturation.  

	As much as care, that probably is still sort of acceptable.

> But the real shocker is what happens in the terminal phase of slow-start, where a rapid correction is required after the minimum 2x overshoot in exponential growth (because it takes an RTT for the CE/ECE/CWR signal to circulate and take effect).
> 

	It would be great to avoid that situation.

Sebastian Mioeller

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Apr 2019 12:29:47 +0200

Hi Greg,

> On Apr 4, 2019, at 01:28, Greg White <[email-masked]> wrote:
> 

[SM]: I happen to believe that head-end AQM will ameliorate the need for an ingress AQM for a lot of casual users, but all those that are constantly running against capacity limits (think ADSL) typically desire a more fine-grained prioritization than a head-end AQM is going to offer, includsing restricting bandwidth to certain proiorety classes to say keep latency increas for game traffic tightly bound. I believe in that case the guarantees L4S offers are not sufficient, as the gaming case requires "guaranteed bandwidth at a certain latency level" as compared to the low_queueung latency guarantee L4S offers (which basically moves the que back into the sending end-host, which is fine, but all it does not help if I need to send @ X Hz).

> [GW] One of the optimizations being investigated for L4S (both TCP/QUIC Prague and (I think) BBRv2) is a revised startup process that avoids the 2x overshoot from slow-start.  Nonetheless, you are right that an issue to be understood (and addressed?) with TCP Prague / fq_codel compatibility is how to prevent TCP Prague seeing worse latency than classic TCP at flow startup.  As you indicated, once the Prague sender falls back to Reno behavior, things are fine.  
> In fq_codel/CAKE currently, does the AQM switch from CE mark to packet drop at some higher threshold of sojourn time?  If so, would this provide for a faster fall-back to Reno than you considered in your analysis?  Alternatively, if fq_codel/CAKE were to treat ECT(1) as non-ECT (or implement the L4S style ECN) this would again not be an issue, I believe.  

[SM] I believe this to be just as viable as having L4S AQMs interpret ECT(1) as pulse-modulated continuous congestion information. In the end, after the dust has settled, one of these is going to happen, as there is only going to be a single graded congestion signaling (if at all). IMHO the jury is still out which approach will win.

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Apr 2019 12:52:55 +0200

> On Apr 4, 2019, at 11:40, Greg White <[email-masked]> wrote:
> 
> 
> 
> [GW] I don't believe that the RFCs are on your side here.   RFC 3168 contemplates, and rejects, that definition of ECT(1),

[SM I believe this to be caused by not considering pulse modulation though, as the argument is that a single code-point might not be "rich" enough to signal graded congestion information, but IMHO DCTCP shows that this is solved. In any way RFC 3168 seems a bit dated for this discussion as all agree to go beyond it, no?

> so I'm not sure how you read it as supporting the use. RFC 3168 explicitly defines ECT(1) as being a transport sender signal that it is "ECN Capable Transport".  I don't see how you can claim adherence to RFC 3168, yet use ECT(1) to mean (S)CE.  Moreover, RFC 3168 has been updated by (Standards-Track) RFC 8311, which enables the L4S experiments, and which the SCE approach also violates.  
> 
> [...] 

	But why, in the end memory is linear and your proposed mandatory "Annex P Queue Protection Algorithm (Normative") seems to care all the costs of tracking flows already, so why is that considerably cheaper than using that information also for scheduling purposes?

> [GW] The concern would be single-queue *ECN* AQMs. Single-queue packet drop AQMs do exist.

[SM] What about my hobby-horse "ingress-ECN-aware-fq-AQM"?

> [GW] Why?  It seems to me that the L4S approach can be implemented easily in fq_codel and CAKE, as well as in systems that can only realistically support dual-queue.

[SM] if that system has memory and a CPU it _can_ support fq as well, that is a _choice_ not an _impossibility_. It might be un-economically, sure. But again, why will such a device be able to do Queue Protection then?

>  It has been demonstrated to provide scalable throughput and ultra-low loss and queuing delay

Which sounds greater than it is since most applications care for end-to-end latency, and L4S basically moves (a part of) the queueing delay from the bottleneck AQM back into the sending application, it does not remove that latency completely.

> (which, I think, is the goal of SCE as well), and has quite a bit of support and momentum behind it. The philosophical debate as to which is better, dualq-coupled-aqm or FQ, can continue and implementers can choose whichever they like.  I don't doubt that the SCE signaling method could be used in a variant of DCTCP, but the *huge* downside is that it can only work on bottleneck links that are able (or willing) to support FQ,

[SM] Why, the AQM will simply supply all ECN enabled with the graded information about the queueing load of the AQM, it is up to the endpoints to make sense out of this.

> which will be precious few.  In 2-3 years' time, 10s of millions of actual bottleneck links in broadband networks will be supporting dual-queue.

[SM]	I guess most bottlenecks nowadays support a variant of RED but support dies not equal exercise it. But sure I like how the DOCSIS standards-process  can influence a large swath of critical bottleneck links, more power to you.

>  It seems that the only real argument in favor of the SCE signaling method is the fallback in presence of single-queue classic ECN (unless you think blocking implementers from supporting dual-queue is some kind of perverse benefit).  

[SM] I see the SCE proposal as the better more evolutionary approach forward, the L4S approach basically spells, the current state is unrecoverable and we need to start fresh, but then does no much of that sort (re-using PIE and DCTCP is also nicely evolutionary).

> Help analyzing (and addressing if need be) that situation would be the most effective way of achieving the shared goal of low latency scalable congestion control.  Continuing to push for (or worse, deploy) an incompatible solution just increases the risk that no one will be able to achieve that goal.

[SM] these wise words typically apply to both sides of a discussion and have just as much merits one way or he other. I note that in reality we often see something else, alternative approaches duking it out and driving their evolution exactly because they are not the only game in town....

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 4 Apr 2019 14:00:14 +0300

> On 4 Apr, 2019, at 12:40 pm, Greg White <[email-masked]> wrote:
> 
> I don't doubt that the SCE signaling method could be used in a variant of DCTCP, but the *huge* downside is that it can only work on bottleneck links that are able (or willing) to support FQ, which will be precious few.

Let me stop you right there.  You are remembering a discussion we had at least two weeks ago, based on an early concept of SCE which has since been superseded.

We have running code which converges rapidly to fairly sharing the link between two SCE flows, with only a single-queue SCE-aware AQM.  That is, we used the "besteffort" and "flowblind" modes of Cake, which disable its Diffserv and flow-isolation capabilities, specifically to test that aspect of our work.  Since our prototype SCE response is DCTCP-like, that result should not be a surprise to you.

We *also* believe we have a solution for fairly sharing a single queue between SCE and non-SCE flows.  We haven't implemented and tested it yet, however.  Work continues.

> (unless you think blocking implementers from supporting dual-queue is some kind of perverse benefit)

Dual-queue AQM can still be supported through Diffserv, as I have repeatedly pointed out.  The associated PHB can specify the "short queue" and "no reordering" behaviours you're so keen on.

Now, reading back up your post:

> Has any progress been made on a variant of Cake that utilizes the L4S ECT(1) semantic?  It would be useful for comparison purposes.

We have not done so.  The code is freely available if you wish to try it yourself; it should not be difficult.  Cake also offers an excellent platform for exploring the behaviour of single-queue ECN-aware AQMs, which I think you would find valuable.

> I don't believe that the RFCs are on your side here.  RFC 3168 contemplates, and rejects, that definition of ECT(1)…

It also contemplates, rejects, and suggests a better alternative for L4S' use of ECT(1), right there in the quote.  And on the subject of RFC-8311, that document contains a section entitled "Effective Congestion Control Is Required", and it specifically leaves alone the several suggested uses of ECT(1) in its edits to RFC-3168.

I note in passing that SCE has the support of the TSVWG Chairs.  That is sufficient motivation to continue work to establish its true merits and limitations.

 - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Thu, 4 Apr 2019 13:52:54 +0200

Several things...

I find discussions of ECN in general very stressful, as I am firmly on
the ecn-sane "yellow" team. ECN scares me on the general internet.
More than anything else, I'd like to get a big DDOS protection
provider and university to tackle all my outstanding issues with using
ECN in a general deployment - ranging from what happens with tcp syn
ecn enabled attacks, to ddos protection from udp based packet floods,
as I outlined in the ecn-sane project's charter:
https://www.bufferbloat.net/projects/ecn-sane/wiki/ before I can
endorse this work.

I have friends that regularly deal with 2TByte/sec DDOS attacks and
they have worse blood pressure problems than I do.

I've long had no problem with ecn in specialized deployments, under
rigidly controlled conditions. The SCE work, in particular, looks to
be useful for technologies like nvme over tcp and roce and stuff like
that, and I look forward to results there.

One of the proudest things I have ever had going in my life is that
the backhauls and groundstations for the arkyd-6 spacecraft, used
fq_codel with ECN enabled... running lede (openwrt). And though I
can't talk about that project much, and probably shouldn't have
mentioned even that much here  - I tend to feel that any non-backward
compatible change to ecn will muck with *my* bird in a 25 year orbit,
and I kind of take it a bit personally any naive interference with
that contribution to the US space program.

I am unsubscribing from the ietf lists as of now, so I can focus on
getting testbeds and analytical tools up, as well as finishing up my
work on the ipv4 unicast extensions project, which is entirely
separate, that I wish I could get more folk to pay attention to.

If there is a productive conversation of interest to me, here, using
debating techniques I can live with (
https://www.bufferbloat.net/projects/ecn-sane/wiki/rules/ ) please
either send a link or cc [email-masked].

Thank you.

Dave

---

From: Greg White <[email-masked]>
Date: Thu, 4 Apr 2019 15:37:02 +0000

[GW] Fairly sharing the link wasn't the concern that I was alluding to by saying that SCE only works with FQ.  My presumption was that this would be possible.   My concern is that the SCE sender gets *zero* benefit from supporting SCE signaling unless the bottleneck link is an FQ link.   In my view the IETF should not consider burning the last code point for such a limited purpose.  The L4S definition provides *value* (i.e. ultra-low latency) when the bottleneck is either FQ or DualQ.

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 4 Apr 2019 18:43:17 +0300

> On 4 Apr, 2019, at 6:37 pm, Greg White <[email-masked]> wrote:
> 
> My concern is that the SCE sender gets *zero* benefit from supporting SCE signaling unless the bottleneck link is an FQ link.

Our early results suggest otherwise.  However, I would like to flesh them out before further detailed discussion.

 - Jonathan Morton

---

From: Greg White <[email-masked]>
Date: Thu, 4 Apr 2019 17:06:18 +0000

On 4/4/19, 11:29 AM, "Sebastian Moeller" <[email-masked]> wrote:

    Hi Greg,

[GW]  In an L4S world, there is nothing that prevents a user running an "ingress AQM" of their choice with whatever traffic controls they like (as long as the ECT(1) implementation lines up with the L4S definition, which is totally doable in fq_codel and CAKE).    The issue we have been discussing is where the user is running an old version of fq_codel/CAKE that hasn't been updated to the L4S version of ECT(1).  In that case, the L4S traffic would see worse queuing latency than classic TCP.

[...]
    > [GW] Alternatively, if fq_codel/CAKE were to treat ECT(1) as non-ECT (or implement the L4S style ECN) this would again not be an issue, I believe.  
    
    [SM] I believe this to be just as viable as having L4S AQMs interpret ECT(1) as pulse-modulated continuous congestion information. [...]

[GW] How?  That seems impossible to me.  What is the classifier that directs traffic to the LL queue vs Classic?

---

From: Greg White <[email-masked]>
Date: Thu, 4 Apr 2019 17:29:13 +0000

I will be interested to see how this is possible.  When an SCE sender and a non-ECT sender share a single queue, the non-ECT sender will be driving the queue delay to the classic AQM target, and the SCE sender will experience the same queue delay as the non-ECT sender.   I've not been able to imagine a dual queue implementation that can identify SCE senders so that they can achieve reliable ultra-low delay (unless you are suggesting that SCE senders are all required to mark their traffic with a particular DSCP).

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 4 Apr 2019 19:48:05 +0200

Hi Greg,

> On Apr 4, 2019, at 19:06, Greg White <[email-masked]> wrote:
> 
> [GW]  In an L4S world,

	But we are not living in an L4S world, so it would behoove L4S to play nicely with the existing world, no?

> there is nothing that prevents a user running an "ingress AQM" of their choice with whatever traffic controls they like (as long as the ECT(1) implementation lines up with the L4S definition, which is totally doable in fq_codel and CAKE).    

	So "my way or the highway", eh?

> The issue we have been discussing is where the user is running an old version of fq_codel/CAKE that hasn't been updated to the L4S version of ECT(1).  In that case, the L4S traffic would see worse queuing latency than classic TCP.

	In the case of ingress Shaping this is simply not true and L4S flows have the potential to aggressively use more than their fair share in a way that fq post-bottleneck can only react to too slowly. This is a use case that exists in the real world right now, and I dislike the idea that this will only continue to be possible under the assumption that I need to upgrade my AQM.

> [GW] How?  That seems impossible to me.  What is the classifier that directs traffic to the LL queue vs Classic?  

Well, my point is, 1 single code-point does not make a robust classifier, as seen in the fact that the ECT(1) flow identity does not survive a CE-marking, Really the L4S RFCs are quite verbose in showing why ECT(1) can not serve as a reliable identifier of L4S-flows, only to then turn around and argue why this does not matter. I would propose to simply use a new protocol and you have a perfect identifier in the IP header that can unambiguously identify your flows, problem solved. And once you done that you do not need to overlay two different meanings on the CE mark anymore, just teach your AQM to set a field in the new protocol's header and you are set (for bonus you could use 8 bits and directly encode a graded load signal which would allow even more fidelity and temporal precision than essentially doing PCM, no). That wat CE can keep its meaning and L4S and the TCP-friendly world stay nicely separated. Now, you might bring tunneling as an example, to which I say let's cross that bridge once we get there.
That seems like a clean implementation for a new protocol, IMHO. Is that easy to roll out, hell no, but it is not unique in that regard.

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 4 Apr 2019 20:54:48 +0300

> On 4 Apr, 2019, at 8:29 pm, Greg White <[email-masked]> wrote:
> 
> When an SCE sender and a non-ECT sender share a single queue, the non-ECT sender will be driving the queue delay to the classic AQM target, and the SCE sender will experience the same queue delay as the non-ECT sender. 

This is true.  However, this is no worse a situation, in terms of peak delay, than a single-queue AQM is today.

The immediate benefit to SCE is the ability to use path capacity left unused by a conventional multiplicative-decrease response, without incurring a latency hit relative to conventional TCP, and without losing TCP-friendly characteristics.  This is easily seen when a single SCE flow occupies the link (we measured roughly 50% greater goodput versus plain NewReno over 30 seconds), and is not materially degraded when a second SCE flow joins it.

What we have not yet had the opportunity to test is the behaviour of an SCE flow versus a non-SCE flow sharing a single queue.  The interactions here are expected to be subtle, involving a mixture of SCE and CE responses.  An important property, however, is that we are sure that conventional TCPs will not be squeezed out of the path by SCE flows - rather, we are working to avoid the opposite scenario.

Just as soon as GitHub stops breaking on us…

> I've not been able to imagine a dual queue implementation that can identify SCE senders so that they can achieve reliable ultra-low delay (unless you are suggesting that SCE senders are all required to mark their traffic with a particular DSCP).

SCE senders are not required to use any particular DSCP.  Nor are they enjoined from doing so.

Understand this: SCE's primary purpose is not to ensure minimal queue delays in all possible circumstances.  It is to improve the quality of congestion control in a thoroughly backwards-compatible manner.  Relying too heavily on Diffserv, which is rarely implemented at bottlenecks today, would defeat that goal.

The combination of SCE with a DSCP associated with an appropriate PHB could work with your preferred DualQ architecture, as an *enhancement* to standard SCE or ECN behaviour.  The only additional requirement is that the DSCP survives the path as far as the bottleneck.  I suggest that the cable industry is in a better position than most Internet entities to solve the latter problem.

 - Jonathan Morton