Subject: [tsvwg] RTT-independence (was: L4S vs SCE)

From: Bob Briscoe <[email-masked]>
Date: Wed, 18 Dec 2019 00:23:28 +0000

Sebastian,

I wanted to pick up your point about RTT-independence that you recently 
said has never been responded to.

I have changed the subject line, because this is not an L4S vs SCE 
issue. With a shared queue it is just as much an issue for SCE as for 
L4S. With an FQ scheduler, it is just as much a non-issue for L4S as for 
SCE.

On 20/11/2019 14:18, Sebastian Moeller wrote:
> Dear Koen, dear all,
>
>
>> On Nov 20, 2019, at 13:40, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]> wrote:
>>
>>
>> One of these opportunities here is to make L4S_TCP less RTT dependent. There have been many working implementations for less RTT-dependent CCs in the past. One that is widely deployed is Cubic, which is doing this for getting more throughput for longer RTTs. The only reason why it didn’t fly for lower RTTs on the current Internet is that they would hurt themselves (get lower throughput, competing with Reno).
> 	[SM] Looking at the pfifo_fast results in Høiland-Jørgensen T, Hurtig P, Brunstrom A (2015) The Good, the Bad and the WiFi: Modern AQMs in a residential setting. Computer Networks 89:90–106. For Cubic/pfifo_fast (linux former default combination), I fail to see a strong indicator that cubic is RTT invariant or getting more thoughput at longer RTTs (except for the 10ms versus 50ms "hump"). What paper/data should I be looking at instead?
>
>>   
>> As we are able to define a new independent behavior and the RTT dependence in L4S is taken serious (some even call it a show stopper) this is even a must do opportunity for L4S. It is all a matter of perspective: show-stopper <-> opportunity.
> 	[SM] I believe that working on more RTT independence is worth-while but no replacement for fixing the isolation system as it it is that equitable isolation from existing traffic that basically gives you the slate clean-"green field" development opportunity, no?

[BB] RTT-dependence is caused by the end-system, and should be solved in 
the end-system (whether or not FQ is also deployed to solve it in some 
places). Importantly, RTT-dependence only needs to be addressed in the 
newly deployed scalable (L4S or SCE) end-systems, not pre-existing 
end-systems like Reno & Cubic, as I'll now explain.

The problem is due to the tiny RTT you can get when you have both a tiny 
base RTT and a tiny queue. For example:

	Base RTT/ms
	Queue/ms
	Total RTT/ms
Close Scalable flow: 	2.5
	0.5
	3
Distant Scalable flow: 	100
	0.5
	100.5
Close Classic flow: 	2.5
	15
	7.5
Distant Classic flow: 	100
	15
	105

If all the above flows were RTT-dependent, the lowest 3ms RTT ('Close 
Scalable') would be the potential hog. For example against 105ms 
('Distant Classic') if they were both 'window fair' like Reno, the 
throughput ratio would be 105:3 = 35:1

Whereas the Close vs Distant Classic case is cushioned by the 15ms 
queue. RTT ratio = 105:7.5 = 14:1, which was the Internet status quo 
pre-L4S (or SCE).

So why is it enough for only scalable flows to reduce their RTT 
dependence? Assuming they do, the lowest 3ms RTT ('Close Scalable') 
doesn't go much faster than 'Distant Scalable'. If it's no longer a hog 
in relation to Distant Scalable, it's no longer a hog in relation to any 
of the other flows. So the worst case reverts to the Close Classic flow, 
which isn't so bad because the total RTT is cushioned by the queue. So 
we will be no worse off than we were before L4S or SCE.

You will see we explained this at the end of Section V.B 'Congestion 
Control Roadmap' in the draft journal paper about L4S that we posted 
earlier this year:
“`Data Centre to the Home’: Deployable Ultra-Low Latency for All 
<http://bobbriscoe.net/projects/latency/dctth_journal_draft20190726.pdf#subsection.5.2>”

But is RTT-independent TCP Prague real? No, not quite yet. At the time 
we designed this, we simulated it, but we're now working on it again, 
including implementing it in Linux, so you should see results soon. The 
approach to RTT-independence that we suggest is explained in section 3.2 
of a paper we presented to the iccrg back in Jul 2017.
“Resolving Tensions Between Congestion Control Scaling Requirements 
<https://riteproject.files.wordpress.com/2015/10/ccdi_tr.pdf#subsection.3.2>“

All the above papers are available via the L4S landing page: 
https://riteproject.eu/dctth/#papers
>> I'm also not against the recent vibrant energy, interest and engagement of people, but I think we can better use this energy in making things work. As you noticed, we can use the help to speed things up on the open source part.
> 	[SM]... while at the same time requesting help for implementation and dealing with the consequences of said "code point and requirements".

If you recall, when I asked you not to keep repeating other people's 
arguments without adding anything, you said the RTT-dependence point was 
your contribution to the debate.

The L4S team identified this problem in early 2015 during testing with 
diverse RTTs. When the Prague requirements were first agreed, we (Koen 
in particular) insisted that this should be included as a basic safety 
requirement, even tho some people said it wasn't important. The WG has 
so far kept it as a MUST requirement, justified by the potentially large 
discrepancy in rates that can result, articulated here:
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08#appendix-A.1.5

If I were as cynical as some on this list, I could point out that 
discovering a requirement by reading one of the L4S documents doesn't 
really count as a contribution. But I'll let that pass.

What I wanted to show by this email is that a significant amount of work 
has been done on this, the problem is well-understood and solutions are 
in progress. RTT-independent congestion controls have been produced 
before - not that I'm belittling how hard it is to resolve the tension 
between requirements identified in the 'Tensions' paper above.

So I ask you to take a more constructive approach. Banging on your 
keyboard in the style of "ner ner ne ner ner, I found a problem with 
L4S" doesn't help anyone, when the problem you've "found" exists with 
any low latency shared queue, including an SCE-based solution. Instead 
you could have read all the papers referenced from the L4S landing page, 
understood how much everyone already understands the problem, how much 
has already been done to solve it, and then you could have tried to work 
out the details of a full solution.

Regards

Bob

>
>
> Best Regards
> 	Sebastian
>
>
>>   
>> Regards,
>> Koen.
>>   
>> From: Ingemar Johansson S <[email-masked]>
>> Sent: Wednesday, November 20, 2019 11:50 AM
>> To: Kyle Rose <[email-masked]>; Ingemar Johansson S <[email-masked]>
>> Cc: Sebastian Moeller <[email-masked]>; G Fairhurst <[email-masked]>; [email-masked]; [email-masked]; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; [email-masked]; Ingemar Johansson S <[email-masked]>
>> Subject: RE: [tsvwg] L4S vs SCE
>>   
>> Hi
>>   
>> So given the imagined outcome that L4S fails.. two scenarios
>>   
>> If other SDOs or developers don’t pick up L4S then things are quite simple I guess, just declare the L4S drafts as deprecated, or is there more to do ?
>>   
>> But, if e.g. 3GPP somehow thinks that this is a good idea and adopts it.. Will the IETF send a message (LS?) to 3GPP with the message “please stop using L4S”, is this even a reasonable scenario?. After all, the fact that it is picked up by other SDOs, speaks against a failure ?
>>   
>> /Ingemar
>>   
>> From: Kyle Rose <[email-masked]>
>> Sent: den 20 november 2019 11:14
>> To: Ingemar Johansson S <ingemar.s.johansson=[email-masked]>
>> Cc: Sebastian Moeller <[email-masked]>; G Fairhurst <[email-masked]>; [email-masked]; Ingemar Johansson S <[email-masked]>; [email-masked]; De Schepper, Koen (Koen) <[email-masked]>; [email-masked]
>> Subject: Re: [tsvwg] L4S vs SCE
>>   
>> On Wed, Nov 20, 2019 at 6:04 PM Ingemar Johansson S <ingemar.s.johansson=[email-masked]> wrote:
>>>        How do you expect an industry/commercial roll-out of L4S
>>> technology to behave, if the L4S experiment should terminated without
>>> adoption as a standard track RFC? Are they supposed to phase-out using
>>> ECT(1) as well, or is it understood that deployed L4S instances continue using
>>> L4S methods?
>> [IJ] The premise would be that L4S is declared a failure. I doubt that anybody has a good enough crystal ball to predict what happens. First it is necessary to come to the conclusion that L4S is a failure and I would argue that we are not yet there and I don't currently see that coming. Before that possible event I don't see it meaningful to speculate.
>>   
>> I'm going to have to go ahead and disagree with you strongly here. Given the potential consequences of cleaning up after a failed experiment without a plan worked out beforehand, this blithe approach is simply not acceptable.
>>   
>> In lots of cases, experiments are easy to terminate in an obvious way: for example, in one typical case, a code point can simply be abandoned, or (even better) a pollutable experimental code point returned to the available pool after some time. If that were the case here, it would not be difficult to enumerate a sequence of steps required to do so. It doesn't appear that's the case, however, so all the more reason to make sure we address this as part of the experiment setup.
>>   
>> A launch escape system of the necessary complexity should be a requirement of any experimental deployment. In this case, that might circumscribe the scope of the experiment itself.
>>   
>> Kyle
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 18 Dec 2019 09:11:48 +0100

Hi Bob,
 more below in-line.

> On Dec 18, 2019, at 01:23, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> I wanted to pick up your point about RTT-independence that you recently said has never been responded to. 
> 
> I have changed the subject line, because this is not an L4S vs SCE issue. With a shared queue it is just as much an issue for SCE as for L4S. With an FQ scheduler, it is just as much a non-issue for L4S as for SCE. 
> 
> 
> [BB] RTT-dependence is caused by the end-system, and should be solved in the end-system (whether or not FQ is also deployed to solve it in some places).

	[SM] I fully agree, and that is also a convincing argument, why increasing RTT independence can not be the solution for the observed isolation failure of the dual queue coupled AQM, glad that you agree.

> Importantly, RTT-dependence only needs to be addressed in the newly deployed scalable (L4S or SCE) end-systems, not pre-existing end-systems like Reno & Cubic, as I'll now explain. 
> 
> The problem is due to the tiny RTT you can get when you have both a tiny base RTT and a tiny queue. For example:
> 
> Base RTT/ms
> Queue/ms
> Total RTT/ms
> Close Scalable flow:	2.5
> 0.5
> 3

	[SM] Note, 15 + 2.5 = 17.5, a typo for sure. If I am wrong and you really ment 7.5, please elaborate why you only add 1/3 of the queueing delay, as I assume we are talking about saturating traffic here.

> Distant Classic flow:	100

	[SM] 15 + 105 = 120

> 
> If all the above flows were RTT-dependent, the lowest 3ms RTT ('Close Scalable') would be the potential hog. For example against 105ms ('Distant Classic') if they were both 'window fair' like Reno, the throughput ratio would be 105:3 = 35:1

	[SM] I am not convinced that "window fairness" is in reality a useful property, throughput fairness seems much more desirable.

> 
> Whereas the Close vs Distant Classic case is cushioned by the 15ms queue. RTT ratio = 105:7.5 = 14:1, which was the Internet status quo pre-L4S (or SCE).

	[SM] If I fill in the modified numbers from above I get 115 / 17.5  ~ 6.6:1, but your point still stands.

> 
> So why is it enough for only scalable flows to reduce their RTT dependence? Assuming they do, the lowest 3ms RTT ('Close Scalable') doesn't go much faster than 'Distant Scalable'. If it's no longer a hog in relation to Distant Scalable, it's no longer a hog in relation to any of the other flows. 

	[SM] That seem strictly depend on your definition of hog 105/120 is not equal still giving L4S an IMHO (hard to justify) edge over standard compliant traffic. But I am happy to be convinced by actual empirical numbers.

> So the worst case reverts to the Close Classic flow, which isn't so bad because the total RTT is cushioned by the queue. So we will be no worse off than we were before L4S or SCE.
> 
> 
> You will see we explained this at the end of Section V.B 'Congestion Control Roadmap' in the draft journal paper about L4S that we posted earlier this year:
> “`Data Centre to the Home’: Deployable Ultra-Low Latency for All”
> 
> But is RTT-independent TCP Prague real? No, not quite yet. At the time we designed this, we simulated it, but we're now working on it again, including implementing it in Linux, so you should see results soon. The approach to RTT-independence that we suggest is explained in section 3.2 of a paper we presented to the iccrg back in Jul 2017. 
> “Resolving Tensions Between Congestion Control Scaling Requirements“

	[SM] At this point I want to see real numbers from empirical testing, thanks for the references though.

> 
> 
> All the above papers are available via the L4S landing page: https://riteproject.eu/dctth/#papers
> If you recall, when I asked you not to keep repeating other people's arguments without adding anything, you said the RTT-dependence point was your contribution to the debate.

	[SM] Then we miscommunicated, my point is, that if you fix the dual queue couplked AQM than you can tackle the worthy goal of less RTT dependence at your own sweet time.

> 
> The L4S team identified this problem in early 2015 during testing with diverse RTTs.

	[SM] And then opted to keep pushing for internet-wide roll-out of L4S with this issue unsolved, color me not impressed.

> When the Prague requirements were first agreed, we (Koen in particular) insisted that this should be included as a basic safety requirement, even tho some people said it wasn't important.
> The WG has so far kept it as a MUST requirement, justified by the potentially large discrepancy in rates that can result, articulated here:
> If I were as cynical as some on this list, I could point out that discovering a requirement by reading one of the L4S documents doesn't really count as a contribution. But I'll let that pass.

	[SM] Except, you just did not let it pass... I will accept this as a passive aggressive offense.

> 
> What I wanted to show by this email is that a significant amount of work has been done on this, the problem is well-understood and solutions are in progress. RTT-independent congestion controls have been produced before - not that I'm belittling how hard it is to resolve the tension between requirements identified in the 'Tensions' paper above. 

	[SM] To repeat myself, this all supports the thesis that making RTT-independence the mechanism by which L4S equitably shares with standards-compliant traffic at the current state is premature. I very much would like to see the dual queue coupled AQM fixed, and once RTT independence (for all users of the L4S queue) is functional then the AQM can be changed again. 

> 
> So I ask you to take a more constructive approach. Banging on your keyboard in the style of "ner ner ne ner ner, I found a problem with L4S" doesn't help anyone,

	[SM] Let me be blunt, without the deserved push-back on the lists, "peaceful coexistence with single queue RFC3168 AQMs" would not have been tackled, the gist of the presentatuins so far has always been to ask the room, but do we ned to care for this at all? So IMHO pointing out L4S current short comings is important as otherwise little progress on the difficult issues would happen at all.

> when the problem you've "found" exists with any low latency shared queue, including an SCE-based solution. 

	[SM] Well, the something called "dual queue" AQM certainly needs to at least behave as if it had two queues in my book. L4S versus SCE is red herring here, my current issue is that the current implementation of L4S's reference AQM does not work as it should and as a normal reader would understand from reading the dualQ ID, IMHO that needs to be fixed. I do not believe that the not yet solved RTT-independence goal is the best approach to solve the issue, but you are free to spend your time as you please. 

> Instead you could have read all the papers referenced from the L4S landing page, \\

	[SM} Regrettably, I have read quite a lot of that material, and I typically see too much hype and too little critical analysis for my taste.

> understood how much everyone already understands the problem,

	[SM] I have no beef with your understanding of the issues, my complaint is that you do not act upon long-standing well-documented show-stopper issues. Not every documented bug can be dressed as a feature.

Best Regards
	Sebastian

> how much has already been done to solve it, and then you could have tried to work out the details of a full solution.
> 
> 
> Regards
> 
> 
> 
> Bob
> 
> 
>> 
>> 
>> Best Regards
>> 	Sebastian
>> 
>> 
>> 
>>>  
>>> Regards,
>>> Koen.
>>>  
>>> To: Kyle Rose 
>>> <[email-masked]>; Ingemar Johansson S <[email-masked]>
>>> 
>>> Cc: Sebastian Moeller 
>>> <[email-masked]>; G Fairhurst <[email-masked]>; [email-masked]; [email-masked]; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>; [email-masked]; Ingemar Johansson S <[email-masked]>
>>> 
>>> To: Ingemar Johansson S 
>>> <ingemar.s.johansson=[email-masked]>
>>> 
>>> Cc: Sebastian Moeller 
>>> <[email-masked]>; G Fairhurst <[email-masked]>; [email-masked]; Ingemar Johansson S <[email-masked]>; [email-masked]; De Schepper, Koen (Koen) <[email-masked]>; [email-masked]
>>> 
>>> On Wed, Nov 20, 2019 at 6:04 PM Ingemar Johansson S 
>>> <ingemar.s.johansson=[email-masked]>
>>>  wrote:
>>>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 29 Dec 2019 23:25:26 +0000

Sebastian,

> 	[SM] I fully agree, and that is also a convincing argument, why increasing RTT independence can not be the solution for the observed isolation failure of the dual queue coupled AQM, glad that you…
[BB] There is no convincing argument yet. This is an opening sentence, 
from which neither I nor you can make any convincing argument. Please 
work through the whole argument below...

>> If all the above flows were RTT-dependent, the lowest 3ms RTT ('Close Scalable') would be the potential hog. For example against 105ms ('Distant Classic') if they were both 'window fair' like Reno…
[BB] Sorry, I screwed up that table. I decided to use a 5ms queue 
instead of a 15ms queue at the last minute (in order to use the worst 
case for my argument), however I only changed the totals, and omitted to 
change the queue itself. Here's the corrected table:

	Base RTT/ms
	Queue/ms
	Total RTT/ms
Close Scalable flow: 	2.5
	0.5
	3
	5.0
	5.0
	105

All the arguments that followed remain the same though, because they 
relied on the totals not the constituent parts of the totals.

> 	[SM] I am not convinced that "window fairness" is in reality a useful property, throughput fairness seems much more desirable.
[BB] 'Window fairness' is not a useful property, it's just a property of 
a large class of congestion controls. The word 'fairness' sounds 
intrinsically like people should think it is desirable. So I try to 
avoid using it, and if I do, I try to always put it in quotes.

> 	[SM] If I fill in the modified numbers from above I get 115 / 17.5  ~ 6.6:1, but your point still stands.
[BB] Please now revisit the arguments below without the distraction of 
the typos.
> 	[SM] That seem strictly depend on your definition of hog 105/120 is not equal still giving L4S an IMHO (hard to justify) edge over standard compliant traffic. But I am happy to be convinced by act…
[BB] Oh dear. What is so important to you about such precise rate 
equality? Please be prepared to set aside your preconceptions about rate 
equality. Please re-read the reference at {Note 1}. No-one has ever been 
able to justify a goal of equal flow-rates, except for starvation 
avoidance (when rough rate equality is sufficient). Congestion levels 
have to be high for starvation to be a concern. So, the lower congestion 
is, the less important rate equality becomes.

There's a lot more to this, so please read and try really hard to 
understand /all/ the points in that reference at {Note 1}.

For instance 'rate fairness' is also not a useful goal because, like 
'window fairness', it makes the comparison in inappropriate units - if 
you try to say what is 'fair' at each instant without taking account of 
congestion over time, you'll get a much worse outcome (see the first 
argument in section 2.1.1 of the reference at {Note 1}).

I recall that you (if my memory serves me correctly) dismissed that 
argument in section 2.1.1  because you couldn't see how to calculate the 
correct shares at any one time. You'd need to read the references for 
that. But even if you can't work out the perfect solution, doesn't mean 
you cannot see that, for flows with different volumes, highly unequal 
flow rates give much shorter flow completion times than equal flow 
rates. The idea is to start you questioning your belief in equal flow rates.

Nonetheless, compared to 'window fairness', 'rate fairness' is a more 
useful /baseline/ on which to build other more useful behaviours, 
because at least it avoids starvation.

So, please understand that the reason I prefer an RTT-independent 
congestion control is not because I believe 'rate fairness' is a 
desirable goal. In the presence of very shallow queues, 'rate-fairness' 
is just less undesirable than 'window fairness', because 'rate-fairness' 
avoids starvation.

> 	[SM] Then we miscommunicated, my point is, that if you fix the dual queue couplked AQM than you can tackle the worthy goal of less RTT dependence at your own sweet time.
[BB] You never say what you mean by 'fix the dualQ coupled AQM'. I am 
not going to know what you mean because, from my perspective, the AQM 
does what it was designed to do very well. It doesn't determine per-flow 
shares of capacity. It merely emits congestion signals. Per-flow shares 
of capacity are determined by each sender's response to those signals.

You clearly want the AQM to do something different, but you don't say 
what. I assume you don't mean just "replace it with FQ" {Note 2}.

> 	[SM] And then opted to keep pushing for internet-wide roll-out of L4S with this issue unsolved, color me not impressed.
[BB] We were pushing for Internet-wide roll-out of the /network/ part of 
L4S. We were from network companies and we thought others would do the 
end-system parts - hence the Prague requirements. We thought DCTCP was a 
sufficient starting point for others to build on. Once I realized that 
wasn't happening (except in Google) without a reference implementation, 
I left CableLabs and got funding to hire folks to work on TCP Prague 
with me.

> 	[SM] To repeat myself, this all supports the thesis that making RTT-independence the mechanism by which L4S equitably shares with standards-compliant traffic at the current state is premature.
[BB] It is not premature to develop an RTT-independence solution that we 
have shown to work in simulations, and that we know has been implemented 
successfully in other congestion controllers in the past.

> I very much would like to see the dual queue coupled AQM fixed, and once RTT independence (for all users of the L4S queue) is functional then the AQM can be changed again.
[BB] If you mean that we should bias the signals emitted by the DualQ to 
counterbalance the incorrect response of RTT-dependent senders, that 
would create a nightmare "incremental undeployment" problem. Then, 
no-one would ever be able to fix any of these broken (RTT-dependent) 
congestion controllers because, during the transition, the 
counterbalances added to AQMs would remain in the network and cause 
RTT-unfairness in the opposite direction. So reverse engineering the AQM 
to counter broken senders would freeze both into the Internet for ever.

So, please accept that is not the way it's going to happen. The DualQ 
Coupled AQM does not need to be fixed because it isn't broken - it is 
very much working as intended.

Network deployments take years and remain for years. End system 
deployments can continuously evolve.

My point at the start was that RTT-dependence is an aspect of the way 
DCTCP senders respond to the signals emitted by queues. So the correct 
place to fix RTT-dependence is in those senders.

> 	[SM] Let me be blunt, without the deserved push-back on the lists, "peaceful coexistence with single queue RFC3168 AQMs" would not have been tackled, the gist of the presentatuins so far has alway…
[BB] The concerns about single-queue RFC3168 AQMs did indeed succeed in 
redirecting my attention from RTT-independence, which I would otherwise 
still be focusing on. Olivier at least is still working on that.

I am concerned that still no-one has yet found a single-queue RFC3168 
AQM deployed on the public Internet, or a network operator who wants 
one. No doubt there are some out there, but I have no idea how large or 
small the problem really is.

It would be preferable if you would at least gather data to help 
prioritize this work, by quantifying the prevalence and impact of these 
problems. But I can't stop you just repeating known problems in email 
after email with no data to support your invective.

> 	[SM] Well, the something called "dual queue" AQM certainly needs to at least behave as if it had two queues in my book.
[BB] It does. The L queue isolates its traffic from the latency of the 
other, and they each emit different but coupled levels of congestion 
signals.

> L4S versus SCE is red herring here,
[BB] What I said was not pitting L4S against SCE. Quite the opposite.

I said that an SCE transport also needs to be RTT-independent, for the 
cases where it runs over a single queue SCE AQM (e.g. CNQ). I do not see 
any work being done on RTT-independence in SCE transports at all. And I 
do not see you criticising anyone for not having done this.

> my current issue is that the current implementation of L4S's reference AQM does not work as it should and as a normal reader would understand from reading the dualQ ID, IMHO that needs to be fixed.
[BB] Which text in the aqm-dualq-coupled draft gives this wrong 
impression? It clearly says:

       For the
       public Internet an L4S transport has to comply with the
       requirements in Section 4 of [I-D.ietf-tsvwg-ecn-l4s-id  <https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10#ref-I-D.ietf-tsvwg-ecn-l4s-id>] (aka.
       the 'Prague L4S requirements').

That clearly does not promise that the dualQ will behave as intended for 
a transport in the L queue that is RTT-dependent, i.e. that does not 
satisfy the stated requirements for that queue.

Whatever, I notice that Appendix C assumes an RTT-dependent L4S 
transport, which will need to be changed once we have empirical results 
for RTT-independent transports. And that assumption needs to be called 
out in the interim. I've made a note-to-self to do both these.

> I do not believe that the not yet solved RTT-independence goal is the best approach to solve the issue, but you are free to spend your time as you please.
It would help if you argued against the fairly straightforward reasoning 
(above, starting "So why is it enough for..."), rather than just 
generally saying you don't believe it.

> 	[SM] I have no beef with your understanding of the issues, my complaint is that you do not act upon long-standing well-documented show-stopper issues. Not every documented bug can be dressed as a…
[BB] When you consider something to be an important bug, but I argue it 
is an important feature, the proper approach is to try to understand why 
something you thought was an accepted truth is being questioned.

Instead, what you've typed is effectively an accusation that I am a 
charlatan with evil intent. This frequent descent into personal slurs 
does not move anything forwards.

Think of all the material about FQ-CoDel. Is that hype, or is it 
enthusiasm about good results? To a sceptical reader who has not 
repeated the results themselves, I'm sure it seems like extreme hype.

I would ask you to presume the L4S results are all genuine until proven 
otherwise (of course, you should also hold a degree of scepticism in 
reserve). Then perhaps you would read enthusiasm as enthusiasm, rather 
than as hype.

(Nonetheless, I am making an editorial pass through the drafts damping 
down the enthusiasm/hype.)

Regards

Bob

{Note 1}: These arguments for allowing rate inequality are developed 
properly in "Per-Flow Scheduling and the End-to-End Argument 
<http://bobbriscoe.net/pubs.html#FQ_v_E2E>". I wrote it this summer 
specially for readers who believe in per-flow scheduling and who missed 
the debates in the research and IETF communities about this over the 
past dozen years.

{Note 2} I would never introduce per-flow rate controls into the network 
anyway. Because I do not want to be to blame for a future where 
high-rate applications (AR, VR, holographic, light-field, remote 
viewing, etc) have to ask permission from the network, just in order to 
maintain a certain amount of throughput in the presence of FQ with 
varying numbers of competing flows (when there is plenty of bandwidth so 
that a less than equal share is still nowhere near starvation). I do not 
think proponents of per-flow scheduling want such a politically 
distasteful shift of control towards network operators either, so I 
don't know why they choose not to see that they are pushing the Internet 
in this direction.

The main reason the early L4S proponents sunk years into developing the 
dualQ coupled AQM was to provide a way to do low latency without FQ.

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 30 Dec 2019 23:10:26 +0100

Dear Bob,

summary to save the rest of the list some time:

I argued that the current implementation of the dual queue coupled AQM (DQCAQM) fails to meet the required isolation quality between L4S two queues to to merit rolling it out of the lab and into the real internet.

The symptom is that in realistic scenarios of high bandwidth flows with short RTTs L4S traffic demonstrably pummels/dominates the non-L4S traffics aggregate throughput. That fact is not under dispute as far as I can tell.

The L4S proponents argue, that this is working as designed, as this is a side-effect of their under-explained choices of acceptable standing queue "targets" for the two queues (1ms for the L4S-queue and 15ms for the non-L4S queue), which for short path RTTs in the limit creates an RTT imbalance of 1:15. (I have also argued that 15ms is the wrong value in theoretical grounds and so far this has not been disproven with facts/data). 

The L4S team seems to accept though that this situation is sub-optimal and argues that by increasing the RTT-independence of all flows admitted to the L4S queue the problem can be ameliorated. 

While I am all for increased RTT-independence, I fail to see how this can be considered a general solution for the problem that the DQCAQM simply fails to do the one thing exists to do. 

Even if this would work for TCP Prague (and the increased RTT independence is not even deomstrated reliably for TCP Prague so far) that still will only solve this issue for flows with that congestion controller, this exercise needs to be repeated for all transports aimed for the L4S queue. 

If on the other hand the DQCAQM simply is fixed to deliver robust and reliable equitable bandwidth sharing between the two queues* then TCP Prague and all other transports intended for the L4S queue can take their sweet tome to develop more RTT-independence.

I am no engineer, but I can easily recognize which of the two options promises robust isolation and which just promises an on-going whack-a-mole.

*) Say under congestion no class gets more than double the bandwidth of the other, ideally obviously exactly 50%

more below in-line, mostly for Bob.

> On Dec 30, 2019, at 00:25, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
>>>>> On Nov 20, 2019, at 13:40, De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
>>>>> 
>>>>>  wrote:
>>>>> 
>>>>> 
> [BB] There is no convincing argument yet. This is an opening sentence, from which neither I nor you can make any convincing argument. Please work through the whole argument below...

	[SM] I disagree, the argument is, since " RTT-dependence is caused by the end-system[s]" and needs to be addressed in the end-systems  we can not assume higher RTT-independence as a means to ameliorate deficiencies in L4S's chosen class separator, the DQCAQM, unless you are willing to post-pone the L4S experiment/roll-out until all end-points are changes to have a higher RTT-independence. Also what is missing is actual data demonstrating, that this will bring the worst case sharing of the DQCAQM into the acceptable range.

> [BB] Sorry, I screwed up that table. I decided to use a 5ms queue instead of a 15ms queue at the last minute (in order to use the worst case for my argument), however I only changed the totals, and omitted to change the queue itself. Here's the corrected table:
> 
> 
> 
> Base RTT/ms
> Queue/ms
> Total RTT/ms
> Close Scalable flow:	2.5
> 0.5
> 3
> All the arguments that followed remain the same though, because they relied on the totals not the constituent parts of the totals.

	[SM] Thanks for fixing the numbers, but as I already conceded below "but your point still stands", meaning I well understood the gist of your argument.

> [BB] 'Window fairness' is not a useful property, it's just a property of a large class of congestion controls. The word 'fairness' sounds intrinsically like people should think it is desirable. So I try to avoid using it, and if I do, I try to always put it in quotes.

	[SM] Why bother? I have read a few of your opinion pieces arguing against flow fairness, and hence understand where you come from. I tend to think that rough equality is the best of all achievable bad splits, I happily concede it to be rarely "optimal". But as you have realized yourself, for any arbitrary hop on a path there typically is insufficient information available to actually optimize the bandwidth distribution between all active flows in any meaningful way; out of the two options, "anything goes" and "try to equalize by some relative objective measure",you seem to prefer the former while I prefer the latter. 

> [BB] Please now revisit the arguments below without the distraction of the typos.
>> 
> [BB] Oh dear. What is so important to you about such precise rate equality?

	[SM] Please remember that this is not a pure discussion about the desirability of higher RTT-independence (I am all for it), but rather about the fact that it is a red herring to bring up RTT-independence as explanation for the fact that the current implementation of dual queue coupled AQM does not seem to meet what both the L4S-arch and the dualQ I.D.'s "promise" to the reader. The only fact that both are not wrong is because they hedge their bets by attaching specific conditions (and fail to mention that dualQ will make sure the required RTT equality will require shorter true RTTs for the non-L4S queue). Hence the text might be technically correct, but it is also misleading.
	So my point is that you need to fix the well-documented deficiencies in the dual queue coupled AQM. AND here I, and I assume most casual readers of the L4S I.D.s will expect equitable sharing between the two traffic classes. I am sure that most reviewers will not see that as strictly as I do, and will accept a fudge factor of say 1:2, but 1:8 and worse is simply an indication that either the implementation or the design are not up for the intended purpose...

> Please be prepared to set aside your preconceptions about rate equality.

	[SM] No, I will not, as far as I understand the rationale behind the dual queue coupled AQM is to allow exposing the internet to the more "aggressive"/nimble 1/p type congestion controllers, and to be fit for roll-out the current guarantees that dual queue AQM will give, namely "not completely starving" non L4S-traffic is simply too weak. 
	The discussion I am leading here is not about what L4S's AQM does inside each of the two traffic classes, but purely between them, so I am NOT talking about flow-fairness at all, but inter-class-fairness. And as I already indicated, even equitable sharing between classes will give the initially rare L4S traffic already a considerable boost over packets in the non-L4S queue, which I consider undesirable, but this is not a discussion about my preferences, but about whether the L4S experiment is sufficiently cautious and safe enough for the rest of the internet to being exposed to L4S traffic. 

> Please re-read the reference at {Note 1}. No-one has ever been able to justify a goal of equal flow-rates, except for starvation avoidance (when rough rate equality is sufficient). 

	[SM] I read that note, but it fully failed to convince me. You seem to argue that equitable sharing prohibits more optimal traffic splits, and I fully agree, but at the same time it also prohibits more pessimal traffic splits, and given that the congested hop typically has no reasonable means to assign "valence" to the flows, I prefer predictable "good enough" sharing over "anything goes". And with that let's put that discussion to rest, given your dogmatic position regarding this topic, I am sure that I will not be able to convince you, and equally you will not be able to convince me of your position, as I have read quite a lot of your prose on that topic already and am still not convinced that "potentially optimal, but also potentially pessimal" is preferable to "mostly good enough and rarely really bad".

> Congestion levels have to be high for starvation to be a concern. So, the lower congestion is, the less important rate equality becomes.
> 
> There's a lot more to this, so please read and try really hard to understand /all/ the points in that reference at {Note 1}.
> 
> For instance 'rate fairness' is also not a useful goal because, like 'window fairness', it makes the comparison in inappropriate units - if you try to say what is 'fair' at each instant without taking account of congestion over time, you'll get a much worse outcome (see the first argument in section 2.1.1 of the reference at {Note 1}).
> 
> I recall that you (if my memory serves me correctly) dismissed that argument in section 2.1.1  because you couldn't see how to calculate the correct shares at any one time. You'd need to read the references for that. But even if you can't work out the perfect solution, doesn't mean you cannot see that, for flows with different volumes, highly unequal flow rates give much shorter flow completion times than equal flow rates. The idea is to start you questioning your belief in equal flow rates.

	[SM] Your leap of faith is that I follow you in accepting "completion time" as the relevant measure, which I do not. BTW, the intermediate hop really has little way to figure out whether a given flow is (close to being) finished or not or whether a sparse flow switches into bulk capacity-seeking mode later. In the light of all that uncertainty I still consider the outcome in b) quite reasonable and good enough. But I really disliked reading that editorial and will stop discussing it further.

> 
> Nonetheless, compared to 'window fairness', 'rate fairness' is a more useful /baseline/ on which to build other more useful behaviours, because at least it avoids starvation. 
> 
> So, please understand that the reason I prefer an RTT-independent congestion control is not because I believe 'rate fairness' is a desirable goal. In the presence of very shallow queues, 'rate-fairness' is just less undesirable than 'window fairness', because 'rate-fairness' avoids starvation. 

	[SM] Bob, while I agree with very little in the above, this is orthogonal to the question that I am really concerned about, so I will let this pass, as it only distracts from the fact that I am not claiming the dual queue coupled AQM is broken because it does not do flow-queueing, to repeat myself, it is BROKEN, because it fails to properly ISOLATE the two traffic types you classify all packets into. So the property I see lacking is class-fairness not flow-fairness.

>> 	[SM] At this point I want to see real numbers from empirical testing, thanks for the references though.

	[SM] And I still do.

> [BB] You never say what you mean by 'fix the dualQ coupled AQM'.

	[SM] I did not have the impression that I left that in doubt, but I am happy to state this explicitly. Under congestion the dual queue coupled AQM needs to strictly maintain inter-class equality just as a casual reader expects it to do. That is, under load and independent of the number os flows and their RTT make-up egress rate of the AQM needs to be equal on a sufficiently small time-scale.

> I am not going to know what you mean because, from my perspective, the AQM does what it was designed to do very well.

	[SM] That was my suspicion as well, that we are not simply talking about an implementation bug, but rather about a deep-rooted design decision behind the dual queue coupled AQM. But when bringing this up with Koen, he seemed to argue that he also is concerned about class inequalities (say at extremely short RTTs) so I am at a loss, is the failure to share equitable between the classes considered as "working as intended" or considered something that should be fixed?

> It doesn't determine per-flow shares of capacity.

	[SM] We are talking per-class sharing here, please do not confuse these two they are conceptually quite different, and I am NOT, to repeat myself, blubbering on about the fact that there is no flow-queueing inside any of the two queues. Your AQM, your rules, as long as you do not completely pummel my non-L4S thoughput I am willing to live and let live.

> It merely emits congestion signals. Per-flow shares of capacity are determined by each sender's response to those signals. 
> 
> You clearly want the AQM to do something different, but you don't say what. I assume you don't mean just "replace it with FQ" {Note 2}.

	[SM] No, Bob I do not propose that. I have wielded the term "equitable sharing" quite a lot, and at least in my humble opinion it was always clear that this is about sharing between the two queues. I am not trying to change your project to do flow-fair queueing inside of each of the two queues at all; I am trying to save the rest of the internet (including my own access link) from the negative side-effects I expect from your experiment. 

> [BB] We were pushing for Internet-wide roll-out of the /network/ part of L4S.

	[SM] Which contains the arguably not-working-as-it-should dual queue coupled AQM, exactly the right tome to voice concerns and objections, IMHO.

> We were from network companies and we thought others would do the end-system parts - hence the Prague requirements. We thought DCTCP was a sufficient starting point for others to build on. Once I realized that wasn't happening (except in Google) without a reference implementation, I left CableLabs and got funding to hire folks to work on TCP Prague with me.

	[SM] So you really set out to spend the ECT(1) codepoint on the intuition that there potentially might be users interested in using such a system initially? 

> [BB] It is not premature to develop an RTT-independence solution that we have shown to work in simulations, and that we know has been implemented successfully in other congestion controllers in the past.

	[SM] Bob, in the context of dual queue coupled AQ's short-comings, RTT-independence is a red herring. Even if you achieve RTT-independence for TCP-Prague, unless you enforce to only allow TCP-Prague into the L4S-queue (which would also allow to get rid of the abuse of ETC(1) as leaky classifier code point) you are not fixing the generic isolation failure of the dual queue coupled AQM, so hurrah for TCP Prague gaining RTT-independence in the future but that does not remove my objection or your AQM's issue.

> [BB] If you mean that we should bias the signals emitted by the DualQ to counterbalance the incorrect response of RTT-dependent senders, that would create a nightmare "incremental undeployment" problem.

	[SM] No, I mean that if your current design does not work at rather normal (given the closeness of CDNs to the edge) short RTT you should be given the chance to redesign your AQM. If you insist upon the AQM sharing to be driven by RTT dependent mechanisms, the onus is on you to remedy this.

> Then, no-one would ever be able to fix any of these broken     (RTT-dependent) congestion controllers because, during the transition, the counterbalances added to AQMs would remain in the network and cause RTT-unfairness in the opposite direction. So reverse engineering the AQM to counter broken senders would freeze both into the Internet for ever.

	[SM] Realistically, what you call "broken senders" is the reality in the internet, the exact reality your new system be better compatible with to merit roll-out in the first place.

> 
> So, please accept that is not the way it's going to happen. The DualQ Coupled AQM does not need to be fixed because it isn't broken - it is very much working as intended. 

	[SM] So I have you on record, that the equitable sharing failure at short RTTs is a property that you designed into the AQM on purpose? Again that seems to contradict my interpretation of Koen's words on this matter, but I am easily confused anyway.

> Network deployments take years and remain for years. End system deployments can continuously evolve. 
> 
> My point at the start was that RTT-dependence is an aspect of the way DCTCP senders respond to the signals emitted by queues. So the correct place to fix RTT-dependence is in those senders. 

	[SM] And I fully agree. Except that I draw the obvious logical conclusion, that hence increasing RTT-independence can NOT be the solution to the unequal between-class sharing of the dual queue coupled AQM. That is my point for some time now, and a rather explicit point.
 
> [BB] The concerns about single-queue RFC3168 AQMs did indeed succeed in redirecting my attention from RTT-independence, which I would otherwise still be focusing on. Olivier at least is still working on that. 

	[SM] Look, all of these issues come from a list of important issues that the L4S project itself created :

Requirements

L4S-ECN Packet Identification: ECT(1) 

Accurate ECN TCP feedback

Reno-friendly on loss

Reno-friendly if Classic ECN bottleneck 

Reduce RTT dependence

Scale down to fractional window

Detecting loss in units of time

I can understand if you prefer working on some of them more than on others, but complaining that woking to make sure your design meets your own REQUIREMENTS seems odd. Or I am misreading your words and all you are saying is that you are dedicating time to this?

> 
> I am concerned that still no-one has yet found a single-queue RFC3168 AQM deployed on the public Internet, or a network operator who wants one. No doubt there are some out there, but I have no idea how large or small the problem really is.

	[SM] IMHO the prevalence of single queue RFC3168 AQM is irrelevant, this is a point of contention that was correctly identified in the TCP Prague requirements and needs a proper solution instead of trying to argue it away. 

> 
> It would be preferable if you would at least gather data to help prioritize this work, by quantifying the prevalence and impact of these problems. But I can't stop you just repeating known problems in email after email with no data to support your invective.

	[SM] @Chairs, I do not want to appear to touchy, butI take offense in this, as long as Bob repeats the same points over and over again (instead of shutting me up with data), I do not seem him getting the right to complain that I also repeat my position instead of doing his research for him.

> [BB] It does. The L queue isolates its traffic from the latency of the other, and they each emit different but coupled levels of congestion signals. 

	[SM] The problem I see is that fine time scales Bandwidth and latency are inter-twined and pretending that they are orthogonal does not really help (at any point in time typical links will only allow to transmit 1 packet that than hogs the link for its transmission time).

> I said that an SCE transport also needs to be RTT-independent, for the cases where it runs over a single queue SCE AQM (e.g. CNQ). I do not see any work being done on RTT-independence in SCE transports at all. And I do not see you criticising anyone for not having done this. 

	[SM] The current status is that L4S will pummel standards-compliant traffic in this condition, while SCE will yield too generously, the latter restricts the fall-out of too high RTT-dependence to the SCE flows themselves, the former will simply bully its way through. Note how the SCE way is both safe for roll-out and carries an in-built incentive to address this issue, while the L4S approach is not safe (from the viewpoint of the existing non-L4S internet) and carries no real incentive to fix this, after all if I make my L4S-endpoint sender less RTT independent at short RTTs I can get an advantage over both non-L4S as well as TCP Prague traffic. 

> [BB] Which text in the aqm-dualq-coupled draft gives this wrong impression? It clearly says:
>       For the
>       requirements in Section 4 of [
> I-D.ietf-tsvwg-ecn-l4s-id
> ] (aka.

	[SM] I have cited these before, but here again:
"This specification defines `DualQ Coupled Active Queue Management
   (AQM)', which enables these Scalable congestion controls to safely
   co-exist with Classic Internet traffic."
"Analytical study and implementation testing of the Coupled AQM have
   shown that Scalable and Classic flows competing under similar
   conditions run at roughly the same rate."
"without compromising the performance of
   the Classic traffic"
"the Coupled AQM that addresses throughput equivalence between
      Classic (e.g.  Reno, Cubic) flows and L4S flows (that satisfy the
      Prague L4S requirements)."
"For safe co-existence, under stationary conditions, a Scalable flow
   has to run at roughly the same rate as a Reno TCP flow (all other
   factors being equal)."
"Nonetheless, coupled marking ensures that giving priority to L4S
   traffic still leaves the right amount of spare scheduling time for
   Classic flows to each get equivalent throughput to DCTCP flows (all
   other factors such as RTT being equal)."

None of this indicates that at short RTTs L4S traffic will dominate/pummel non-L4S traffic at short RTT paths, and nothing indicates that this is made worse by a severely under-explained set of latency targets for the two queues. Is that clear enough?

	[SM] And I note that not even TCP Prague seems to comply with all of these L4S requirements yet... 
	Look, in my discipline the onus to demonstrate that something new and wonderful works and has no/little/acceptable side-effects is on the person proposing the new thing. So with that background I very much like to see a working-in-the-lab-as-designed-and-as-considered-safe version first before even contemplating inflicting this on the wider internet. As it stands the dual queue coupled AQM is willing to give both a latency and a bandwidth advantage to any flow willing to set ECT(1), I am quite troubled that I seem to be the only person that sees this as an obvious unsafe design...

> 
> 
> That clearly does not promise that the dualQ will behave as intended for a transport in the L queue that is RTT-dependent, i.e. that does not satisfy the stated requirements for that queue. 

	[SM] Which IMHO is another problem, thanks for highlighting this. But also if the class-isolation is fixed that will also solve this issue.

> 
> Whatever, I notice that Appendix C assumes an RTT-dependent L4S transport, which will need to be changed once we have empirical results for RTT-independent transports. And that assumption needs to be called out in the interim. I've made a note-to-self to do both these.
> 
> It would help if you argued against the fairly straightforward reasoning (above, starting "So why is it enough for..."), rather than just generally saying you don't believe it. 

	[SM] Well, I do not buy your "reasoning" what I see is you trying to shut me up instead of engaging in a discussions of the points I bring and I do not appreciate that.

> [BB] When you consider something to be an important bug, but I argue it is an important feature, the proper approach is to try to understand why something you thought was an accepted truth is being questioned.

	[SM] Bob, please re-read my arguments under the light that I am in no way trying to make you do flow-fair queueing inside of your two queues. All I want is that the network side of L4S is robust and reliable and does not harm the rest of the internet (well, my emphasis is on the latter part, as I have severe doubts that L4S will in reality work as intended anyway). 

> 
> Instead, what you've typed is effectively an accusation that I am a charlatan with evil intent.

	[SM] I fail to see such an accusation in my mails could you please point out a specific instance, so I can try to avoid this in the future.

> This frequent descent into personal slurs does not move anything forwards.

	[SM] It seems I offended you, since that was not my intention, I offer my apology for all phrases that you consider offensive. But that does not one iota affect my criticism.

> 
> Think of all the material about FQ-CoDel. Is that hype, or is it enthusiasm about good results? To a sceptical reader who has not repeated the results themselves, I'm sure it seems like extreme hype. 

	[SM] You are really comparing the hype content of a single 25 page RFC versus a triplet of 35 (https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-04), 49 (https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10), and 42 page monsters (https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08), really?

> 
> I would ask you to presume the L4S results are all genuine until proven otherwise (of course, you should also hold a degree of scepticism in reserve). Then perhaps you would read enthusiasm as enthusiasm, rather than as hype. 
> 
> (Nonetheless, I am making an editorial pass through the drafts damping down the enthusiasm/hype.)

	[SM] Thanks.

> 
> Regards
> 
> 
> 
> Bob
> 
> {Note 1}: These arguments for allowing rate inequality are developed properly in "Per-Flow Scheduling and the End-to-End Argument". I wrote it this summer specially for readers who believe in per-flow scheduling and who missed the debates in the research and IETF communities about this over the past dozen years.
> 
> {Note 2} I would never introduce per-flow rate controls into the network anyway.

	[SM] I know, but luckily that is not at issue here, we are merely talking about per-class rate controls...

> Because I do not want to be to blame for a future where high-rate applications (AR, VR, holographic, light-field, remote viewing, etc) have to ask permission from the network, just in order to maintain a certain amount of throughput in the presence of FQ with varying numbers of competing flows (when there is plenty of bandwidth so that a less than equal share is still nowhere near starvation).
> I do not think proponents of per-flow scheduling want such a politically distasteful shift of control towards network operators either, so I don't know why they choose not to see that they are pushing the Internet in this direction.
> 
> The main reason the early L4S proponents sunk years into developing the dualQ coupled AQM was to provide a way to do low latency without FQ.

	[SM] It still remains to be seen, whether they succeeded though.

Best Regards
	Sebastian

>>>>> From: Ingemar Johansson S 
>>>>> 
>>>>> <[email-masked]>
>>>>> 
>>>>>  
>>>>> From: Kyle Rose 
>>>>> <[email-masked]>
>>> Bob Briscoe                               
>>> http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 31 Dec 2019 16:55:23 +0000

Roland,

I just noticed that Koen's response (which I agreed with) snipped some 
of your responses that I ought to have addressed.
For this one, I've forked a new thread...

On 22/11/2019 02:32, Roland Bless wrote:
> Hi Bob,
>
> see inline.
>
> On 21.11.19 at 15:44 Bob Briscoe wrote:
>> On 21/11/2019 09:32, Roland Bless wrote:
>>> On 21.11.19 at 19:34 Bob Briscoe wrote:
>>>> On 20/11/2019 21:22, Roland Bless wrote:
> I find it architecturally cleaner to have an additional ECN codepoint
> used for indicating "SCE" rather than (ab)using it as classifier for the
> dualQ AQM.

The idea that it is abuse to use ECN as a classifier has never been 
justified, only asserted.

The actual position is pretty much the opposite. Below I explain why L4S 
follows the original way the architecture of the ECN field was defined. 
Whereas the way SCE uses the ECN field for 2-level marking is counter to 
the original ECN architecture.

The ECN field is unlike any other field in the IP header, because it 
conveys a value from the sender down the layers to L3 or L2 and it 
conveys a 'return' value from any node in the network back up the stack 
to the receiving endpoint.

Originally, in RFC3168:

  * for the sender->network interface, the ECN field was defined as a
    way for the sender to classify a packet between different code paths
    in AQM algorithms (drop for 00 vs. marking for the other 3
    codepoints). Certainly, it wasn't /envisaged/ that it would be used
    to classify between different queues. But /architecturally/ there is
    no difference.
  * for the network->receiver interface, the two semantically identical
    ECT(X)->CE transitions were defined as the only way for the network
    to signal congestion.

The two original standards track RFCs that originally defined ECN 
tunnelling [RFC3168] & [RFC4301] were both incompatible with 2-level 
marking like SCE. They required the decapsulation at the end of a tunnel 
to forward the inner header's ECN field unless the outer was CE. So they 
would strip any SCE signal applied to the outer header.

Around 2010, when I was rationalizing these two diverging tunnel 
definitions into one [RFC6040] (also standards track), I noticed that 
precluding 2-level marking was unnecessarily restrictive. So we defined 
the way ECN propagates to the forwarded header from the outer so that 
future tunnels would support either meaning of ECT(1): either equivalent 
to ECT(0) or a weaker congestion signal than CE, like SCE.

However, the original RFC3168 ECN architecture (that is incompatible 
with SCE) remains and will remain more widely supported in the Internet, 
probably for decades. Indeed, even today, the RFCs to make that change 
to all encapsulations (not just IP-in-IP tunnels) are not past the WG 
stage yet.

Tunnels and encapsulations are everywhere. Jonathan M says it doesn't 
matter if the SCE signal gets black-holed in tunnels and encapsulations. 
Eventually new ones compatible with the SCE architecture will be 
deployed everywhere (which will take decades). But it does matter - 
because SCE wants to claim the last available ECN codepoint, which L4S 
has already developed to give consistently low latency - within the 
original ECN architecture.

Regards

Bob

> Roland

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 31 Dec 2019 19:42:29 +0200

> On 31 Dec, 2019, at 6:55 pm, Bob Briscoe <[email-masked]> wrote:
> 
> However, the original RFC3168 ECN architecture (that is incompatible with SCE) remains and will remain more widely supported in the Internet, probably for decades. Indeed, even today, the RFCs to make that change to all encapsulations (not just IP-in-IP tunnels) are not past the WG stage yet.
> 
> Tunnels and encapsulations are everywhere. Jonathan M says it doesn't matter if the SCE signal gets black-holed in tunnels and encapsulations.

It's not that it doesn't matter at all - erasing the SCE signal would degrade the behaviour to RFC-3168 standard.  But that's an acceptable fallback position which allows for incremental deployment, even before all the tunnels are upgraded to the new standard.

Thus the existing RFC-3168 compliant architecture is *not* incompatible with SCE as you claim.  So please don't claim that, when it isn't true.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 31 Dec 2019 23:11:20 +0000

Roland,

As I just said, I noticed that Koen's useful response snipped some of 
your responses about evolvability that I ought to have addressed.
Sorry, you'll need to reload state from November...

On 22/11/2019 02:32, Roland Bless wrote:
>> I'm not getting it....
>>>>> Yes, but as I also expressed my concerns w.r.t. the L4S codepoint
>>>>> earlier, at the cost of binding this to a quite fixed set of L4S
>>>>> behaviors and "burning" the last ECT codepoint. Personally, I like
>>>>> concepts with a little bit more potential to be useful for future
>>>>> development (evolvability) of congestion controls, e.g., BBRv2 and
>>>>> LoLa could also benefit from an SCE-like marking...
>>>>>
[BB] BBRv2 has not used SCE, but it has used L4S marking. And the 
authors of the LoLa draft have not used SCE but they have merged with 
the NQB draft, so that they can work with L4S as well. So perhaps your 
belief that SCE is "more evolvable" is just that - a belief.

>> This last sentence... please really spell it out what you could possibly
>> be meaning here. What has made you suddenly think this particular
>> marking scheme has got magical properties that bestow evolvability? I'm
>> serious, if you can't explain why you've said a sentence like this, it
>> implies you are under the spell of some cult or fad.
> Not sure what you are trying to indicate by your last sentence, but
> sure, I can elaborate a bit on my last sentence.
[BB] I just forked another thread 'ECN as a classifier' to address that 
assertion.
> The SCE markings can be used independently whether you have multiple
> queues or just a single queue.
[BB] This seems to be the opposite of reality. Perhaps you've used the 
word 'can' because you think it might be possible. But there's good 
reason to doubt that. As follows...

Packets from non-SCE and from SCE senders are indistinguishable by just 
looking at them. So, if there's a single queue, they have to be mixed 
together in it {Note 1}. But a single queue can only have one length. 
Any transports that don't understand SCE drive the single queue to a 
deeper operating point (defined by CE or drop). This overruns the SCE 
operating point, and causes SCE marking to approach 100% {Note 2}. Then 
those transports that do understand SCE will starve.

In contrast, L4S works in dual queues or multiple queues (where 'works' 
means it gives very low latency without losing utilization).

Also, there is considerable scope for evolvability with both:
* the dualQ is a framework that is intended to encompass different AQMs 
in future.
* and there is considerable scope for developing the ways FQ uses L4S.

{Note 1}: Unless L4 identifiers are used to separate out microflows, and 
identify low latency flows by their behaviour. But then it's not really 
a single queue, unless you make it look like a single queue, but still 
treat it like multiple queues, as in LFQ. However, schemes like LFQ are 
"single-queue" in name only. They still have all the downsides of 
multiple queues.

{Note 2}: Unless the SCE operating point is hardly any shallower than 
that for CE, in which case SCE gives hardly any improvement.

> If one leaves the coupling aside the
> algorithm for marking will probably not differ so much from what is
> proposed in L4S.
[BB] You still haven't explained (rather than asserted a belief) why the 
choice between SCE and L4S has anything to do with evolvability.

>>>> My whole purpose in solving the problem of deploying scalable CCs over
>>>> the Internet was to re-juvenate evolution (to widen the range of
>>>> applications that could be supported by different transport behaviours,
>>>> particularly for real-time with low latency and high throughput at the
>>>> same time). One of the main things that has stopped CCs evolving so far
>>>> is the need for friendliness with the Reno behaviour that was not
>>>> scaling over the years.
>>> FWIW, our delay-based approach has deployment problems at the
>>> other end of the spectrum: it gets suppressed by loss-based CCs...
>>> (and we don't want to sacrifice our low delay goal).
>>>
>>>> If SCE is primarily supported in FQ AQMs, that will constrain flows to
>>>> be capped at the rate that FQ gives them. How is that doing anything
>>> I was solely referring to the marking behavior of SCE, not a particular
>>> implementation based on FQ-AQMs.
>> This implies you believe all that silliness about a shallower SCE
>> threshold not starving an SCE flow in a single queue, because it makes
>> SCE flows not want to use the queue.
> I do not understand what you are saying here.
[BB] I've now explained this above (after "But there's good reason to 
doubt that...").

The silliness I'm referring to is the statement by Jonathan M in his 
tsvwg talk at IETF-106 that CNQ is backward compatible with RFC3168 
traffic, because its performance for SCE traffic when mixed with 3168 is 
bad enough that it discourages use of SCE alongside 3168.

>>>> other than massively constraining future evolution of CCs, especially
>>>> real-time ones? See Per-Flow Scheduling and the End-to-End Argument
>>>> <http://bobbriscoe.net/projects/latency/per-flow_tr.pdf>. I don't need
>>>> to tell you that the e2e argument is all about giving end systems the
>>>> power to innovate without permission.
>>>> Anyway, what are you imagining would stop CCs evolving alongside other
>>>> scalable CCs? In much the same way CCs have always evolved. With L4S you
>>>> have a clean slate that seems just like a FIFO with a shallow ECN-only
>>>> immediate AQM. And other flows are causing you hardly any delay and very
>>>> rarely any loss. Think of all the things you can do with that. Go
>>>> evolve, Roland!
>>> I already said that the Dual Queue AQM is nice, but comes with the
>>> problem due to its built-in dropping law. Other CCs may react differently
>>> and BBR was one example of newer CCs that did not work in the L4S queue
>>> without further adaptation.
>> The built in dropping law is for the old traffic ('pre-evolved' if you
>> like) that doesn't respond to anything else but loss. That's what
>> drop-based AQMs were for - to fool Reno etc into thinking they had hit
>> the buffers, so to speak.
> Yes, I understand that. But what happens if the classic traffic
> once vanishes and we only have low-delay congestion control and
> want to re-use the "second" queue for other purposes, e.g., using
> it for flows that do not use loss-based CC?
[BB] Let me repeat back the question, 'cos it seems rather odd, so 
perhaps I've misunderstood.

You want me to assume that the classic behaviour of repeatedly seeking 
out more capacity until loss occurs doesn't exist on the Internet any 
more. There's only low-delay CC, by which I think you mean CC based on 
L4S ECN (but I'm not sure that's your meaning).

Then, it sounds like everything would be working well. So why do you 
want to re-use the "second" queue for other purposes? Nothing is wasted 
if you don't use it. It has no capacity associated with it. it's just 
some lines of code sitting there in case packets appear from a classic CC.

>>>    > The key thing here is the wording of the Prague requirements
>>>> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-08#section-4>.
>>>> We have a session in the 'Prague' side-meeting tomorrow to review them
>>>> (and I encourage this on this list too).
>>>>
>>>> Later down the line, if the L4S experiment is successful, there will be
>>>> an opportunity to review these requirements if a standards track doc
>>>> replaces the experimental (it is easier to relax CC requirements than
>>>> tighten them). So, for the expt track, the requirements are designed to
>>>> protect competing flows from harm in a tighter way than you might find
>>>> in RFC5033 or similar.
>>>>
>>>>
>>>> Nonetheless, even the 1/p isn't tightly spec'd, quoting:
>>>>
>>>>      The inverse proportionality requirement above is worded
>>>>      as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility
>>>>      when defining these specifications.
>>> Yes, but something has to be implemented (in hardware) and therefore it
>>> is fixed for some time and it should be consistent along a path, so I
>>> don't see a viable path for evolvement of this coupling law...
>> Why does the coupling law need to evolve? It's a relationship between
>> something invariant (scalable) and the past (which is over, by
>> definition, so it's not going to change now).
> See above, probably we want to put something into queue which is
> using a non-loss-based congestion control and/or we need to change
> the (1/p,1/p^2) marking.
[BB] The queue doesn't induce loss, the CC does. The queue isn't 1/p or 
1/p^2 or whatever, the different CCs are. So, if a CC is using the C 
queue without inducing any loss (e.g. a delay-based CC might be able to 
keep the queue under the AQM's target delay), then the coupling won't 
couple any marking across to the L queue.

But I can't imagine why a delay-based CC that can keep the queue below 
the target of the AQM would want to classify itself into a queue 
designed for CCs that need to induce a decent queue (in order to 
maintain full utilization), given such queue-inducing traffic might 
start up at any time.

>> If you want to evolve, you select what's best for you - probably the
>> nice clean L4S ECN queue. I still don't get what sort of evolution you
>> can't do? Evolvability isn't about a researcher being able to do what
>> they'd like.
> I also don't understand it in this way, but investigating
> invariables and degrees of freedom prudently may enable or facilitate
> deployment of new stuff. If that new stuff cannot be deployed
> it never gets a chance of being weeded out later either.
[BB] If you have something specific in mind, please say it. This is all 
getting rather abstract.

This does make me want to question your notion of evolvability. Usually, 
when we make sure the Internet supports evolvability, we mean 
evolvability of new applications. We don't mean the ability for 
researchers to come up with different ways to solve problems that are 
already solved. There seems to be a hint of a complaint in your emails 
that L4S doesn't leave room for researchers to solve the same problem 
differently. That sort of evolvability is rather a luxury isn't it? If 
one approach solves an enduring problem with the Internet, it would be 
rather churlish to say "No, we can't use this solution, because it might 
preclude some ideas that might lead to other solutions to the same 
problem in the future, but we're not sure."

Having said that, I hope I (and Koen) have shown that L4S still leaves 
scope for complementary delay-based CCs, not to mention scope for 
different AQMs and for different FQ-based solutions.

Regards

Bob

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 1 Jan 2020 00:49:12 +0000

Jonathan,

> Thus the existing RFC-3168 compliant architecture is *not* incompatible with SCE as you claim.  So please don't claim that, when it isn't true.
You can say SCE is backward compatible with 3168 tunnels. But it is 
disingenuous to say it the other way round - that 3168 tunnels are 
compatible with SCE. Because many/most people will understand this to 
mean that 3168 tunnels are compatible with what SCE aims to achieve.

(An analogy would be to claim that a bare FIFO queue is compatible with 
ECN, which it isn't. Yes, ECN is backward compatible with FIFO queues, 
but not the other way round.)

When you say the 3168 architecture is compatible with SCE, you have 
quietly defined what SCE aims to achieve as "either SCE or 3168 
performance". This allows you to make a statement that only reveals the 
good part of a truth that is mostly bad. I have noticed you play with 
words like this a lot, presumably because you think that you might fool 
some casual observers.

To state the whole truth you ought to say, if a 3168-architecture tunnel 
contains the SCE bottleneck, until the tunnel is updated to support the 
SCE architecture SCE will always fall back to 3168 performance.

This adds up to 4 independent upgrades being needed before SCE can 
provide any benefit: sender, receiver, bottleneck AQM and potentially 
tunnel as well. Worse, tunnel processing is typically implemented in 
hardware. But you always describe SCE as a minor change.

Happy New Year

Bob