Subject: Re: [tsvwg] L4S and the detection of RFC3168 AQMs

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 9 Feb 2021 12:05:28 +0100

Martin, group,

AccECN is not tied to RFC3168 (and quite the opposite, it tries to be as
impartial to the specifics of when/why packets were marked in whichever
way).

Therefore the Packet-CE counter (you refer to it as PCE, while the draft
refers to the (full) counter as cep [from CE_packet] is incremented
regardless of which packet arrives with the CE mark (control, data,
retransmission, ...).

Similar for the Byte-CE counter (BCE, in the AccECN draft ceb from
CE_bytes). As only TCP payload bytes arriving with the CE mark would be
accounted, the behavior as expected in the discussion below is already
implicitly available when using AccECN.

There may be some ambiguity, as AccECN doesn't strictly require an
immediate ACK on a CE (only that the next packet after a received CE
mark has to convey the changed CE-related counters), but in the general
case, AccECN would allow and support such a detection scheme.

Best regards,
    Richard

Am 12.12.2020 um 00:22 schrieb Martin Duke:
> Excellent, thank you for the reminder. So the L4S sender could
> interleave some ECT(0) marked pure ACKs (or retransmissions of the last
> acknowledged byte) and hope that the PCE counter increases without
> corresponding increases in BCE.
>
> The AccECN spec might need to be updated to specify that these should be
> reported in PCE even though they are not 3168 compliant.
>
> Scheduling these probes might not exactly be trivial, but if they are
> temporally correlated with ECT(1)->CE marks, this would be highly
> suggestive, yes?
>
> On Fri, Dec 11, 2020 at 3:03 PM Black, David <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     In which case, RFC 8311 Section 4.3 allows experimental usage of ECN
>     with such packets (https://tools.ietf.org/html/rfc8311#section-4.3
>     <https://tools.ietf.org/html/rfc8311#section-4.3>).____
>
>     __ __
>
>     Thanks, --David____
>
>     __ __
>
>     [EXTERNAL EMAIL] ____
>
>     My understanding of 3168 is that only in-window data packets are
>     marked ECT(0). A zero-length segment is a equivalent to a pure ACK,
>     which is not marked.____
>
>     __ __
>
>     On Fri, Dec 11, 2020 at 12:09 PM C. M. Heard <[email-masked]
>     <mailto:[email-masked]>> wrote:____
>
>         On Fri, Dec 11, 2020 at 11:51 AM Martin Duke
>         <[email-masked] <mailto:[email-masked]>>
>         wrote:____
>
>             This falls under the "much easier to do in other transports"
>             category, where I could just send a PING or HEARTBEAT marked
>             ECT(0) to test the queue in mid-connection, without
>             affecting the latency of anything that matters. But in the
>             TCP case, I'm not sure how to resolve Bob's second objection
>             (running ECT(0) for a long time would be unacceptable).____
>
>         __ __
>
>         Could zero-length TCP segments be used instead of PING or
>         HEARTBEAT? ____
>
>         ____
>
>         Mike Heard ____
>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 9 Feb 2021 23:09:20 +0000

Richard,

First, thank you for reviving this thread, because it brought Alex's 
last posting (in Dec) to my attention. I'll respond to that separately 
(probably mañana, as it's getting late).

Pls see inline tagged [BB]...

> impartial to the specifics of when/why packets were marked in whichever

[BB] Also, in case people aren't aware: an L4S sender using TCP MUST 
negotiate the use of Accurate ECN TCP feedback with its peer. So I think 
there's no case where a transport would not provide feedback for Alex's 
idea.

> implicitly available when using AccECN.

[BB] ...only if the data receiver supports sending the AccECN TCP Option 
(optional) and the new TCP Option traverses the path successfully and 
the data sender supports reading the TCP Option (optional).

Nonetheless, if any of these three is not the case, it might still be 
possible to work out whether a CE marking was on a probe packet...

The ACK that carries the CE packet counter also carries an ACKno (and 
possibly SACK options). When a zero-sized probe is sent, it won't elicit 
an ACK on its own (TCP doesn't ACK pure ACKs). But if the probe is CE 
marked, when the data receiver ACKs subsequent data packet(s), the CE 
packet counter will include any CE marking on the probe. It seems that 
doesn't help, because the data sender cannot tell whether the CE mark 
was on the data packets or the probe. But if the counter ever increases 
by more than the number of data packets covered by the ACK, the probe 
must have been marked as well.

That sounds like having to wait quite a time for the possibility of all 
packets together being marked. But there might be a more deterministic 
way by use the same trick that keep alive probes use...

That is, send a zero-sized probe with SEG.SEQ = SND.NXT-1. Being out of 
window, each of these probes would immediately elicit a pure ACK from 
the receiver. This might cause problems alongside the other regular 
ACKs, because I think these might look like Dup ACKs (when the trick was 
invented for keepalives, this wasn't a problem 'cos there were no other 
data packets being ACK'd). However, it's possible the sender can work 
out that they are not Dup ACKs, given it knows it sent a probe.

(I'm not totally certain of my facts here - we'd need a TCP expert to 
confirm - Richard?)

Bob

> case, AccECN would allow and support such a detection scheme.

[BB]

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Martin Duke <[email-masked]>
Date: Wed, 10 Feb 2021 10:37:04 -0800

Well this all seems very promising!

Does anyone have the bandwidth to write something down? Should there be a
separate draft that articulates the design? Would this replace the current
Prague approach?

Martin

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 10 Feb 2021 23:54:51 +0000

Alex,
As I said yesterday, it was only through Richard reviving this thread 
that I saw your response, which I missed last Dec. Sorry.
See inline tagged [BB]

On 13/12/2020 21:32, [email-masked] wrote:
> Martin,Bob,
>   I'm glad you replied, Martin; firstly for your kind words, but also because Bob's reply below was in my spam folder, so I hadn't seen it. More inline [AB]
>
>
> On Friday, December 11, 2020, 8:11:23 PM GMT, Martin Duke <[email-masked]> wrote:
>
>
> Alex,
>
> Thanks for thinking outside the box like this -- it may be this sort of creativity that gets out of the box we're in.
>
>
> This falls under the "much easier to do in other transports" category, where I could just send a PING or HEARTBEAT marked ECT(0) to test the queue in mid-connection, without affecting the latency of anything that matters. But in the TCP case, I'm not sure how to resolve Bob's second objection (running ECT(0) for a long time would be unacceptable).
>
> On Wed, Dec 9, 2020 at 3:23 PM Bob Briscoe <[email-masked]> wrote:
>>    
>>    Alex,
>>
>> An interesting idea.
>>
>> You've given some pro points.
>> Here are a few con points:
>>
>> 1) Requires L4S FQ AQMs to actively remove Classic ECN support.
>> Many FQ AQMs already support Classic ECN. By this proposal, if they wanted to add L4S ECN support (which is pretty simple to do), they would have to remove Classic ECN support, dropping instead of marking ECT(0) packets. It is possible (likely?) that a number of FQ AQM implementers would vote with their feet and not remove Classic ECN support, even if the IETF published an RFC adopting your proposal. Then the certainty that your proposal is designed to create would be lost. Then we would be worse off in two directions: still no way to be certain about ECT(0) markings, while at the same time having reduced what little Classic ECN deployment there is.
> [AB] That would be a bad outcome. Perhaps the constraint could be modified to allow that FQ AQMs supporting L4S were allowed to mark ECT(0) following RFC3168, so long as they ONLY do so in buckets which have not seen ECT(1) for some period?

[BB] Good point. That would work. To keep this email unambiguous, I'll 
call that A2 (for Alex's idea #2).

It's just occurred to me that, even if an FQ AQM did not implement A2, 
the AQM would still be very unlikely to mark any ECT(0) packets if they 
were mixed into an L4S flow of ECT(1) packets, as long as...:
* the threshold for marking ECT(0) is deeper than for ECT(1);
* and particularly if ECT(1) marking is immediate, whereas ECT(0) 
marking needs persistent queuing
    (e.g. CoDel's requirement to exceed 'target' for the duration of 
'interval').
Reason: at any one time a queue can only have one length. So if the L4S 
flow's congestion control is keeping the queue mostly at the shallow 
operating point, it will hardly ever be able to persistently exceed a 
higher threshold.

I'm not saying your A2 idea wouldn't be worth implementing for stronger 
certainty. I'm just saying that the certainty that ECT(0) packets would 
not be marked by any L4S AQM would still be near-certainty, even if some 
FQ implementers didn't implement A2.

Some detailed thoughts on your A2 idea (skip if you want):

    Within one bucket, after an ECT(1) packet, it could disable ECT(0)
    for the remaining life of the bucket, rather than for a set duration.
    (This straightforward latch would be a little simpler than a timer.
    However, it would be more susceptible to an attack on Classic ECN
    support in which the attacker generates ECT(1) packets with random
    flow IDs and throws them at an FQ. Nonetheless, I'm sure attackers
    would rather spend their time mounting more harmful attacks than this.)

> I think this would also help with your 3) below.

[BB] Definitely.
[To save readers scrolling, my point (3) said Alex's original idea would 
be good for out-of-band testing, but only if it also applies to FQ 
(which it does now, given idea A2)].

> Maybe this would also help with the issues of unfairness with a) tunnels and b) L4S and normal flows hashing to the same bucket, since the queue in that bucket would build up to the point of dropping and then L4S would fall back to reno.

[BB] I'm afraid it won't help. Remember, a single queue can only have 
one length at a time.
If ECT(0) and ECT(1) flows mix in the same bucket (due to hash collision 
or a tunnel that looks like one flow), the ECT(0) flow will not see any 
marking at the shallower threshold, so it will drive the queue to the 
deeper threshold where ECT(0) packets are dropped or marked (depending 
if A2 is implemented or not).  Then, nearly all the ECT(1) packets will 
be marked, because they all arrive at a queue that exceeds the shallower 
threshold.

So the L4S CC will continually yield until it is starved.

Thinking further about idea A2:
Starting from today's situation, with only Classic flows, hash 
collisions and tunnels are detrimental to a small proportion of Classic 
flows. For this small proportion of flows, L4S and idea A2 would add to 
this detriment by disabling ECN as well.

Given long-running flows are relatively rare, two together are more 
rare, two with different ECTs will be even rarer, and two together in a 
tunnel or hash collision are even rarer still, I think that's a 
probability most people could live with. But others might say the sky is 
falling.

> It would need some additional storage per bucket, to track those which have recently seen ECT(1), but that could be a 2 bit countdown counter.

[BB] [Detail: Can be skipped]
I'm intrigued to know how to do this with only a 2-bit counter.
Perhaps whenever a queue sees ECT(1), it sets its counter to 3 then the 
whole scheduler decrements all non-zero counters every T seconds, so 
there will be 2T to 3T duration after the last occurrence of ECT(1) in 
any queue before that queue marks ECT(0) again.

more...

>> 2) For in-band Classic ECN AQM detection, the proposal requires L4S not to be used for a long time in order to detect whether L4S can be used.
>> L4S hosts are the ones that need to check for Classic ECN AQMs. But they don't send ECT(0). So to take advantage of your proposal, they would have to send ECT(1) packets then, if they see one CE marking, they would have to switch to sending ECT(0) packets, and actively send enough ECT(0) to be sure that /lack/ of CE marking was not just lack of congestion. Given CE marking is currently very rare, how long without CE marking do they have to send ECT(0) packets until they can assume it's an L4S AQM? They don't want to alternate ECT0 & ECT1, cos if it is an L4S bottleneck, that will lead to reordering.
>>
>> So, lack of CE marking would be ambiguous. Would it mean the bottleneck is L4S? Would it mean the capacity has increased and the flow needs to increase its rate more to find the limit? If there was a loss or losses, that would also be ambiguous. Are they congestion losses?
>>
>> All the time that the L4S source is sending ECT(0) packets, the flow is losing the benefit of the low delay it could have been getting if the original CE mark was emitted from an L4S queue. This is a similar problem to an algorithm that gives false positives (like the v2 Classic ECN AQM detection that Asad & I published). It sometimes detected an L4S AQM as Classic. So it was argued that this would probably prevent people from using it. The same would surely be true of an algorithm that requires L4S not to be used in order to detect whether L4S can be used.
>>
> [AB] I was indeed assuming that the transport could send ECT(0) probe whenever it wanted, and that probes would not contain data and so would not cause an ordering problem. I admit that I hadn't realised this might be difficult in tcp, although further in the thread Mike Heard and David Black seem to have identified a possible approach.
>
> I am not sure if, supposing a probe can be sent at any time,  this completely addresses your objection. "not using L4S" could be
> a) not using ECT(1) and therefore not getting the benefit of the L queue, or
> b) initially assuming that CE marks have the RFC3168 meaning, rather than the L4S meaning.
>
> If a way is found to send ECT(0) probes without causing reordering, presumably this addresses a)?  b) is more tricky.
>
> Let me take a step back:
>   One approach to getting this WG unstuck, would be to find some fallback algorithm which could easily be shown to be safe, even if it has other deficiencies from the point of view of those who want to deploy L4S. That would remove the most serious objection to starting the experiment; and then the WG might agree that since it would now be safe, the experiment could start, so long as there is a reasonable prospect of developing some improved fallback algorithm  . Once the experiment starts, then it would be easier to test and develop better detection algorithms along side one already known to be safe.
>
> You guys are in a better position to know what kind of deficiencies are tolerable to those who want to use L4S.
>
> The reason probing seems attractive is that, once you assume L4S doesn't mark ECT(0) , if probe packets are arranged such that each RFC3168 CE mark has the same, independent probability of hitting an ECT(0) nprobe packet, then the mathematics is very simple. At the risk of teaching everyone to suck eggs, if the proportion of probe packets is P then the expected number of CE marks until detection is 1/P. For safety, the number of CE marks until detection seems to me the right metric, since if there are no CE marks it's not a problem, and the higher frequency CE marks are, the more we care about detecting.And to calculate statistics about that, we don't need any more information than P, and any more assumptions other than that the frequency of CE marks is nonzero.
>
> Making P large  causes overhead in terms of the number of probe packets, which is one deficiency . On the other hand, even the most safety-minded might be placated with a sufficiently large P, and if also reduces the length of time you might need to wait to decide there is no RFC3168 AQM.

[BB] This all makes sense.

However, one additional consideration makes this much harder: some 
Classic AQMs reduce drop probability for smaller packets (for instance 
DOCSIS PIE [RFC8034; Sec 4.6], which is used as the Classic AQM in the 
DOCSIS dual queue alongside L4S). So the probability calculation would 
need to take into account the relative sizes on the wire of data packets 
and probes. Unfortunately an L4S sender doesn't know which sort of 
Classic AQM it might encounter, so I think it will have to assume the 
worst case.

> I had actually assumed that the L4S team's  intention was to start by assuming that there were no RFC3168 AQMs. This seems to be the behavior of the pseudocode in the paper.

[BB] That's not really a correct way to say it.
Our approach in the experiments in our report was: while a flow is 
starting to measure whether there's a Classic ECN AQM at the bottleneck 
(for perhaps 20 or 30 round trips), It can afford to /behave/ like L4S. 
That's not the same as assuming there are no 3168 AQMs. To justify this, 
as our report says, I checked the original paper onTCP Cubic, and 
convergence was still described as ’short’ when it took one or two 
hundred rounds. Anyway,from the user point of view, surely it's not 
significant if one flow outcompetes another for a few seconds after 
flow-start.

> If that is the case (and the WG accepts it) then there are no false positives at startup. The only reason then  to worry about the amount of time to wait without seeing ECT(0)->CE is if there is an RFC3168 AQM, which you previously detected, and now might no longer be the bottleneck. under those conditions it would seem reasonable to be conservative about switching back.

[BB] How and when did you "previously detect" an RFC3168 ECN AQM? I've 
read this para (and the whole thread) over and over, but I still can't 
work it out.

Whatever, as I say at (3) below, even if this approach cannot detect an 
RFC3168 ECN AQM quickly enough for in-band testing, it could still be 
useful for out-of-band testing.

still more...

>
> On the other hand if L4S transports should start in 'RFC3168  safe mode' and only switch to L4S if it decides that there is no RFC3168 AQM on the bottleneck, or if it is important to switch quickly back when the bottleneck changes, then you do need to make some assumptions about the frequency of marking  (of this stream). Unfortunately I don't have enough knowledge  of tp  (or tcp-prague) to devise such an assumption. Given some worst-case assumption of the rate at which an actually-marking AQM would mark, it would be straightforward to calculate the probability P(we should have seen ECT(0)->CE by now) by standard bayesian reasoning, and the expected number of packets before this probability would fall below a given threshold. But, I do not know if this would be a number that you would like.
>
> Unfortunately I don't have much  spare effort to develop this, but  I hope that has provided food for thought.
>
> Alex
>
>
>
>
>
>> 3) Could be used for Out-of-band Classic AQM detection, but...:
>> The proposal would help server operators who wanted to run pre-L4S-deployment tests to check for the presence of Classic ECN AQMs. Then ehy could run saturating ECT(0) flows for long enough to be sure that lack of CE meant absence of Classic ECN bottlenecks. But only if all L4S implementers complied (see item #1). The fact that this proposal is only really useful for pre-deployment testing would surely make FQ implementers even less keen on removing Classic ECN support.

[BB] I still think out-of-band is where the real merit of the idea lies. 
Especially, because we have your 'A2' idea.

The server could use it as a pre-L4S-deployment check. And/or servers 
could use it to infrequently check different paths in order to check 
whether Classic ECN AQMs have appeared, or to check where they are.

Then all the problems I raised about "not using L4S in order to use L4S" 
would melt away.

Cheers

Bob

>>
>> Perhaps this conversation will spark a variant of your idea. However, in its present form, unless I'm missing something, it doesn't seem to stand up to scrutiny.
>>
>> Cheers
>>
>>
>> Bob
>>
>>
>> On 08/12/2020 19:37, Greg White wrote:
>>
>>
>>>    
>>>    
>>> Hi Alex,
>>>
>>>   
>>>
>>> This does seem worth considering.  FWIW in Low Latency DOCSIS the implementation will be as you describe.  ECT(0) packets will not be marked CE.  This was done for practical reasons, since the classic AQM re-uses the DOCSIS-PIE AQM which is implemented in hardware in many devices, and does not support ECN.  For consistency, any ECT(0) traffic that were to make its way somehow (e.g. DSCP marked as NQB) into the LL queue, will also not be CE marked.
>>>
>>>   
>>>
>>> -Greg
>>>
>>>   
>>>
>>>   
>>>
>>>    
>>> From: "[email-masked]" <[email-masked]>Reply-To: "[email-masked]" <[email-masked]>Date: Monday, December 7, 2020 at 3:44 PMTo: ""[email-masked]"" <[email-masked]>, ""[email-masked]"" <[email-masked]>, Greg White <[email-masked]>Cc: ""[email-masked]"" <[email-masked]>Subject: L4S and the detection of RFC3168 AQMs
>>>
>>>
>>>    
>>>   
>>>
>>>
>>>    Hi all,  At present the DUALQ draft leaves it to implementers to decide if traffic in the classic queue (IE, ECT(0) traffic that uses currently standard congestion controllers) should be dropped or marked (except, apparently, when an ECT(0) packet for some reason turns up in the L queue?). Perhaps it would be wise to specify that during the experiment, deployments of L4S AQMs (whether DUALQ or the other alternatives mentioned in draft-ietf-tsvwg-l4s-arch) SHOULD NOT (or maybe even MUST NOT) mark ECT(0) traffic, and only drop. This would be to preserve the fact that, as currently, those RFC3168 AQMs which are unaware of L4S, can be identified by the fact that they mark ECT(0) packets with CE. If L4S AQMs are required not to mark ECT(0) as CE, then if an endpoint (or other monitoring method) sees a CE mark on an otherwise ECT(0) flow, then it knows with more or less 100% confidence that an RFC3168-only AQM is on the path. Presumably this is safe, since L4S nodes are required to be able to support Not-ECT traffic, and ECT(0) traffic is required to be able to cope with drop.  At present the only definitive way for endpoints identify an RFC3168 AQM in use, is to observe CE marks on an ECT(0) marked flow.Bob's paper [1] gives various other methods, but this seems to be a research project at present. If any of these approaches were found to be reliable, then the above requirement could be relaxed fairly easily; the reverse is not so easy. There are, as is probably obvious, a number of reasons for wanting to identify RFC3168 AQMS: - Current efforts to gather statistics on the number of RFC3168 AQMs which might encounter problems when L4S traffic passes through them. - The the ops draft (sec 3.1)  envisages that CDN operators should test for the presence of RFC3168 AQMs, but doesn't yet specify how this should be achieved - Within a L4S transport protocol, in order to fall back to RFC3168 behavior All of these would presumably be assisted by being able to classify observed ECT(0)->CE transitions unambiguously being the result of an L4S-unaware node. Unless I'm missing something it seems very much worth considering to restrict L4S marking behavior of ECT(0) for the time being? I am not sure which drafts would need updating; DUALQ at least, but presumably the ops draft and maybe L4S-arch. regards, Alex [1] https://arxiv.org/pdf/1911.00710.pdf
>>>
>>>
>> --

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Feb 2021 00:22:57 +0000

Martin,

Greg and I spoke about this very question earlier in the week.

At minimum, we could write all the details covered in this thread into 
the tech report about Classic ECN AQM Fallback that Asad & I prepared 
earlier. "We" might mean Alex or me, or someone else? I would be happy 
to either ACK Alex or include him as a co-author - if he was willing.

I feel that would be more appropriate than an IETF draft at the mo, at 
least while it's not implemented or tested. Then the IETF drafts 
(l4sops, ecn-l4s-id, dualQ, etc.) could give a brief outline of the 
idea, while referring out for fuller details.

In that tech report, there is already a section on ideas for the design 
of active tests, which it says have not been implemented or tested. It 
would be published on arXiv as a new version, which is suitable as an 
archival ref.

Then, as Alex suggests, we would need to update the dualQ draft.
As Alex pointed out, not supporting ECT0 can later be reversed (e.g. if 
the L4S experiment later moves to PS), whereas supporting ECT0 from the 
start, them removing it later would miss the opportunity to provide 
certainty. This is the part I like most.

We would also have to not make ECT0 support the default in the reference 
Linux implementation of the DualQ.

Bob

On 10/02/2021 18:37, Martin Duke wrote:
> Does anyone have the bandwidth to write something down? Should there 
> be a separate draft that articulates the design? Would this replace 
> the current Prague approach?
>
> Martin
>
> On Tue, Feb 9, 2021 at 3:09 PM Bob Briscoe <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Richard,
>
>     >> On Fri, Dec 11, 2020 at 3:03 PM Black, David
>     <[email-masked] <mailto:[email-masked]>
>     >> <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >>
>     >>     with such packets
>     (https://tools.ietf.org/html/rfc8311#section-4.3
>     >>     On Fri, Dec 11, 2020 at 12:09 PM C. M. Heard
>     <[email-masked] <mailto:[email-masked]>
>     >>     <mailto:[email-masked] <mailto:[email-masked]>>> wrote:____
>     >>
>     >>         <[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>>
>     >>         wrote:____
>     >>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 12 Feb 2021 23:31:35 +0000

Martin, Alex, list,

Towards a write-up, here's the way I see Alex's ideas being used for 
out-of-band testing (in contrast to Alex's thoughts, which were for 
in-band testing).

First a reminder of Alex's first idea for a DualQ Coupled AQM (idea A1):
* Make it mandatory for a DualQ Coupled AQM not to mark ECT0 packets, so 
Classic ECN AQMs without L4S support are distinguishable as the only 
AQMs that mark ECT0.

Second a reminder of Alex's second idea for an FQ-L4S AQM (idea A2):
* always marks ECT1,
* but it drops rather than marks ECT0 packets if there has been any 
(recent) ECT1 in the same flow.
* If there hasn't been any ECT1 in a queue, it just marks ECT0 classically.

==Out of band test==
A test server would fill the path with ECT0 until it's getting a strong 
congestion signal (CE or loss), then it would switch a low fraction (say 
5%) of packets to ECT1. The table below then shows the various patterns 
of marking that might occur and what can be concluded about the 
bottleneck. An empty cell means "don't care".

For completeness, I've included all the invalid cases too (explained 
below the table).
If the table is unreadable without HTML email, see 
https://bobbriscoe.net/projects/latency/DisableECT0_OOB_TruthTable.pdf
which also includes the full Truth Table.

All ECT0 	95% ECT0 	5% ECT1 	
drop 	mark 	drop 	mark 	drop 	mark 	Bottleneck queue behaviour
 >0 	0 	>0 	0 	>0 	0 	Tail-drop or Non-ECN AQM (FQ or FIFO)

	>0 	
	>0 	
	>0 	Classic ECN AQM (FQ or FIFO)

	>0 	
	>0 	
	~100% 	FQ-L4S AQM without A2

	>0 	>0 	0 	
	>0 	FQ-L4S AQM with A2
 >0 	0 	>0 	0 	
	>0 	DualQ L4S AQM
*All possible invalid experiment runs*
0 	0 	

	Uncongested

	0 	0 	
	
	Uncongested

	
	0 	0 	Uncongested

	>0 	
	0 	ECT0 but no ECT1 marking yet

	>0 	

	0 	ECT0 but no ECT1 marking yet

	0 	
	>0 	
	
	No ECT0 marking until ECT1 starts

Terminology:

  * congestion mark = CE
  * congestion indication = drop or CE

So that I could check the design of the experiment, I wrote out all the 
possible 2^6 combinations of congestion outputs and crossed off the ones 
that represented each valid output pattern. Then I kept adding invalid 
cases until all the output combinations were crossed off. This led to 
the following invalid cases where an experiment run has to be aborted:

  * If no congestion indication (drop or marking) can be induced on an
    ECT0 packet, either before ECT1 packets are introduced or after, the
    run has to time out;
  * Whether or not there are congestion indications on any ECT0 packets,
    a run has to eventually time out if there is never a congestion
    indication on an ECT1 packet;
  * If any ECT0 packet has been marked, a run has to eventually time out
    if no ECT1 packet is ever marked;
  * If no ECT0 packet is marked before ECT1 packets are introduced, but
    an ECT0 packet is marked after, the run has to abort.

Cheers

Bob

On 11/02/2021 00:22, Bob Briscoe wrote:
> In that tech report, there is already a section on ideas for the 
> design of active tests, which it says have not been implemented or 
> tested. It would be published on arXiv as a new version, which is 
> suitable as an archival ref.
>
> As Alex pointed out, not supporting ECT0 can later be reversed (e.g. 
> if the L4S experiment later moves to PS), whereas supporting ECT0 from 
> the start, them removing it later would miss the opportunity to 
> provide certainty. This is the part I like most.
>
> We would also have to not make ECT0 support the default in the 
> reference Linux implementation of the DualQ.
>
>
> Bob
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 15 Feb 2021 11:23:18 +0000 (UTC)

Hi Bob, Martin, tsvwg

I mostly concur with Bob's points below, so the remainder of this email is largely to satisfy Bob's curiosity (See inline tagged [AB2])

I'm glad that this discussion has been a useful one.

On Thursday, February 11, 2021, 12:35:26 AM GMT, Bob Briscoe <[email-masked]> wrote: 

Alex, 
As I said yesterday, it was only through Richard reviving this thread that I saw your response, which I missed last Dec. Sorry.
See inline tagged [BB]

> [AB] That would be a bad outcome. Perhaps the constraint could be modified to allow that FQ AQMs supporting L4S were allowed to mark ECT(0) following RFC3168, so long as they ONLY do so in buckets…

[BB] Good point. That would work. To keep this email unambiguous, I'll call that A2 (for Alex's idea #2).

It's just occurred to me that, even if an FQ AQM did not implement A2, the AQM would still be very unlikely to mark any ECT(0) packets if they were mixed into an L4S flow of ECT(1) packets, as long as...:
* the threshold for marking ECT(0) is deeper than for ECT(1);
* and particularly if ECT(1) marking is immediate, whereas ECT(0) marking needs persistent queuing 
   (e.g. CoDel's requirement to exceed 'target' for the duration of 'interval'). 
Reason: at any one time a queue can only have one length. So if the L4S flow's congestion control is keeping the queue mostly at the shallow operating point, it will hardly ever be able to persistently exceed a higher threshold.

I'm not saying your A2 idea wouldn't be worth implementing for stronger certainty. I'm just saying that the certainty that ECT(0) packets would not be marked by any L4S AQM would still be near-certainty, even if some FQ implementers didn't implement A2.

Some detailed thoughts on your A2 idea (skip if you want):

> Within one bucket, after an ECT(1) packet, it could disable ECT(0) for the remaining life of the bucket, rather than for a set duration.  
> (This straightforward latch would be a little simpler than a timer. However, it would be more susceptible to an attack on Classic ECN support in which the attacker generates ECT(1) packets with random flow IDs and throws them at an FQ. Nonetheless, I'm sure attackers would rather spend their time mounting more harmful attacks than this.)
> 

>  I think this would also help with your 3) below. 

[BB] Definitely. 
[To save readers scrolling, my point (3) said Alex's original idea would be good for out-of-band testing, but only if it also applies to FQ (which it does now, given idea A2)].

>  Maybe this would also help with the issues of unfairness with a) tunnels and b) L4S and normal flows hashing to the same bucket, since the queue in that bucket would build up to the point of dropp…

[BB] I'm afraid it won't help. Remember, a single queue can only have one length at a time.
If ECT(0) and ECT(1) flows mix in the same bucket (due to hash collision or a tunnel that looks like one flow), the ECT(0) flow will not see any marking at the shallower threshold, so it will drive the queue to the deeper threshold where ECT(0) packets are dropped or marked (depending if A2 is implemented or not).  Then, nearly all the ECT(1) packets will be marked, because they all arrive at a queue that exceeds the shallower threshold. 

[AB2] Oh yes, you are right. 

[BB]
Thinking further about idea A2:
Starting from today's situation, with only Classic flows, hash collisions and tunnels are detrimental to a small proportion of Classic flows. For this small proportion of flows, L4S and idea A2 would add to this detriment by disabling ECN as well.

Given long-running flows are relatively rare, two together are more rare, two with different ECTs will be even rarer, and two together in a tunnel or hash collision are even rarer still, I think that's a probability most people could live with. But others might say the sky is falling.

[BB] [Detail: Can be skipped]
I'm intrigued to know how to do this with only a 2-bit counter.
Perhaps whenever a queue sees ECT(1), it sets its counter to 3 then the whole scheduler decrements all non-zero counters every T seconds, so there will be 2T to 3T duration after the last occurrence of ECT(1) in any queue before that queue marks ECT(0) again.

[AB2] Exactly. It's a bit expensive, so to be practical it depends on the interval being large enough that it doesn't happen frequently.

However, one additional consideration makes this much harder: some Classic AQMs reduce drop probability for smaller packets (for instance DOCSIS PIE [RFC8034; Sec 4.6], which is used as the Classic AQM in the DOCSIS dual queue alongside L4S). So the probability calculation would need to take into account the relative sizes on the wire of data packets and probes. Unfortunately an L4S sender doesn't know which sort of Classic AQM it might encounter, so I think it will have to assume the worst case.

[AB2] That would make sense.

[BB] That's not really a correct way to say it.
Our approach in the experiments in our report was: while a flow is starting to measure whether there's a Classic ECN AQM at the bottleneck (for perhaps 20 or 30 round trips), It can afford to /behave/ like L4S. That's not the same as assuming there are no 3168 AQMs. To justify this, as our report says, I checked the original paper on TCP Cubic, and convergence was still described as ’short’ when it took one or two hundred rounds. Anyway, from the user point of view, surely it's not significant if one flow outcompetes another for a few seconds after flow-start.

[AB2] I stand corrected.

>  If that is the case (and the WG accepts it) then there are no false positives at startup. The only reason then  to worry about the amount of time to wait without seeing ECT(0)->CE is if there is a…

[BB] How and when did you "previously detect" an RFC3168 ECN AQM? I've read this para (and the whole thread) over and over, but I still can't work it out. 

[AB2] Most likely you are looking for something more clever than I intended to write. 'Previous detection' here just meant that in-band detection previously observed a packet it expected to be ECT(0) marked as CE, and thus assumes it has detected an RFC3168 ECN AQM - but sufficiently long ago that it is worth considering if that AQM is no-longer the bottleneck.

[BB]
Whatever, as I say at (3) below, even if this approach cannot detect an RFC3168 ECN AQM quickly enough for in-band testing, it could still be useful for out-of-band testing.

still more...

>> The proposal would help server operators who wanted to run pre-L4S-deployment tests to check for the presence of Classic ECN AQMs. Then ehy could run saturating ECT(0) flows for long enough to be…

[BB] I still think out-of-band is where the real merit of the idea lies. Especially, because we have your 'A2' idea.

The server could use it as a pre-L4S-deployment check. And/or servers could use it to infrequently check different paths in order to check whether Classic ECN AQMs have appeared, or to check where they are. 

Then all the problems I raised about "not using L4S in order to use L4S" would melt away.

Cheers

Bob

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 15 Feb 2021 11:26:56 +0000 (UTC)

Martin, Bob, all,
I am happy for these ideas to be taken forward by Bob or anyone else  - unfortunately I am unlikely to have time to contribute, as I am not working on networking at present.  As such I don't have any opinions on what is the appropriate doc.

best,
Alex

On Thursday, February 11, 2021, 12:29:45 AM GMT, Bob Briscoe <[email-masked]> wrote: 

Martin,

At minimum, we could write all the details covered in this thread into the tech report about Classic ECN AQM Fallback that Asad & I prepared earlier. "We" might mean Alex or me, or someone else? I would be happy to either ACK Alex or include him as a co-author - if he was willing. 

I feel that would be more appropriate than an IETF draft at the mo, at least while it's not implemented or tested. Then the IETF drafts (l4sops, ecn-l4s-id, dualQ, etc.) could give a brief outline of the idea, while referring out for fuller details.

In that tech report, there is already a section on ideas for the design of active tests, which it says have not been implemented or tested. It would be published on arXiv as a new version, which is suitable as an archival ref.

Then, as Alex suggests, we would need to update the dualQ draft.
As Alex pointed out, not supporting ECT0 can later be reversed (e.g. if the L4S experiment later moves to PS), whereas supporting ECT0 from the start, them removing it later would miss the opportunity to provide certainty. This is the part I like most.

We would also have to not make ECT0 support the default in the reference Linux implementation of the DualQ.

Bob

On 10/02/2021 18:37, Martin Duke wrote:

Does anyone have the bandwidth to write something down? Should there be a separate draft that articulates the design? Would this replace the current Prague approach?

Martin

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 15 Feb 2021 17:14:40 +0000

Alex,

I'll go ahead as I proposed (add write-up of this conversation to tech 
report on arXiv), unless some other process is agreed on this thread in 
the near future. Thanks for the ideas, and the responses.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 15 Feb 2021 17:17:23 +0000

Alex,

> [AB2] Most likely you are looking for something more clever than I intended to write. 'Previous detection' here just meant that in-band detection previously observed a packet it expected to be ECT(…

Understood. I (wrongly) took it to possibly imply that some other test 
had been running before your test.

Thanks for the other responses (snipped).

Bob