Subject: Re: [tsvwg] start of WGLC on L4S drafts

From: "Holland, Jake" <[email-masked]>
Date: Tue, 1 Feb 2022 15:59:06 +0000

Hi Bob,

Sorry for the delay, I thought we were done.

Thanks for the accepted updates and related edits, I think they’re mostly helpful improvements.

And rather than going in circles again, as I said originally I propose to agree to disagree on the one remaining point (re: SHOULD NOT be sent across classic queues, MUST NOT be so sent repeatedly and persistently).  I don’t think there’s enough evidence to claim that essentially all current and future deployed marking queues use a sufficiently good fq hash to allow for disregarding of RFC 4774’s model for new ECN semantics, nor that the problem-detection approach is robust enough (not even sure it’s coherent enough) to rely on as the basis for the key normative requirements for a safe operational congestion response.

So if I’m in the rough on that point, to Wes: just please include my position in the shepherd write-up, as requested in my prior message.

-Jake

From: Bob Briscoe [email-masked]<mailto:[email-masked]>n
Date: Mon,2022-01-31 at 1:30 PM
To: "Holland, Jake" <[email-masked]>, Wesley Eddy <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] start of WGLC on L4S drafts

Jake,
Ping
Bob
On 14/01/2022 13:57, Bob Briscoe wrote:
Jake,

Apologies for over a week's delay.

See [BB2] inline for details, including the actual proposed text.

If TL;DR:
In summary, the alternative text suggested in the second half of your email doesn't seem to reflect the views of many in the WG, so it's not going to be a useful basis for further text. Instead we've used the suggestions in the first half to improve the existing text, and tried to cover the points you've made in your second half, by improving the text already in the draft, rather than starting again from your suggested text:
* we've taken on board your suggestion to mandate that detection MUST be enabled not just implemented, but generalized that to "MUST be used".
* we've clarified the different detection cases (real-time vs offline) in the normative requirements, and suggested both in-band and offline are useful together.
* We've made it clear that detection of potential problems includes either detection of actual problems or detection of the likely type of AQM to infer potential problems.
* Also, regarding your 'PS', we've removed the clause about self-interest from the appendix.

On 04/01/2022 19:17, Holland, Jake wrote:

Bob,

From: Bob Briscoe <[email-masked]><mailto:[email-masked]>

Date: Thu,2021-12-23 at 2:29 PM

The current text, as far as I can tell, does not *require* any attempt

to detect problems due to "legacy queue" issues (only "SHOULD" language

here, I think?), and then makes the safety response of failing over to

classic CC conditional on "detecting problems".  Certainly if there is

a "MUST" or "SHALL" requirement, it's hard for me to nail down exactly

where.

[BB] Doesn't the first sentence of the Classic ECN AQM requirement (bullet #3 under §4.3) say what you're wanting?:

See https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22#section-4.3<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22*section-4.3__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPyf1OgXM$>

Possibly you're saying this doesn't make detection mandatory, only monitoring.

Yes, exactly.  This doesn't make detection of problems mandatory, only

monitoring.

If that's your concern, we can easily fix that.  For instance, at the beginning of the second para of this requirement bullet we could add the following highlighted sentence.

I don't believe any implementers will object to this, because I imagine most thought detection was already mandatory. But if any do object, pls do so now!

   3.  In uncontrolled environments, monitoring MUST be implemented to

       support detection of problems with an ECN-capable AQM at the path

       bottleneck that appears not to support L4S and might be in a

       shared queue.  Such monitoring SHOULD be applied to live traffic

       that is using Scalable congestion control.  Alternatively,

       monitoring need not be applied to live traffic, if monitoring has

       been arranged to cover the paths that live traffic takes through

       uncontrolled environments.

       A function to detect the above problems with an ECN-capable AQM

       MUST also be implemented. The detection function SHOULD be capable of making the congestion

       control adapt its ECN-marking response to coexist safely with

       Classic congestion controls such as standard Reno [https:/datatracker.ietf.org/doc/html/rfc5681<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPTzzZSVY$>], as

       required by [https:/datatracker.ietf.org/doc/html/rfc5033<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPMsjjgTI$>].  Alternatively, if adaptation is not

       implemented and problems with such an AQM are detected, the

       scalable congestion control MUST be replaced by a Classic

       congestion control.

[BB2] Later you say 'see below' where you say you'd rather have completely different text in the bullet in 4.3. We (the co-authors) have responded to that in line as well, but we'll answer your first responses here too, which aim to improve the text from where it is.

If that's all that happens I'd ask at least for "implemented and

enabled" in the added sentence, rather than just implemented.

[BB2] I'd rather say 'MUST be implemented and used' because, in the case of offline detection, it doesn't make sense to say "enabled".

To make it clearer which part is talking about online vs offline detection, we also suggest the following changes, highlighted in bold. I've included the unchanged first para of the requirement for context:

   3.  In uncontrolled environments, monitoring MUST be implemented to

       support detection of problems with an ECN-capable AQM at the path

       bottleneck that appears not to support L4S and might be in a

       shared queue.  Such monitoring SHOULD be applied to live traffic

       that is using Scalable congestion control.  Alternatively,

       monitoring need not be applied to live traffic, if monitoring has

       been arranged to cover the paths that live traffic takes through

       uncontrolled environments.
CURRENT 2nd para:

       A function to detect the above problems with an ECN-capable AQM

       MUST also be implemented.  The detection function SHOULD be

       capable of making the congestion control adapt its ECN-marking

       response to coexist safely with Classic congestion controls such

       as standard Reno [RFC5681<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPTzzZSVY$>], as required by [RFC5033<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPMsjjgTI$>].

       Alternatively, if adaptation is not implemented and problems with

       such an AQM are detected, the scalable congestion control MUST be

       replaced by a Classic congestion control.
PROPOSED 2nd para:

       A function to detect the above problems with an ECN-capable AQM

       MUST also be implemented and used.  The detection function SHOULD be

       capable of making the congestion control adapt its ECN-marking

       response in real-time to coexist safely with Classic congestion controls such

       as standard Reno [RFC5681<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPTzzZSVY$>], as required by [RFC5033<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPMsjjgTI$>]. This could

       be complemented by more detailed offline detection of potential problems.

       If only offline detection is used and potential problems with

       such an AQM are detected on certain paths, the scalable congestion control MUST be

       replaced by a Classic congestion control, at least for the problem paths.

       See Section 4.3.1, Appendix A.1.5<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-23*appendix-A.1.5__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPQReeP_k$> and [I-D.ietf-tsvwg-l4sops<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-23*ref-I-D.ietf-tsvwg-l4sops__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkP7FVBxIE$>] for rationale.

RATIONALE:
* As well as the rationale already discussed, we've made sure that, when solely offline detection is used, it has to detect /potential/ problems, which is more stringent than only having to detect actual problems.
* We've also now made it clear that it's OK to replace the CC on a path-specific basis.
* The references to the appendix and l4sops have been moved up one para, because they don't apply to the para they previously followed.

(Among my concerns here are that false positives will be too common

and get the detection turned off, as the testing has shown some

of the proposed detection algorithms had some trouble on that front

IIRC.  But it's also possible to tune the parameters until false

positives are eliminated, with probably a cost to the detection

success in false negatives, so maybe "implemented and enabled"

still doesn't go far enough.)

[BB2] To be clear, the v2 detection algorithm is by no means the end of the road. I stopped work on the algorithm at v2 for two reasons:
a) The grant I had been awarded for a brief project on this problem ran out, and although my business continued funding Asad & myself for a short while, I couldn't sustain that.
b) It became increasingly obvious that we would never satisfy some people, who would always be able to contrive scenarios where the algorithm would fail. So the question would become one of the likelihood of different scenarios. I wasn't willing to keep applying for cash to flush down a hole of such arbitrary size.

Nonetheless, if we had continued, two simple improvements would have drastically cut the false positive rate in the two areas where the very large majority of false positives were:
* link rate < 15MB/s (when serialization delay of single packets confused the algo)
* RTT > 80ms (when the algo needed to allow for more delay variation)
See https://l4s.net/ecn-fbk/results_v2.2/full_heatmap/<https://urldefense.com/v3/__https:/l4s.net/ecn-fbk/results_v2.2/full_heatmap/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPFWqvcqg$>

If that wasn't enough, we had other ideas (see the report), e.g. taking account of more metrics.

However, it's never sensible to assume a problem can be fixed until it's been fixed.
So at this stage, I can only agree with the premise that a high degree of false positives would lead to the algo being disabled.

But really, I think this change misses the point. (See "below".)

(For context, recall that we separated monitoring and detection to allow for the possibility of live monitoring but offline detection, perhaps involving analysis of the monitoring logs supported by further offline testing).

Good point, a reference in that section to the known detection

algorithms would make it much simpler to understand what the

operator is actually required to do by this MUST.

[BB2] We've moved the reference to appendix A.1.5 closer to the MUST requirements. This appendix describes the recommended offline test (at the link I gave to page-50 below), and refers to the tech report for the live test.

We don't want to make any specific test mandatory, because better tests are likely to developed by others. We'd much prefer to mandate the desired outcome, not each action for each case. The future is unpredictable, so attempting to predict every case is likely to miss some, which will leave loop holes.

...

[BB] Offline detection of a Classic ECN AQM (shared queue or FQ) involves a straightforward test (see table 2 at the top of https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22#page-50<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22*page-50__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPBG4QtvY$> ).

Sure, but for a lot of senders it's not operationally simple to

run 2 sustained simultaneous flows to run that test long enough

that you can compare the marking counts.

[BB2] For in-band testing, we understand that.
For out of band, the appendix describes the possibility of serving a test javascript, which opens a pair of connections. Other approaches would be possible, like exploiting probes already deployed for other testing (RIPE Atlas, Sam Knows, planetlab, etc), or deploying test code in clients as part of the download for beta L4S trials, etc.

I'd much rather have a pointer near the appropriate normative

text to your and Asad's Feb 2021 paper ("TCP Prague Fall-back

on Detection of a Classic ECN AQM"), with its overview of several

different detection algorithms for classic queues.  Some of the

tests don't require multiple simultaneous flows, like the active

in-band probe, so they might be usable for services that can't do

the multiple competing flow tests.

[BB2] We'd rather keep the reference to this report in the appendix, which hedges it round with caveats. We don't want to encourage implementers to go straight to the report without reading the context in the appendix.

(The "below" from "see below" starts here: )

I'm now going to merge a response to a point in your Sun 2021-11-07

8:40am message here, because to me this thread is talking about the

same issue, and I don't think I fully responded at the time.  The

full original:

https://mailarchive.ietf.org/arch/msg/tsvwg/fTRTX5ntxhOqO61ggVaIPH8Vwfw/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/fTRTX5ntxhOqO61ggVaIPH8Vwfw/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkP2sYcbhY$>

And the main excerpt I'm answering:

We didn't think it was reasonable to give the sender a requirement that

depends on what type of queue it encounters, because the sender doesn't

know the type for sure. That's why we made this conditional on

'problems' instead. The last sentence of this passage from the

referenced Appendix A.1.5 explains why:

    CDN servers placed within an access ISP's

    network can be considered as a single controlled environment, but any

    onward networks served by the access network, including all the

    attached customer networks, would be unlikely to fall under the same

    degree of coordinated control.  Monitoring is expressed as a 'MUST'

    for these uncontrolled segments of paths (e.g.  beyond the access ISP

    in a home network), because there is a possibility that there might

    be a shared queue Classic ECN AQM in that segment.  Nonetheless, the

    intent is to only require occasional monitoring of these uncontrolled

    regions, and not to burden CDN operators if monitoring never uncovers

    any potential problems, given it is anyway in the CDN's own interests

    not to degrade the service of its own customers.

In summary I think your suggestion of conditioning on the type of queue

is likely to be infeasible and therefore prone to being ignored.

This is not easy but, on balance, I don't think I've seen anything here

that is tighter than the existing wording.

I don't want to go around in circles on this, but lest it be lost:

I don't agree here and still hold an objection open, which is

still the one I'm trying to get covered in this thread about the

normative language.

I'm not sure why you said detection of the type of queue is infeasible,

when in this thread you said "Offline detection of a Classic ECN AQM

(shared queue or FQ) involves a straightforward test".

[BB2] That was my summary of a longer sequence of logic. The 2-flow test doesn't actually determine the type of queuing structure. It determines the relative behaviour of two flows, which directly detects the problem of interest. Then, in turn, one could also /infer/ the type of queuing structure (shared queue, FQ, etc) based on knowledge and assumptions about what is deployed. That in turn could help to detect /potential/ problems even if no actual problem materializes in the specific test.

Whatever, the text you quoted was an email summary of what was in the draft. We only need to focus on the text in the draft.

Also, as you can see, mandating detection of potential problems does not preclude detecting the type of queue and assuming potential problems from that queue type. That's the whole point of mandating the objective, rather than all the possible ways of reaching it. We've added the second sentence below in appendix A.1.5 where it defines what detection is:

   Detection:  Detection involves analysis of the monitored data to

      detect the likelihood of a Classic ECN AQM.  Detection can

      either directly detect actual coexistence problems between flows,

      or it can aim to identify AQM technologies that are likely to

      present coexistence problems, based on knowledge of AQMs deployed

      at the time. ...

Already, since the WGLC, the range of types of FQ has increased, and it is likely to continue to grow:
* From v5.16 of the Linux mainline, CoDel and FQ-CoDel include L4S support as well as Classic, so that will be likely to gradually deploy in more distributions.
* Then there's the suggestion (written up in l4s-arch) to include the last bit of the ECN field in the FQ hash, which would address coexistence within a VPN if implemented.
* And some VPNs might themselves divide into separate SPIs for L4S and Classic.

So, it could become infeasible to determine potential problems based on accurately detecting all these queue sub-types. Which is a good reason for mandating the objective, rather than trying to predict all the possible paths to reach it.

Maybe more importantly, the sender usually can't observe the problems.

The impact is to someone else's traffic, so the sender has no metrics

by which to notice when fairness problems have damaged the competing

traffic.

[BB2] That's the point of 2-flow tests, which can directly observe the problem.

Nonetheless, a live test is still preferred (a 'SHOULD'), because it can probably catch and avoid potential problems /at the time/, even though it probably can't use 2 flows to directly detect problems. Instead it can monitor metrics like delay variation and/or the spacing of marks to infer /potential/ problems, again based on knowledge and assumptions about what is deployed.

This also seems to be claiming that detection of problems caused by a

classic marking queue is more feasible than detecting existence of a

classic marking queue?  I don't see how this can be the case, since

the cause of problems is an inherently hard question, and observed

problems in someone else's traffic can (and usually do) come from

other issues besides congestion.  It would be a major undertaking to

pin down an observed problem in the impacted service's production

traffic to having been caused by this issue.

[BB2] As already explained, we focused on problem detection because it was the more general objective. So it can include:
* detection of queue types in order to infer potential problems,
* or direct detection of actual problems,
* or both for that matter (you'll see the main normative text we proposed earlier now suggests this).

I admit I said that detecting the queue type could be infeasible, but I have now realized there's a broader reason for not mandating detection of queue type: we don't have to make judgements in the draft about feasibility when the requirement is solely to reach an objective, which allows the implementer or operator to choose whatever approach is most feasible to reach that objective. Then the copious supporting text gives guidance (§4.3.1, Appx A.1.5 and draft-l4sops and the tech report on fallback).

Anyway, I still think detection of a classic queue that's marking

traffic from an L4S sender should come with the presumption that a

problem is present, for the reasons explained in RFC 4774 about

different marking semantics.  So I would still like to see normative

language that directly addresses the incompatibility issue by saying

something like:

- L4S traffic MUST NOT be sent persistently and repeatedly over a

  long period of time through a marking classic queue bottleneck.

- L4S traffic SHOULD NOT be sent through a marking classic queue at

  all.  However, classic marking queues MAY be presumed absent on

  unknown internet paths as long as detection techniques are employed

  to notice classic marking if present. [ref. to poll/prevalence stats,

  ref. to detection techniques]

[BB2] These words are only useful if one believes that FQ is not and never will be a sufficient form of flow rate isolation between L4S and Classic flows.

The relatively new section 4.3.1 attempts to document why the WG did not adopt that position in relation to the coexistence required by RFC4774.

(I'd be fine here with a timescale of several hours to a day, if you

want to pin down "long period of time" and allow for offline and

probing-based detection.)

There were a lot of words responding the first time I asked for

something like that (more than just the above-quoted excerpt), but

to summarize, I think I saw you say you disagree because:

a. it's too strict since FQ means there's not a serious problem except

   for tunneled traffic

   (but IIRC you cited "60-70%" of traffic as tunneled, back when

   we were debating the feasibility of different signaling, so I'm

   not sure why it's ignorable?:

   https://mailarchive.ietf.org/arch/msg/tsvwg/-uFazVf2tupkK011Y5ps83ENCvc/<https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/-uFazVf2tupkK011Y5ps83ENCvc/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPG3uopsw$> )

[BB2] I said there's not a serious problem except where the FQ AQM is within a /L3 VPN/ tunnel (not all tunnels, because they propagate ECN to the outer and back).
The 60-70% was for /all/ tunnelled traffic (and it was a guess). L3 VPNs are only a part of that.
And bear in mind that there is only a coexistence problem if there are simultaneously 2 or more long-running flows within the VPN and that they are from a mix of L4S and Classic sources.

b. it's not feasible (or too burdensome?) to detect the type of queue

   that's marking

   (but also straightforward to do offline and thus already obvious to

   implementors what they'll need to do?)

c. the current language does a better job expressing what's needed

   (but implicitly requires senders to somehow get informed when

   they've degraded traffic sent by others, via some unspecified

   method?)

[BB2]
For b) see earlier point that focusing on detection of problems allows the most feasible approach.
For c) see earlier point about testing degradation using 2-flow testing.

I'm unconvinced by any of these claims, and I still don't think this

doc should move forward without normative language that clearly

requires the sender to actively notice when a classic queue is marking

L4S traffic and to avoid allowing it to continue happening.

And with the proposed change at the beginning of this message (the

one I said misses the point), I think the language doesn't do what

it needs to do.  When I read section 4.3 of l4s-id (including the

update) I don't see a sufficiently clear "it's my job as a sender

to ensure I don't damage other people's traffic by carefully making

sure I never collide with them in a classic queue to their detriment

even though it doesn't hurt me".  Nor do I see anywhere a way

described for a sender to detect that a fairness problem has

occurred, as opposed to just detecting presence of a classic marking

queue.

[BB2] The normative text was never intended to tell L4S senders to never collide in a classic queue. It is intended to ensure that (paraphrasing), where such a collision could potentially be problematic, the sender has to switch to Classic, preferably in real-time, but otherwise on management timescales. I hope you agree that the text does do that, even if you still don't think that's the right thing to do.

At this point I'm not clear whether your position changed after the

Nov 7 email or whether there's a narrower context about the

infeasibility of the classic queue detection that I missed or where

exactly the disagreement is.  Detecting problems in other people's

traffic via monitoring and properly assigning the blame to the wrong

marking semantics from a classic queue seems much less feasible than

detecting that a classic queue is performing marking.

[BB2] I don't think the position has changed, but I believe we are now articulating the reasoning better. In particular the point that mandating detection of potential problems doesn't preclude detecting the queue type to infer problems.

But like I said, I don't want to go in circles here.  If you still

disagree that we should change the normative language to strongly

discourage sending L4S across classic marking queues and the wg wants

to move on, I'll just ask that my objection be included in the

shepherd's write-up when it goes to the IESG, and we can maybe agree

to disagree.

HTH.

[BB2] Yes it does. Thank you.

-Jake

PS: After reading again the text you cited from Appendix A.1.5 quoted

above, I noticed it mis-states matter, and also should probably change.

The issue is that if a CDN does consistently send L4S traffic that

collides with competing traffic in a classic marking queue, the CDN's

performance for its customers will improve, not degrade.  It's the

competing traffic, coming from a service not using that CDN, that will

be degraded.  The end user experience for that competing traffic would

also likely degrade, but that's not the CDN customer.

The current text makes it sound like the incentives are aligned

between the CDNs and end users, but I think that's backwards.  The

incentive for the CDN is to use the difficulties in detection as an

excuse to crowd out the competing traffic and claim superior

performance for their traffic to the extent they can get away with

it, even to the detriment of end users' experience coming from

non-customers of the CDN.

End users who have trouble will complain to the impacted service, who

was NOT the sender of L4S traffic, and was NOT customer of the

non-conformant CDN.

I considered this less urgent than the normative language, but if it's

the justification for the reasoning here, perhaps it's worth raising

this point after all.  Sorry for leaving it out of the last review, I

was trying to stay focused on what's most important.

[BB2] I assume you're only referring to the following clause, because the rest of the appendix doesn't even get close to mentioning motivating interests, AFAICT:
    "given it is anyway in the CDN's own interests not
   to degrade the service of its own customers"

We can just remove that (it is now removed). Nothing else relies on it.

Nonetheless, although I'm not going to fight for this text, there is an argument for it:
It's based on a similar argument, which is the only decent explanation I've seen for the continued voluntary use of congestion control. It runs like this: End-users might detect that one service is performing poorly, but they are not usually in a position to detect unfairness as a cause, because it is too intermittent and intangible. It is more feasible that unfairness problems will be uncovered by researchers, then get posted on forums. End-users would not appreciate finding out that one information service is improving its own performance at the expense of other services that the end-user values. So it is not in the reputational interest of a service to be caught degrading other services used by their end-users.

That argument can be extended to an intermediary, such as a CDN.

Whatever, we don't need to go anywhere near this argument in this draft.

Thank you for continuing to engage with improving these drafts.

Bob & Koen

--

________________________________________________________________

Bob Briscoe                               http://bobbriscoe.net/<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPx506Uc8$>

--

________________________________________________________________

Bob Briscoe                               http://bobbriscoe.net/<https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPx506Uc8$>

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 1 Feb 2022 16:44:22 +0000

OK Jake, I understand now.
We'll go ahead and post the draft with the updated wording, and yes, I'm 
sure Wes will add your position to the Shepherd's write-up so the IESG 
can take it into account.

Bob

On 01/02/2022 15:59, Holland, Jake wrote:
> And rather than going in circles again, as I said originally I propose 
> to agree to disagree on the one remaining point (re: SHOULD NOT be 
> sent across classic queues, MUST NOT be so sent repeatedly and 
> persistently).  I don’t think there’s enough evidence to claim that 
> essentially all current and future deployed marking queues use a 
> sufficiently good fq hash to allow for disregarding of RFC 4774’s 
> model for new ECN semantics, nor that the problem-detection approach 
> is robust enough (not even sure it’s coherent enough) to rely on as 
> the basis for the key normative requirements for a safe operational 
> congestion response.
>
> So if I’m in the rough on that point, to Wes: just please include my 
> position in the shepherd write-up, as requested in my prior message.
>
> -Jake
>
> *From: *Bob Briscoe [email-masked]
> *Date: *Mon,2022-01-31 at 1:30 PM
> *To: *"Holland, Jake" <[email-masked]>, Wesley Eddy 
> <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>
> *Subject: *Re: [tsvwg] start of WGLC on L4S drafts
>
> Jake,
> Ping
> Bob
>
>     In summary, the alternative text suggested in the second half of
>     your email doesn't seem to reflect the views of many in the WG, so
>     it's not going to be a useful basis for further text. Instead
>     we've used the suggestions in the first half to improve the
>     existing text, and tried to cover the points you've made in your
>     second half, by improving the text already in the draft, rather
>     than starting again from your suggested text:
>     * we've taken on board your suggestion to mandate that detection
>     MUST be enabled not just implemented, but generalized that to
>     "MUST be used".
>     * we've clarified the different detection cases (real-time vs
>     offline) in the normative requirements, and suggested both in-band
>     and offline are useful together.
>     * We've made it clear that detection of potential problems
>     includes either detection of actual problems or detection of the
>     likely type of AQM to infer potential problems.
>         From: Bob Briscoe<[email-masked]>  <mailto:[email-masked]>
>
>             Seehttps:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22#section-4.3  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22*section-4.3__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPyf1OgXM$>
>
>                     Classic congestion controls such as standard Reno [https:/datatracker.ietf.org/doc/html/rfc5681  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPTzzZSVY$>], as
>
>                     required by [https:/datatracker.ietf.org/doc/html/rfc5033  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPMsjjgTI$>].  Alternatively, if adaptation is not
>
>     [BB2] Later you say 'see below' where you say you'd rather have
>     completely different text in the bullet in 4.3. We (the
>     co-authors) have responded to that in line as well, but we'll
>     answer your first responses here too, which aim to improve the
>     text from where it is.
>
>
>     [BB2] I'd rather say 'MUST be implemented and used' because, in
>     the case of offline detection, it doesn't make sense to say "enabled".
>
>     To make it clearer which part is talking about online vs offline
>     detection, we also suggest the following changes, highlighted in
>     *bold*. I've included the unchanged first para of the requirement
>     for context:
>
>             as standard Reno [RFC5681  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPTzzZSVY$>], as required by [RFC5033  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPMsjjgTI$>].
>
>             MUST also be implemented*and used*.  The detection function SHOULD be
>
>             response*in real-time*  to coexist safely with Classic congestion controls such
>
>             as standard Reno [RFC5681  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5681__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPTzzZSVY$>], as required by [RFC5033  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/rfc5033__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPMsjjgTI$>].*This could*
>
>     *       be complemented by more detailed offline detection of
>     potential problems.*
>
>             If*only offline detection*  is*used*  and*potential *problems with
>
>             such an AQM are detected*on certain paths*, the scalable congestion control MUST be
>
>             replaced by a Classic congestion control*, at least for the problem paths*.
>
>             See*Section 4.3.1,*  Appendix A.1.5  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-23*appendix-A.1.5__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPQReeP_k$>  and [I-D.ietf-tsvwg-l4sops  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-23*ref-I-D.ietf-tsvwg-l4sops__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkP7FVBxIE$>] for rationale.
>
>
>     RATIONALE:
>     * As well as the rationale already discussed, we've made sure
>     that, when solely offline detection is used, it has to detect
>     /potential/ problems, which is more stringent than only having to
>     detect actual problems.
>     * The references to the appendix and l4sops have been moved up one
>     para, because they don't apply to the para they previously followed.
>
>
>     [BB2] To be clear, the v2 detection algorithm is by no means the
>     end of the road. I stopped work on the algorithm at v2 for two
>     reasons:
>     a) The grant I had been awarded for a brief project on this
>     problem ran out, and although my business continued funding Asad &
>     myself for a short while, I couldn't sustain that.
>     b) It became increasingly obvious that we would never satisfy some
>     people, who would always be able to contrive scenarios where the
>     algorithm would fail. So the question would become one of the
>     likelihood of different scenarios. I wasn't willing to keep
>     applying for cash to flush down a hole of such arbitrary size.
>
>     Nonetheless, if we had continued, two simple improvements would
>     have drastically cut the false positive rate in the two areas
>     where the very large majority of false positives were:
>     See https://l4s.net/ecn-fbk/results_v2.2/full_heatmap/
>     <https://urldefense.com/v3/__https:/l4s.net/ecn-fbk/results_v2.2/full_heatmap/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPFWqvcqg$>
>
>     So at this stage, I can only agree with the premise that a high
>     degree of false positives would lead to the algo being disabled.
>
>
>
>     [BB2] We've moved the reference to appendix A.1.5 closer to the
>     MUST requirements. This appendix describes the recommended offline
>     test (at the link I gave to page-50 below), and refers to the tech
>     report for the live test.
>
>     We don't want to make any specific test mandatory, because better
>     tests are likely to developed by others. We'd much prefer to
>     mandate the desired outcome, not each action for each case. The
>     future is unpredictable, so attempting to predict every case is
>     likely to miss some, which will leave loop holes.
>
>
>             ...
>
>             [BB] Offline detection of a Classic ECN AQM (shared queue or FQ) involves a straightforward test (see table 2 at the top ofhttps:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22#page-50  <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-22*page-50__;Iw!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPBG4QtvY$>  ).
>
>     For out of band, the appendix describes the possibility of serving
>     a test javascript, which opens a pair of connections. Other
>     approaches would be possible, like exploiting probes already
>     deployed for other testing (RIPE Atlas, Sam Knows, planetlab,
>     etc), or deploying test code in clients as part of the download
>     for beta L4S trials, etc.
>
>
>     [BB2] We'd rather keep the reference to this report in the
>     appendix, which hedges it round with caveats. We don't want to
>     encourage implementers to go straight to the report without
>     reading the context in the appendix.
>
>
>
>         https://mailarchive.ietf.org/arch/msg/tsvwg/fTRTX5ntxhOqO61ggVaIPH8Vwfw/  <https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/fTRTX5ntxhOqO61ggVaIPH8Vwfw/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkP2sYcbhY$>
>
>     [BB2] That was my summary of a longer sequence of logic. The
>     2-flow test doesn't actually determine the type of queuing
>     structure. It determines the relative behaviour of two flows,
>     which directly detects the problem of interest. Then, in turn, one
>     could also /infer/ the type of queuing structure (shared queue,
>     FQ, etc) based on knowledge and assumptions about what is
>     deployed. That in turn could help to detect /potential/ problems
>     even if no actual problem materializes in the specific test.
>
>     Whatever, the text you quoted was an email summary of what was in
>     the draft. We only need to focus on the text in the draft.
>
>     Also, as you can see, mandating detection of potential problems
>     does not preclude detecting the type of queue and assuming
>     potential problems from that queue type. That's the whole point of
>     mandating the objective, rather than all the possible ways of
>     reaching it. We've added the second sentence below in appendix
>     A.1.5 where it defines what detection is:
>
>            detect the likelihood of a Classic ECN AQM.*Detection can *
>
>     *      either directly detect actual coexistence problems between
>     flows, *
>
>     *      or it can aim to identify AQM technologies that are likely to *
>
>     *      present coexistence problems, based on knowledge of AQMs
>     deployed *
>
>     *      at the time.*  ...
>
>
>     * From v5.16 of the Linux mainline, CoDel and FQ-CoDel include L4S
>     support as well as Classic, so that will be likely to gradually
>     deploy in more distributions.
>     * Then there's the suggestion (written up in l4s-arch) to include
>     the last bit of the ECN field in the FQ hash, which would address
>     coexistence within a VPN if implemented.
>     So, it could become infeasible to determine potential problems
>     based on accurately detecting all these queue sub-types. Which is
>     a good reason for mandating the objective, rather than trying to
>     predict all the possible paths to reach it.
>
>
>     Nonetheless, a live test is still preferred (a 'SHOULD'), because
>     it can probably catch and avoid potential problems /at the time/,
>     even though it probably can't use 2 flows to directly detect
>     problems. Instead it can monitor metrics like delay variation
>     and/or the spacing of marks to infer /potential/ problems, again
>     based on knowledge and assumptions about what is deployed.
>
>
>     [BB2] As already explained, we focused on problem detection
>     because it was the more general objective. So it can include:
>     I admit I said that detecting the queue type could be infeasible,
>     but I have now realized there's a broader reason for not mandating
>     detection of queue type: we don't have to make judgements in the
>     draft about feasibility when the requirement is solely to reach an
>     objective, which allows the implementer or operator to choose
>     whatever approach is most feasible to reach that objective. Then
>     the copious supporting text gives guidance (§4.3.1, Appx A.1.5 and
>     draft-l4sops and the tech report on fallback).
>
>
>
>     [BB2] These words are only useful if one believes that FQ is not
>     and never will be a sufficient form of flow rate isolation between
>     L4S and Classic flows.
>
>     The relatively new section 4.3.1 attempts to document why the WG
>     did not adopt that position in relation to the coexistence
>     required by RFC4774.
>
>
>             https://mailarchive.ietf.org/arch/msg/tsvwg/-uFazVf2tupkK011Y5ps83ENCvc/  <https://urldefense.com/v3/__https:/mailarchive.ietf.org/arch/msg/tsvwg/-uFazVf2tupkK011Y5ps83ENCvc/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPG3uopsw$>  )
>
>
>     [BB2] I said there's not a serious problem except where the FQ AQM
>     is within a /L3 VPN/ tunnel (not all tunnels, because they
>     propagate ECN to the outer and back).
>     And bear in mind that there is only a coexistence problem if there
>     are simultaneously 2 or more long-running flows within the VPN and
>     that they are from a mix of L4S and Classic sources.
>
>
>     [BB2] The normative text was never intended to tell L4S senders to
>     never collide in a classic queue. It is intended to ensure that
>     (paraphrasing), where such a collision could potentially be
>     problematic, the sender has to switch to Classic, preferably in
>     real-time, but otherwise on management timescales. I hope you
>     agree that the text does do that, even if you still don't think
>     that's the right thing to do.
>
>
>
>     [BB2] I don't think the position has changed, but I believe we are
>     now articulating the reasoning better. In particular the point
>     that mandating detection of potential problems doesn't preclude
>     detecting the queue type to infer problems.
>
>
>     [BB2] I assume you're only referring to the following clause,
>     because the rest of the appendix doesn't even get close to
>     mentioning motivating interests, AFAICT:
>     It's based on a similar argument, which is the only decent
>     explanation I've seen for the continued voluntary use of
>     congestion control. It runs like this: End-users might detect that
>     one service is performing poorly, but they are not usually in a
>     position to detect unfairness as a cause, because it is too
>     intermittent and intangible. It is more feasible that unfairness
>     problems will be uncovered by researchers, then get posted on
>     forums. End-users would not appreciate finding out that one
>     information service is improving its own performance at the
>     expense of other services that the end-user values. So it is not
>     in the reputational interest of a service to be caught degrading
>     other services used by their end-users.
>
>     Bob Briscoehttp://bobbriscoe.net/  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPx506Uc8$>
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!G9ZlHoKROZdEPK06q7pJ3PDsM2ushLsEOEJ3GzRFnAw0uDur-pZgZMkPx506Uc8$>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 17 Feb 2022 18:33:27 +0000

Jonathan,

David has pointed me at the description of your scenario about a 
throughput bonus that he was talking about in the interim yesterday, and 
I have traced it back to an experiment you described in the paper linked 
in your August posting below.

1/ Firstly, let's get a goal straight. In the L4S architecture, where it 
offers either an FQ or a DualQ option, it says:

           The L4S architecture does not require the IETF to commit to
           one approach over the other, because it supports both, so that
           the 'market' can decide.  Nonetheless, in the spirit of 'Do
           one thing and do it well' [McIlroy78  <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-l4s-arch#ref-McIlroy78>], the DualQ option
           provides low delay without prejudging the issue of flow-rate
           control.  Then, flow rate policing can be added separately if
           desired.

Do you understand and accept that as a goal of the L4S activity, 
irrespective of whether you would advocate per-flow throughput control 
yourself?

2/ Turning to your scenario, you say the following:
"Both flows were configured to use CUBIC congestion control with ECN 
negotiated, but one was additionally tweaked to set ECT(1) instead of 
ECT(0) on all
data segments, and to pace its output at 40Mbps."
Does "pace" mean sending at 40Mb/s irrespective of congestion feedback?

You say "This latter measure [the pacing] prevents the L queue from 
seeing any need to apply congestion signals, because it is always empty."
However, in the scenario shown, although the L queue is indeed always 
empty, it will see a high level of congestion signals (~10% in this 
case) via the coupling.

Your paper reads as if you don't understand how the coupled AQM works. 
So I don't know whether you /thought/ the L queue would see no marking, 
or whether you actually measured the marking. If you found none, there 
must have been something wrong with your setup.

If we assume your setup was fine, then the 'tweaked' ECT(1) CUBIC flow 
would have seen about 10% ECN marking which would have driven its rate 
down well below 40Mb/s. This is why I suspect that the 40Mb/s pacing was 
actually unresponsive pacing.

If so, the DualQ would have been working as intended. This is not a 
'fast lane' or a 'bonus' to L4S. Because whether you had fed the 
'tweaked' unresponsive 40Mb/s flow into the L queue or the C queue, it 
would have got 40Mb/s.  Because that's what unresponsive traffic gets in 
a FIFO, which is what the DualQ is intended to emulate (as in the quote 
at the start). Then the operator can add per-flow rate policing if it 
wants to, just as it can in a FIFO today (but hardly any do).

Bob

On 26/08/2021 23:20, Jonathan Morton wrote:
>> On 29 Jul, 2021, at 7:18 pm, Wesley Eddy<[email-masked]>  wrote:
>>
>> This message is starting a combined working group last call on 3 of the L4S drafts:
>>
>> - Architecture:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/
>>
>> - DualQ:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-aqm-dualq-coupled/
>>
>> - ECN ID:https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>>
>> The WGLC will last through 4 weeks from today, and then we'll see what to do next.  Please submit any comments you have on these to the TSVWG list in that timeframe.
> I have read through these drafts more times than I really wanted to, and frankly can't shake the feeling of listening to a mistuned piano whilst doing so.  I hereby object to the advancement of these drafts to the next stage of publication, as they are clearly in no state to do so.
>
> My detailed comments, which among other things provide a technical justification for the bullet points read out by David Black during the recent meeting, are available here:
>
> 	https://sce.dnsmgr.net/downloads/L4S-WGLC2-objection-details.pdf
>
> - Jonathan Morton