Subject: Re: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

From: Sebastian Moeller <[email-masked]>
Date: Sun, 1 Nov 2020 10:55:07 +0100

Hi Koen,

more below in-line, prefixed with [SM]

Gesendet: Samstag, 31. Oktober 2020 um 23:33 Uhr
Von: "De Schepper, Koen (Nokia - BE/Antwerp)" <[email-masked]>
An: "Sebastian Moeller" <[email-masked]>, "Bob Briscoe" <[email-masked]>
Cc: "tsvwg IETF list" <[email-masked]>, "iccrg IRTF list" <[email-masked]>, "TCP Prague List" <[email-masked]>
Betreff: RE: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

Hi Sebastian,
 
It was never our intent to develop the Congestion Control itself, as it was not in our interest (as a network provider). 

        [SM] I understand that, but it turns out that to asses functionality and safety of the proposed changes to ECN signalling you absolutely required a way to actually test under realistic conditions, and hence your hand was forced if ever to succeed. As a consequence TCP Prague developed into your umping ground for features you did not really want but had to accept... like the abysmal 15ms hack (good luck getting that into any OS networking stack).

There are plenty of bright people that can develop a Prague-compliant CC that will benefit their business (and at least comply to the safety measures). 

        [SM] But it still is incumbent upon you top demonstrate that the requirements are actually achievable and sufficient to guarantee functionality and safety (as that is the prove that L4S actually would work). I understand that most of the little conditions tested used DCTCP as protocol, so even for the one condition tested so far, short-RTT low HOP-count paths, most tests will need to be repeated with TCP Prague, not that I am holding my breath to see those results, given that I outlined a number of completely missing tests to evaluate functionality and safety outside the narrow conditions tested so far.

>From the start we realized that the small buffer is a potential safety issue for the typical r~=1/RTT congestion controls (also confirmed by our early tests and later by Pete’s tests too).

        [SM] IIRC, that was a well established fact and not a terribly novel insight...

 We hoped by showing that (once one puzzles out how) making a CC RTT independent for the unsafe range (small RTTs) is an easy obvious hack. To please you, we even made a “Sebastian-version” with f(rtt)=rtt+15ms that only removes the RTT benefit from the DualQ (one of your main complaints), and behaves further as a Classic RTT dependent flow. The Sebastian-version solves the safety issue and your fairness issue, but is not our preferred version.

        [SM] I do take offence in that name Sebastian-version, and in presenting that hack as anything but a work-around for a DualQ design error. As I explained patiently in the past, normal RTT dependence of TCP throughput is not a function of the endpoints (as as long as a single flow along a path can saturate the link, the end-points obviously are capable of reaching their goal independent of RTT, hence are RTT independent, I know this is within reason, as there are limits on the aviebale throughput depending on RTT and maximal window sizes, but these are not the issues at play here) but of the buffering in the network, most importantly the buffering at the bottlenecks. And hence that issue should be solved there.
BUT to add insult to injury, you are only proposing to make up for the 15ms DualQ might be configured with (might be as last time I looked at the code it looked more like classic's target was configured to 10ms, but I digress).

 We like more the one that solves the safety issue with f(rtt)=max(rtt, 15ms), and still allows L4S to benefit from the lower delay in all other cases.

        [SM] This is not a safety issue. It is a rate throttling issue. But humour me, how does curvy RED actually perform with all of these gross hacks, under which version does it achieve equitable sharing? 

Then thanks to your insisting to realizse literally what we have written, 

        [SM] So you intended to formulate requirements in your drafts as loose guiding principles without actual interpreting these as binding definitions? And you needed my input to realise basically what "requirement" means. Koen, I have a hard time believing that. 

we realized that the text was indeed too strict if read out of the larger context. In the context of the safety requirements section, we never wanted to enforce RTT fairness for flows with a bigger than the reference RTT, as they don’t pose a safety issue (it is a reality for all longer RTTs today, partially solved by Cubic already), and the quoted text could indeed be interpreted that way out of the larger context. Blindly trying to implement RTT independence for larger RTTs could even make it unsafe, if not done correctly. Of course, if wanted and beneficial, others are free to further optimize.
 
        [SM] You have been advertising L4S on pie-in-the-sky features like RTT-independence for years, in spite of having been told about it impossibility before, and now you seem to finally understand that. Again, I have a hard time believing that. I am also puzzled how this fits with how engineers typically are thought to work.

So I hope you understand now why we want to fix this text (and don’t see this as a failure), and hope you are ok too, to keep this safety requirement to the safety aspect only. 

        [SM] No, as I stated in the past, the 15ms issue, is a DialQ issue, and needs to be fixed in DualQ, then the RTT-independence requirement can be kept as purely inspirational. The current version with its hedge "as wide a range of RTTs as possible" is IMHO still better than the new "the minimum likely RTT and typical RTTs expected in the intended deployment scenario" because that is plainly wrong. By artificially coupling the CC response and the AQM (with the brain dead 15ms) your AQM no MUST not account for likely or expected RTTs but it MUST account for what the CCs are configured for, otherwise the uneqitable sharing issue will crop up again.
       Again this is a logical conclusion based on the unfortunate layer violating design choices you made, so please do not frame this as an issue of RTTs.
        IMHO, the fact that DualQ has this issue is a clear indication, that the cute coupling idea, simply is not suitable for the messy reality, and hacking around that by a requirement for transport protocols is insane, BUT then not clearly pointing out the dependency in the requirement section like "the CC's temporal response dynamic lower bound needs to be configured to match the L4S AQM expected on the path, or 15ms by default" is not going to make protocol implementers understand this subtle issue, as this is NOT an RTT bias issue to start with that is supposed to be covered by this requirement, but a DualQ misdesgn. Hiding this relations does not help anybody.

If you think the text is still not clearly representing the safety requirement of the RTT independence aspect, then your suggestions are very welcome.
 
        [SM] If you truly believe that, then I fear you have not understood the issue very well. Try to read this recommendation from the perspective of a naive protocol implementer. How will she know that she needs to look into the code/default configuration of the DualQ to be able to properly define the response dynamics lower bound, heck how is she supposed to understand that she needs to artificially slow down the CC response for some RTTs and make the CC control loop actively worse from the text of the requirement. IMHO the way this is phrased tries to hide the actual causalities at play here, and that never is a good idea in engineering or science.

As to your request to let the AQM make flows RTT independent, that is unfortunately impossible without knowing the RTT of each packet or identifying and treating the flows individually. 

        [SM] Well, it is less impossible than for the end-points. As long as you keep that requirement at all (and I can agree with getting rid of of it, as it is impossible to achieve realistically) you IMHO NEED to solve it at the component/layer responsible for that issue. Arguing it is hard for the AQM does not hold water as it is IMPOSSIBLE for the end-points in any useful generalized fashion. Giving the current proposals, I can almost guarantee that none of alternative transport protocols to TCP Prague will implement that  15ms hack at all (as the requirement is sufficiently vague to make it impossible to understand what is actually going on unless the implementer followed the L4S saga on this list*), and hence your safety issue will not be addressed at all. Again, I have a hard time believing that you assume that the changed text can be interpreted correctly without a lot of unwritten information, so how about you add a description of how a errr in DualQ makes a 15ms recommended?

So can you explain why you claim that what actually is easy (RTT-indep CC) is impossible and why you insist on what is impossible (RTT-indep AQM) should be done instead???
 
        [SM] Yes, easily. RTT dependence is a direct result of processes of competition at the bottleneck queue, where shorter RTT translates to tighter control loops and hence faster reactions to changing estimates of capacity, that way shorter RTT flows pick up the excess capacity the transiently appears when (longer RTT) flows go to multiplicative decrease. In addition the competition the bottleneck buffer can lead to a higher dropping probability for log RTT flows (precisely because they react slower). There are two ways to tackle this:
a) artificially slow down all flows to the same response dynamics as the longest RTT flow displays. That is the way you started on, but you clearly realised how sub-optimal this is, given that RTT are not really bounded to anything sane (think paths over geostationary satellites), so you already cheated by limiting that to the 15ms you foster on non L4S flows in your AQM.
       Now, I admit that technically it is not completely IMPOSSIBLE to generally solve this issue, by making all flows react as if their RTT would be >= say 500ms, but I do mantain, that nobody sane would accept that as a viable solution. Making it effectively possible but inacceptable.

b) ameliorate the competition in the bottleneck buffers. That is exactly par for the course for advanced queue management.

I have made the same arguments in the past, without convincing demonstrations that my theory is flawed, but feel free to poke holes into it. But again, hard to believe that you seem to have forgotten, that the two of us had pretty much that discussion before.

Best Regards
        Sebastian

*) Comments from members of the BBR and Quicc communities indicate no sufficiently deep involvement as far as I can tell.

Thanks and regards,
Koen.

From: Sebastian Moeller <[email-masked]>
Sent: Saturday, October 31, 2020 4:50 PM
To: Bob Briscoe <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>; iccrg IRTF list <[email-masked]>; TCP Prague List <[email-masked]>; De Schepper, Koen (Nokia - BE/Antwerp) <[email-masked]>
Subject: Aw: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

Dear list,

please note that both of these "requirements" hedge against the fact that RTT independence is almost impossible to achieve from the endpoints (as it is a consequence of buffering at the bottleneck), originally bzy stating "as wide a range of RTTs as possible" (which given the phisical constraints in reality is equal to the empty set). Replacing that non-requirement with a differently worded version ("as much as possible") does not change anything substantially, except for making TCP Prague appear less of a failure. 

IMNHO this requirement for an L4S compatible transport, should be replaced with a requirement for an L4S compliant AQM (as it is limited queueing at the bottleneck that causes most the issue) MUST at least perform as well as a dumb FIFO, a mark which DualQ at short RTTs misses badly. But I guess, a no regressions against the status quo requirement, will probably not apply to L4S.

Best Regards

        Sebastian

Gesendet: Freitag, 30. Oktober 2020 um 19:42 Uhr
Von: "Bob Briscoe" <[email-masked][mailto:[email-masked]]>
An: "tsvwg IETF list" <[email-masked][mailto:[email-masked]]>
Cc: "iccrg IRTF list" <[email-masked][mailto:[email-masked]]>, "TCP Prague List" <[email-masked][mailto:[email-masked]]>, "De Schepper, Koen (Koen)" <[email-masked][mailto:[email-masked]]>
Betreff: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text

Folks,

The co-authors of ECN L4S ID have been reviewing the correctness of the normative 'Prague' requirements.
    See https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3[https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3]
This is the first of 2 emails, about 2 of the requirements that we think ought to be reworded a little.

If you agree with the rationale, but think the new wording still doesn't capture the requirement well, pls suggest sthg better.
If you disagree with the rationale, pls discuss.
 
4.3.  Prerequisite Congestion Response
...
CURRENT:
   o  A scalable congestion control MUST reduce or eliminate RTT bias
      over as wide a range of RTTs as possible, or at least over the
      typical range of RTTs that will interact in the intended
      deployment scenario (see Appendix A.1.5[https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#appendix-A.1.5] for rationale).

PROPOSED:
A scalable congestion control MUST eliminate RTT bias as much as possible in the range between the minimum likely RTT and typical RTTs expected
in the intended deployment scenario  (see Appendix A.1.5 for rationale).

RATIONALE:
1/ "eliminate as much as possible" is stronger than "reduce or eliminate".
2/ This requirement was motivated by 'do no harm to others' relative to existing standard (RFC5681 Reno) congestion control. So there is no need to mandate that an L4S implementer does no harm to themselves, which window-based congestion controls tend to do at higher RTT. Of course, this doesn't preclude implementers reducing or eliminating RTT bias for larger than typical RTTs, but it removes any requirement to do so.

Cheers

Bob
 
--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/[http://bobbriscoe.net/]

---

From: Sebastian Moeller <[email-masked]>
Date: Sun, 1 Nov 2020 12:18:19 +0100

Bob, Koen, Wes,

please find my proposal for a clarification below, prefixed [SM]

Gesendet: Freitag, 30. Oktober 2020 um 19:42 Uhr
Von: "Bob Briscoe" <[email-masked]>
An: "tsvwg IETF list" <[email-masked]>
Cc: "iccrg IRTF list" <[email-masked]>, "TCP Prague List" <[email-masked]>, "De Schepper, Koen (Koen)" <[email-masked]>
Betreff: [tsvwg] ecn-l4s-id: Proposed Changed to Normative RTT Bias Text
Folks,

The co-authors of ECN L4S ID have been reviewing the correctness of the normative 'Prague' requirements.
    See https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-10#section-4.3
This is the first of 2 emails, about 2 of the requirements that we think ought to be reworded a little.

[...]

       [SM]
CLARIFIED: Any protocol aiming at L4S compliance will need to account for the fact that L4S dedicated reference AQM, the dual queue coupled AQM, inflates the RTT of flows in the deep-queue by 15ms (if the AQM is configured as described in the DualQ internet draft). This will effectively lead to short RTT flows in the shallow queue staving flows in the deep queue. Since DualQ#s coupling heuristic is not designed to undo this damage, it is incumbent on L4S-compliant transport protocols to undo this damage, by artificially slowing down their congestion response "simulating" an RTT of at least that 15ms introduced by DualQ.

RATIONALE: This a) clearly names the root cause of the issue and spells out the remedy in a way that protocol implementers do not need to guess what is implied by "eliminate RTT bias" and why this is not just an generally aspirational goal, but why the 15ms hack is required for L4S to actually be deployable, as the avoidance of congestion collapse for short RTT flows in the deep queue really depends on all L4S protocols implementing that hack faithfully.
       The proposed text also removes any residual mentioning of elimination of RTT bias on first principle, since not even the designated reference protocol TCP Prague actually achieves any meaningful generic RTT de-biasing that is actually applicable in good faith to arbitrary large RTTs. 

I add, that I still consider that hack unacceptable and am of the opinion, that L4S AQM's MUST to be required not to create this situation in the first place. BUT if the WG disagrees and is fine with rolling out a clearly sub-optimal AQM with a-priori known catastrophic failure modes, then at least my proposed text should be added, to unambiguously inform protocol designers and implementers of their responsibility.

Best Regards
         Sebastian

RATIONALE:
1/ "eliminate as much as possible" is stronger than "reduce or eliminate".
2/ This requirement was motivated by 'do no harm to others' relative to existing standard (RFC5681 Reno) congestion control. So there is no need to mandate that an L4S implementer does no harm to themselves, which window-based congestion controls tend to do at higher RTT. 

        [SM] You realise that this is pretty much included in the definition of reducing or eliminating RTT bias, to not do harm to flows of the same protocol at larger RTTs? That requirement, I am willing to bet, was not originally written to save Reno from L4S flows (if it has, it has been exceptionally badly phrased) but really to imply that the "new" way of things is going to be better in all relevant dimensions, more RTT-independent, less re-ordering sensitive, ... This whole rationale makes me somehow think "Oceania had always been at war with Eastasia."
        Then again, this whole wiggling around and post-hoc changing of draft text, basically just confirms my statement, that generic RTT-independence can not be achieved by transport protocols, not without significant involvement of the bottleneck queue's management entity. So again, I am okay with dropping the RTT-independence requirement, as "long as the fix DualQ's 15ms hack" is still added as requirement, unless preferably the AQM design and implementation is fixed and the issue is eliminated at its source.
BUT some chairs seem to indicate that the time for real changes is over and all that is still available are small changes in the text, and so I propose this change.

Of course, this doesn't preclude implementers reducing or eliminating RTT bias for larger than typical RTTs, but it removes any requirement to do so.

Cheers

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 13 Nov 2020 09:13:17 +0100

Dear list,

given the recent data by Pete Heist and Jonathan Morton, it seems that the combination of TCP Prague and DualQ introduce a level of RTT bias not seen before in the internet, including noticeably stronger suppression of long-RTT Prague throughput by short RTT Prague. Under that perspective it is clear that L4S does not at all "reduce or eliminate RTT bias" but makes the situation considerably worse. Allowing that behavior to stand, by changing the requirement to Bob's proposal below is IMHO not acceptable, unless our goal in the WG is to built a fast priority lane for short-RTT flows at the expense of everything else, like long-RTT flows or god-forbid non TCP Prague flows.
	The proposed rationale " there is no need to mandate that an L4S implementer does no harm to themselves" is problematic, if L4S and L4S-compliant transport protocols are intended as replacement for traditional CCs, as we clearly would turn the internet into a mostly short range affair.
I have claimed in the past that RTT bias is an unavoidable consequence of differential "reaction times" for control loops at different delays, so I always was critical of the "reduce or eliminate RTT bias" mandate, but I believe firmly that a "do not make RTT (considerably) bias worse" than it is today should be a non-negotiable MUST requirement for any L4S-compliant transport.

Best Regards
	Sebastian

> On Oct 30, 2020, at 19:42, Bob Briscoe <[email-masked]> wrote:
> 
> Folks,
> 
>       deployment scenario (see 
> Appendix A.1.5
>  for rationale).
> Bob Briscoe                               
> http://bobbriscoe.net/