Subject: [tsvwg] L4S issue #22: CE Ambiguity and Reordering

From: Bob Briscoe <[email-masked]>
Date: Fri, 21 Feb 2020 08:51:38 +0000

Jake, Sebastian,

In yesterday's tsvwg L4S interim, issue #22 was discussed.
RTT dependence has been moved to issue #28.
Classic ECN FIFO AQM concerns are in issue #16.
That leaves issue #22 solely for the remaining issue about CE ambiguity 
mentioned by Jake under #22: reordering.

There was a stand-off in the discussion yesterday as to who is now more 
obliged to produce data about this: the complainant about L4S or the 
defendant of L4S.

As a defendant, let me explain why the WG needs the complainant to give 
more data or more argumentation. In summary, this judgement call was 
made when the WG considered it back in the 2017 timeframe. If you want 
to resurrect the question, we need new data or new argumentation.

I have pointed to the justification for this not being an issue in the 
ecn-l4s-id draft. It is pasted below, and has been broadly unchanged 
since the very first draft (during the process of choosing an 
identifier, we obviously had this concern ourselves - until we took 
advice from the tcpm WG). Jake's posting in the issue tracker says this 
argument relies on RACK deployment. You will see that the argument does 
not rely on RACK deployment, it is merely even less likely to be an 
issue as RACK is deployed.

No-one can produce data for the lack of prevalence of a rare event, at 
least not without enormous cost. So, before we can proceed, if you think 
it might not be rare, we need to know:

  * which of that list of points you dispute, with some evidence or
    argumentation for why it is wrong.
  * Then the WG can discuss whether that threatens the whole argument or
    whether you're just clutching at straws (i.e. does the risk of an
    event with minor impact become worrying when the probability of it
    happening is the combination of 2 middling probabilities and 3 small
    probabilities instead of 5 small ones?).
  * Only then do we need to discuss whether more data is needed to
    resolve a specific dispute and who needs to produce that data.

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

    "Risk of reordering Classic CE packets" in
    Appendix B.1  <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-09#appendix-B.1>  discusses the resulting ambiguity if packets originally
    marked ECT(0) are marked CE by an upstream AQM before they arrive at
    a node that classifies CE as L4S.  It argues that the risk of re-
    ordering is vanishingly small and the consequence of such a low level
    of re-ordering is minimal.

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

If you follow the ref to the appendix, you find the explanation below.

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

    Risk of reordering Classic CE packets:  Classifying all CE packets
       into the L4S queue risks any CE packets that were originally
       ECT(0) being incorrectly classified as L4S.  If there were delay
       in the Classic queue, these incorrectly classified CE packets
       would arrive early, which is a form of reordering.  Reordering can
       cause TCP senders (and senders of similar transports) to
       retransmit spuriously.  However, the risk of spurious
       retransmissions would be extremely low for the following reasons:

       1.  It is quite unusual to experience queuing at more than one
           bottleneck on the same path (the available capacities have to
           be identical).

       2.  In only a subset of these unusual cases would the first
           bottleneck support Classic ECN marking while the second
           supported L4S ECN marking, which would be the only scenario
           where some ECT(0) packets could be CE marked by an AQM
           supporting Classic ECN then the remainder experienced further
           delay through the Classic side of a subsequent L4S DualQ AQM.

       3.  Even then, when a few packets are delivered early, it takes
           very unusual conditions to cause a spurious retransmission, in
           contrast to when some packets are delivered late.  The first
           bottleneck has to apply CE-marks to at least N contiguous
           packets and the second bottleneck has to inject an
           uninterrupted sequence of at least N of these packets between
           two packets earlier in the stream (where N is the reordering
           window that the transport protocol allows before it considers
           a packet is lost).

              For example consider N=3, and consider the sequence of
              packets 100, 101, 102, 103,... and imagine that packets
              150,151,152 from later in the flow are injected as follows:
              100, 150, 151, 101, 152, 102, 103...  If this were late
              reordering, even one packet arriving 50 out of sequence
              would trigger a spurious retransmission, but there is no
              spurious retransmission here, with early reordering,
              because packet 101 moves the cumulative ACK counter forward
              before 3 packets have arrived out of order.  Later, when
              packets 148, 149, 153... arrive, even though there is a
              3-packet hole, there will be no problem, because the
              packets to fill the hole are already in the receive buffer.

       4.  Even with the current TCP recommendation of N=3 [RFC5681  <https://tools.ietf.org/html/rfc5681>]
           spurious retransmissions will be unlikely for all the above
           reasons.  As RACK [I-D.ietf-tcpm-rack  <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-09#ref-I-D.ietf-tcpm-rack>] is becoming widely
           deployed, it tends to adapt its reordering window to a larger
           value of N, which will make the chance of a contiguous
           sequence of N early arrivals vanishingly small.

       5.  Even a run of 2 CE marks within a Classic ECN flow is
           unlikely, given FQ-CoDel is the only known widely deployed AQM
           that supports Classic ECN marking and it takes great care to
           separate out flows and to space any markings evenly along each
           flow.

       It is extremely unlikely that the above set of 5 eventualities
       that are each unusual in themselves would all happen
       simultaneously.  But, even if they did, the consequences would
       hardly be dire: the odd spurious fast retransmission.  Admittedly
       TCP (and similar transports) reduce their congestion window when
       they deem there has been a loss, but even this can be recovered
       once the sender detects that the retransmission was spurious.

/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ 

Before you ask about bullet 1 (multibottleneck path), I described it as 
"quite unusual" because the measurement approach used in the only study 
I could find to support this assertion was rather suspect. The whole 
argument only hangs on the likelihood of experiencing multiple 
bottlenecks on your path being "quite unusual", which is supported by:

  * most paths are from data centres to clients where only the client
    access link is the bottleneck,
  * but still many paths do have 2 access bottlenecks, ...
  * there would be a chance of having the same available bandwidth in
    both access links in two cases:
      o the traffic is alone in both links and the capacities of both
        are the same
      o there is other traffic in either link that temporarily makes the
        available bandwidths the same
        I think it's fair to say this latter case will be transient, so
        I'll only proceed with the question of whether the link
        capacities might be the same...
  * taking residential or mobile access technologies first, the
    bandwidths of some tend to be sold in round numbers, however the
    bandwidth of most access technologies is asymmetric, so the upstream
    round number of one would have to match the downstream round number
    of the other
  * large multi-user access technologies (e.g. DC network access, campus
    network access, corporate network access) will nearly always be
    carrying other traffic, so they fall into the transient case

Taking all this, I think it is reasonable to say multiple bottlenecks 
will happen, but they will be 'quite unusual'. Even if this is not the 
case, the other 4 bullets lead to a very small probability once all is 
considered. And the impact if the event does happen is itself minor anyway.

Note also that bullet 2 makes this solely a transition phenomenon. It 
only becomes a permanent phenomenon if the L4S experiment half succeeds 
so that Classic ECN and L4S ECN permanently co-exist.

Regards

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 24 Feb 2020 15:58:21 +0100

Dear Bob,

more below in-line.

> On Feb 21, 2020, at 09:51, Bob Briscoe <[email-masked]> wrote:
> 
> Jake, Sebastian,
> 

	[SM] I am not sure that #28 is a reasonable place to discuss RTT dependence. #28 it is about dualQ's brokenness in the light of the required degree of isolation. Proposing to wedge the ~15ms under-explained delay (of dualQ's non-LL queue) into TCP Prague's congestion response calculations temporarily might be viable hack to confirm the hypothesis that the unequal between class sharing is a consequence of the coupling method. But nobody really doubted that and the real solution to the problem is IMNHO to put the isolating AQM on solid grounds and not make it depend on end-point compliance and on all L4S-aware protocols adding the same ~15ms hack to their congestion response function calculation. 
	Otherwise please start by showing theoretically and empirically why the 15 ms are a magic value that will always be correct/ideal/optimal/? I have asked about the justification for the non-LL queue's reference delay value of 15ms now multiple times and have not gotten a satisfactory response from Team L4S at all. I would be delighted if you could just refer me to a paper/white paper giving rationale and data supporting 15ms as being superior to say 5ms.

> That leaves issue #22 solely for the remaining issue about CE ambiguity mentioned by Jake under #22: reordering.
> 
> There was a stand-off in the discussion yesterday as to who is now more obliged to produce data about this: the complainant about L4S or the defendant of L4S.

	[SM] Yes, I was quite baffled about that. This is not my call obviously, but I want to mention, that you want something of the IETF here, so whether you like it or not, the onus is on you to convince the IETF that your design is safe. I have been arguing for a long time, that real hard data is a much better argument than theoretical musings, but I seem to not have gotten through to you. So let me repeat, the best way to invalidate my objections is to demonstrate with real data, that the objections are unfounded. Is that tedious? you bet, but if you dislike it, just retract your drafts and the issue goes away ;) 

> As a defendant,

	[SM] IMHO that is a problematic mindset, but it explains a lot. By looking at yourself as the defendant, you will with all means at your disposal defend your existing proposal vigorously, but isn't the goal here is to see whether RFC drafts can be improved by considering and incorporating ideas brought up in the discussion?

> let me explain why the WG needs the complainant to give more data or more argumentation. In summary, this judgement call was made when the WG considered it back in the 2017 timeframe. If you want to resurrect the question, we need new data or new argumentation. 

	[SM] That argument seems backward, as by this rationale the IETF would be bound to all precedent no matter how bad or what was learned in the mean time. For a legal entity that might be a viable position, but for engineers? But humor me, and point to a detailed minutes/video showing that this point has truly been discussed and consensus was reached, preferably by a non-generic hum.

> 
> 
> I have pointed to the justification for this not being an issue in the ecn-l4s-id draft. It is pasted below, and has been broadly unchanged since the very first draft (during the process of choosing an identifier, we obviously had this concern ourselves - until we took advice from the tcpm WG). Jake's posting in the issue tracker says this argument relies on RACK deployment. You will see that the argument does not rely on RACK deployment, it is merely even less likely to be an issue as RACK is deployed.
> 
> No-one can produce data for the lack of prevalence of a rare event, at least not without enormous cost. So, before we can proceed, if you think it might not be rare, we need to know:

	[SM] That is not the data that was requested though. The data requested was to quantify the level of re-ordering that the systemic routing of all CE-marked packets into the LL-queue produces for flows in the non-LL queue. And that seems not to involve enormous costs, given the L4S test-bed you already implemented. Let me put it that way to assess risk you ideally both asses the likelihood of an event as well as the cost, and I am asking for a cost assessment, while you argue that assessing the likelihood is costly and hence you reject the idea of assessing the "cost", what do misunderstand here?

This constant haggling about not having to demonstrate that your claims actually hold water in reality, instead of simply running experiments and arguing based on real data makes me uneasy. Not the least as our discussions about Codel type AQMs demonstrated a rather unexpected lack of understanding of the details on your side. So arguing about real data seems a much better way forward.

> 	• which of that list of points you dispute, with some evidence or argumentation for why it is wrong.
> 	• Then the WG can discuss whether that threatens the whole argument or whether you're just clutching at straws (i.e. does the risk of an event with minor impact

	[SM] Except you have not demonstrated the "minor impact" assertion...

> become worrying when the probability of it happening is the combination of 2 middling probabilities and 3 small probabilities instead of 5 small ones?).

	[SM] Yes, that is going to be a judgement call the ietf needs to make, and it is well possible/probably that the final judgement is going to be to your liking, but that in no way invalidates the request to give real hard data to base the judgement on. To stay in your legal framework, the ietf or the plaintiff  should be allowed meaningful discovery.

> 	• Only then do we need to discuss whether more data is needed to resolve a specific dispute and who needs to produce that data.
> 
> Appendix B.1
>  discusses the resulting ambiguity if packets originally

	[SM] "vanishingly small" indicates that some estimate has been made about the likelihood, but below you claim to have found one paper you also find suspicious. 

	[SM] While I think it is obvious that this is a simplification, let me stipulate this, but also ask you to translate "unusual" into numerical probability, please.

>           delay through the Classic side of a subsequent L4S DualQ AQM.

	[SM] Right now, the ratio of non-L4S AQMs to L4S AQM on the internet is going to be massively biased against L4S, so much that initially the assumption needs to be that the subset in 2) is close to the set in 1).

>           contrast to when some packets are delivered late.

	[SM] I believe we all agree that a few early reorderings  is less severe than a few late. But since the I see no proposal to make L4S cause the second kind of re-ordering I would ask that we drop discussion that as it is a tangent. My request was/is that you run a few reasonably diverse tests to see whether you manage to ever create a 3 dupACK scenario. Combine an aggressive rfc3168 AQM with a more lenient L4S AQM and test with a number of different overload scenarios, if you fail to cause 3 dupACKs I would consider my objection eviscerated. Less talk, more data, please.

>  The first
>       4.  Even with the current TCP recommendation of N=3 [
> RFC5681
> ]
>           reasons.  As RACK [
> I-D.ietf-tcpm-rack
> ] is becoming widely
> Before you ask about bullet 1 (multibottleneck path), I described it as "quite unusual" because the measurement approach used in the only study I could find to support this assertion was rather suspect.

	[SM] So you base your probability assessment on a single study which you do not even believe? Did you try to sanity check /reproduce those results with your own set-up? Because that is a rather strong argument for my request for real data. Or did I misunderstand your argument here?

> The whole argument only hangs on the likelihood of experiencing multiple bottlenecks on your path being "quite unusual", which is supported by:
> 	• most paths are from data centres to clients where only the client access link is the bottleneck, 

	[SM] If that is the only use-case for L4S this should be added to the draft text please. Otherwise I consider this to be irrelevant to this discussion about making L4S safe for the rest of the internet.

> 	• but still many paths do have 2 access bottlenecks, ...
> 	• there would be a chance of having the same available bandwidth in both access links in two cases:
> 		• the traffic is alone in both links and the capacities of both are the same
> 		• there is other traffic in either link that temporarily makes the available bandwidths the same

	[SM] I claim that is not how all of the internet works. With my former ISP during peak hours traffic from direct peering content providers usually was fast, while traffic coming over "transit" links was often slower and had encountered congestion (as seen in increased delay and packet loss). So here I routinely saw flows that had experienced congestion upstream and where mixed with flows that where experiencing congestion through my own link's AQM. Now, I have no indication/data that the "slow" packets actually carried CE marks, but your argument is about prevalence of the dual bottleneck scenario and there I think you are simply overly optimistic.
	So there exist real scenarios in which an access link will be both the bottleneck and will also see flows that already experienced congestion, and there seems no fixed relation between such a congested flow's rate and the rate of other flows. That reality does not seem to match your assumptions. So what shall we do, adjust reality?

> I think it's fair to say this latter case will be transient, so I'll only proceed with the question of whether the link capacities might be the same...
> 	• taking residential or mobile access technologies first, the bandwidths of some tend to be sold in round numbers, however the bandwidth of most access technologies is asymmetric, so the upstream round number of one would have to match the downstream round number of the other
> 	• large multi-user access technologies (e.g. DC network access, campus network access, corporate network access) will nearly always be carrying other traffic, so they fall into the transient case
> Taking all this, I think it is reasonable to say multiple bottlenecks will happen, but they will be 'quite unusual'. Even if this is not the case, the other 4 bullets lead to a very small probability once all is considered. And the impact if the event does happen is itself minor anyway.

	[SM] I find it puzzling that you both argue the impact of re-ordering to be minor and at the same time reject any proposal that you empirically test the impacts potential realistic/worst-case magnitude. If risk = impact * probability, and assessing probability is hard and/or costly, assessing (worst-case) impact could well be cheap and easy and it will still give you a handle on the risk.

> 
> Note also that bullet 2 makes this solely a transition phenomenon. It only becomes a permanent phenomenon if the L4S experiment half succeeds so that Classic ECN and L4S ECN permanently co-exist.

	[SM] If L4S reaches experimental status and will see some roll-out into the real world it will be really hard to put the genie gack into the bottle even with strict contingency plans in place for experimental failure and revocation, and so far I have seen very little in that regards (except from CableLabs for the  docsis specs).

Best Regards
	Sebastian

---

From: "Black, David" <[email-masked]>
Date: Mon, 24 Feb 2020 15:30:07 +0000

Hi Bob,

While this email wasn’t addressed to me, I’ll offer what I hope is a useful response, based on something that I heard in the recorded discussion of this topic after the formal conclusion of the interim meeting.  This note is posted as an individual, e.g., it should not be treated as instructions from a TSVWG chair.

I’ll start by addressing the implied question about what’s wrong with the current Appendix B.1 text in the ecn-l4s-id draft.  That Appendix B.1 is primarily an argument that reordering of CE-marked packets is sufficiently unlikely that it can be safely ignored (itemized points 1-5 are the core of this argument, and are most of the content of Appendix B.1).  While that ordering may be very unlikely, that does not lead to the conclusion that it can be safely ignored – this is because applying the law of large numbers to the overall size/scale/scope of the internet suggests that such reordering is almost certain to happen, somewhere.   Moreover, if that reordering has security implications, e.g., could be used as part of mounting a denial of service attack, then it has to be considered as likely to happen … because the adversary/attacker gets to choose her shots.

The good news starts with something that was mentioned in the discussion after the end of the official part of the interim meeting: Because the reordered packets are CE-marked, a sender congestion response is coming independent of whether the packets are reordered or not.   That point needs to be added to Appendix B.1, as it may be the core of an argument to the effect of: if this unlikely reordering occurs, its effects are limited to a few spurious retransmits and the difference in congestion response to CE vs. loss/drop (in the CE case, ABE backs off somewhat less than the backoff for drop, with credit to Gorry for the reminder about that), which could be argued to be acceptable (I will want to see the detailed text before agreeing).

Thanks, --David

From: tsvwg <[email-masked]> On Behalf Of Bob Briscoe
Sent: Friday, February 21, 2020 3:52 AM
To: Sebastian Moeller; tsvwg IETF list; Holland, Jake
Subject: [tsvwg] L4S issue #22: CE Ambiguity and Reordering

[EXTERNAL EMAIL]
Jake, Sebastian,

As a defendant, let me explain why the WG needs the complainant to give more data or more argumentation. In summary, this judgement call was made when the WG considered it back in the 2017 timeframe. If you want to resurrect the question, we need new data or new argumentation.

  *   which of that list of points you dispute, with some evidence or argumentation for why it is wrong.
  *   Then the WG can discuss whether that threatens the whole argument or whether you're just clutching at straws (i.e. does the risk of an event with minor impact become worrying when the probability of it happening is the combination of 2 middling probabilities and 3 small probabilities instead of 5 small ones?).
  *   Only then do we need to discuss whether more data is needed to resolve a specific dispute and who needs to produce that data.

   Appendix B.1<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-09#appendix-B.1> discusses the resulting ambiguity if packets originally

          be identical).

          a packet is lost).

      4.  Even with the current TCP recommendation of N=3 [RFC5681<https://tools.ietf.org/html/rfc5681>]

          reasons.  As RACK [I-D.ietf-tcpm-rack<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-09#ref-I-D.ietf-tcpm-rack>] is becoming widely

          flow.

Before you ask about bullet 1 (multibottleneck path), I described it as "quite unusual" because the measurement approach used in the only study I could find to support this assertion was rather suspect. The whole argument only hangs on the likelihood of experiencing multiple bottlenecks on your path being "quite unusual", which is supported by:

  *   most paths are from data centres to clients where only the client access link is the bottleneck,
  *   but still many paths do have 2 access bottlenecks, ...
  *   there would be a chance of having the same available bandwidth in both access links in two cases:

     *   the traffic is alone in both links and the capacities of both are the same
     *   there is other traffic in either link that temporarily makes the available bandwidths the same
I think it's fair to say this latter case will be transient, so I'll only proceed with the question of whether the link capacities might be the same...

  *   taking residential or mobile access technologies first, the bandwidths of some tend to be sold in round numbers, however the bandwidth of most access technologies is asymmetric, so the upstream round number of one would have to match the downstream round number of the other
  *   large multi-user access technologies (e.g. DC network access, campus network access, corporate network access) will nearly always be carrying other traffic, so they fall into the transient case
Taking all this, I think it is reasonable to say multiple bottlenecks will happen, but they will be 'quite unusual'. Even if this is not the case, the other 4 bullets lead to a very small probability once all is considered. And the impact if the event does happen is itself minor anyway.

Regards

Bob

--

---

From: "Holland, Jake" <[email-masked]>
Date: Mon, 24 Feb 2020 16:15:19 +0000

Hi Bob,

In issue 22 I asked whether there were any tests demonstrating the scope of the problem, but I did not mean it as a demand to conduct any tests, and did not ask for any during the call.

Sorry if I gave the impression I was asking anyone to perform new tests about reordering.

What I was aiming toward when asking the question originally was that if there’s something that already exists and could be cited that demonstrates the scope of the problem justifying a normative SHOULD for RACK-like behavior, it would be beneficial to include it.  If there is no such source to cite, I didn’t mean to say it’s necessary to create one before moving forward.

To me the reordering issue itself seems so minor that I’m confused why there’s so much text and meeting time discussion about it.  I was worried I was missing something.  If the whole issue is addressed by the appendix text, it seems like even the “SHOULD” for RACK-like behavior from the transport is not justified, since there’s basically no harm even without it?

Regarding the disposition of the overall ticket itself, I have no strong opinions.  I’m not clear on whether anybody else might.  If this ticket is only about dependencies on other tickets, I’m not sure how much value it adds, whereas if it’s about making sure to check consensus and/or text about the overall feasibility of deployment before going to last call, it might be reasonable if it helps anyone remember to do so.

Best regards,
Jake

[1] Issue 22, for easier reference: https://trac.ietf.org/trac/tsvwg/ticket/22

PS: I’ll reiterate my previous request to please treat this discussion as a collaborative attempt to reach a common goal of a better internet, rather than a partisan us-vs-them conflict.  For my own part, the “complainant/defendant” framing with “clutching at straws” comments makes it harder to respond productively.
https://mailarchive.ietf.org/arch/msg/tsvwg/Z6noB1zZmS3_D4g_MqmXpk2gSMY

From: Bob Briscoe <[email-masked]>
Date: Friday, February 21, 2020 at 12:51 AM
To: Sebastian Moeller <[email-masked]>, tsvwg IETF list <[email-masked]>, "Holland, Jake" <[email-masked]>
Subject: [tsvwg] L4S issue #22: CE Ambiguity and Reordering

Jake, Sebastian,

   Appendix B.1<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dtsvwg-2Decn-2Dl4s-2Did-2D09-23appendix-2DB.1&d=DwMCaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=5Lm-6-FmqGOVeBXR4G16wtzc06aYAOpupKs7Mek3GdY&s=FvnYf5WbqWT6hbAVY7X041Tcw4qbo-tmPLA--w7T53E&e=> discusses the resulting ambiguity if packets originally

          be identical).

          a packet is lost).

      4.  Even with the current TCP recommendation of N=3 [RFC5681<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_rfc5681&d=DwMCaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=5Lm-6-FmqGOVeBXR4G16wtzc06aYAOpupKs7Mek3GdY&s=NSwLhh1QRMaM6W2pyzTH2hgO3vq7RD30Yg4AWUyiTRo&e=>]

          reasons.  As RACK [I-D.ietf-tcpm-rack<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dtsvwg-2Decn-2Dl4s-2Did-2D09-23ref-2DI-2DD.ietf-2Dtcpm-2Drack&d=DwMCaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=5Lm-6-FmqGOVeBXR4G16wtzc06aYAOpupKs7Mek3GdY&s=l3jwsKsRU7m140lCG-7oVocV7AdSsfRSXTXx8EQkt3Q&e=>] is becoming widely

          flow.

Regards

Bob

--

Bob Briscoe                               http://bobbriscoe.net/<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMCaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=5Lm-6-FmqGOVeBXR4G16wtzc06aYAOpupKs7Mek3GdY&s=p-tsqybSF_v6EblH3fwyZSVlphv__H1bkuHWfa8VBGc&e=>

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 24 Feb 2020 18:11:31 +0100

> On 24 Feb, 2020, at 5:15 pm, Holland, Jake <jholland=[email-masked]> wrote:
> 

My understanding here is that the root of this issue is that CE-marked packets originating upstream of the DualQ may be classified into a different queue than the rest of their flow, and may therefore be delivered out of order.  DualQ treats CE packets as always belonging to the L4S queue, so this issue affects Classic flows only.

The maximum expected quantity of reordering is easiest to define in time units; the CE packet would be about 0.5ms late if the L4S queue was at its target depth while the Classic queue was empty, while it would be 15ms early if the reverse were true.  The behaviour of a RACK-type transport can be directly predicted based on those values, while predicting the behaviour of a traditional stack may require those times to be converted into packet serialisation times by way of the flow's throughput.

The "RACK requirement" applies only to L4S flows, so I believe it is not relevant to resolving this.  Classic flows will continue to use a mixture of RACK (eg. Linux) and traditional (eg. default FreeBSD at this time) stacks.  It is the latter that are most likely to be affected, so testing (and thought experiments) should focus there.

 - Jonathan Morton

---

From: Luca Muscariello <[email-masked]>
Date: Tue, 25 Feb 2020 12:36:22 +0100

I think issue #22 (https://trac.ietf.org/trac/tsvwg/ticket/22) is more than
just out of order.
It is related to issue #28 and #26.

I did not follow the WebEx recording (not yet shared I think) so I do not
know what
has been discussed during the virtual meeting. Please share the recording.

Out of order is an effect of mis-classification that can happen frequently
in presence of
unresponsive traffic in either the priority or non-priority queues (see
issue #26).
Depending on the rate of unresponsive traffic, not only average rate but
instantaneous rate,
out-of-order events can oscillate frequently. I still have not found any
reference that
proves this does not happen. I tried to raise this point in the past but
it's been ignored
or dismissed by saying that unresponsive traffic is negligible. I disagree.

https://mailarchive.ietf.org/arch/msg/tsvwg/gKFIrBVcNCEZm9wQOIDvyBEi0yM/

There are several potential issues coming from that phenomenon. One in
particular is that (issue #28)
the overall system will not be able to guarantee the latency targets at
stake.

It looks like most of the discussion on this issue is about how this
affects TCP, but the applications
that would mostly benefit from the priority queue are not really using TCP.
So out-of-order/mis-classification for real-time traffic would be worse.

I think that this issue is also related to how much dualQ can provide
predictable performance
under a general traffic mix which constitutes today and future Internet
traffic.

The problem of predictability looks fundamental here, even to deploy an
experiment.
The risk is that we build something to satisfy customers' low latency
request that won't meet the
expectation. Building dualQ has a cost and we better know what we get out
of it before building it
in silicon (issues #22 #26 #28 at least).

Luca

On Mon, Feb 24, 2020 at 6:12 PM Jonathan Morton <[email-masked]>
wrote:

> > What I was aiming toward when asking the question originally was that if
> there’s something that already exists and could be cited that demonstrates
> the scope of the problem justifying a normative SHOULD for RACK-like
> behavior, it would be beneficial to include it.  If there is no such source
> to cite, I didn’t mean to say it’s necessary to create one before moving
> forward.
> >
> > To me the reordering issue itself seems so minor that I’m confused why
> there’s so much text and meeting time discussion about it.  I was worried I
> was missing something.  If the whole issue is addressed by the appendix
> text, it seems like even the “SHOULD” for RACK-like behavior from the
> transport is not justified, since there’s basically no harm even without it?
>
> My understanding here is that the root of this issue is that CE-marked
> packets originating upstream of the DualQ may be classified into a
> different queue than the rest of their flow, and may therefore be delivered
> out of order.  DualQ treats CE packets as always belonging to the L4S
> queue, so this issue affects Classic flows only.
>
> The maximum expected quantity of reordering is easiest to define in time
> units; the CE packet would be about 0.5ms late if the L4S queue was at its
> target depth while the Classic queue was empty, while it would be 15ms
> early if the reverse were true.  The behaviour of a RACK-type transport can
> be directly predicted based on those values, while predicting the behaviour
> of a traditional stack may require those times to be converted into packet
> serialisation times by way of the flow's throughput.
>
> The "RACK requirement" applies only to L4S flows, so I believe it is not
> relevant to resolving this.  Classic flows will continue to use a mixture
> of RACK (eg. Linux) and traditional (eg. default FreeBSD at this time)
> stacks.  It is the latter that are most likely to be affected, so testing
> (and thought experiments) should focus there.
>
>  - Jonathan Morton
>
>

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 26 Feb 2020 00:11:06 +0000

Jonathan,

> My understanding here is that the root of this issue is that CE-marked packets originating upstream of the DualQ may be classified into a different queue than the rest of their flow, and may theref…
Yes.
>
> The maximum expected quantity of reordering is easiest to define in time units; the CE packet would be about 0.5ms late if the L4S queue was at its target depth while the Classic queue was empty,
A CE packet from a Classic flow will never be delivered late.
I think you have not understood how the Coupled DualQ works.
> while it would be 15ms early if the reverse were true.
This part is correct.

This is why the phenomenon can only occur if there are (or have very 
recently been) two bottlenecks on the path:
1) a Classic AQM with a queue sufficient for for it to mark the Classic 
flow with CE
2) a Coupled DualQ with a queue built up in its Classic queue

> The behaviour of a RACK-type transport can be directly predicted based on those values, while predicting the behaviour of a traditional stack may require those times to be converted into packet serialisation times by way of the flow's throughput.
>
Correct.

Bob