Subject: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-08.txt

From: Greg White <[email-masked]>
Date: Fri, 5 Nov 2021 22:57:41 +0000

Sebastian,
As you admitted at the end, we've discussed most of these in the past. You weren't convincing me then, and you aren't convincing me now.  I'll repeat some of the counter-arguments below.
Greg

On 10/26/21, 11:54 AM, "tsvwg on behalf of Sebastian Moeller" <[email-masked] on behalf of [email-masked]> wrote:

    Dear list,

    to be explicit here the new "safety for innocent bystanders" section :

    "4.3.1.  Unmanaged Networks

       In cases where a network operator is delivering traffic into an
       unmanaged network outside of their control (e.g. a residential ISP
       delivering traffic to a customer's home network), the network
       operator should presume that the existing network equipment in the
       user network supports the WiFi default DSCP mapping (see Section 8.3)
       and/or IP Precedence, and does not support the safeguards that are
       provided by the NQB PHB requirements.  As a result, the network
       operator should take precautions to prevent issues.  

    [SM] This ignores the most likely scenario to beginn with an NQB-unaware leaf network that buys internet access from an NQB-unaware ISP. Show me how you can guarantee that no ISP (world-wide) will stay NQB-aware you might have an argument, but can you gurantee that and by what time?

[GW] As I believe I've said before, AFAIK, current consumer-grade WiFi gear does not bleach the DSCP on inbound traffic from the WAN, nor does it provide the option for a consumer to do so.  I believe this is true both for factory firmware and third-party firmware.  So, in effect all residential leaf networks are both NQB-unaware and DiffServ-unaware in general and they will happily grant AC_VO or AC_VI to any traffic that arrives with the right IPP bits on their WAN port from the ISP. The DSCP re-marking feature apparently hasn't been needed because residential leaf networks world-wide have been able to rely on their ISP to "groom" the DSCP on their behalf (bleaching it appears to be the most common method). NQB does not change this.  Since this "problem" extends to all DSCPs, not just NQB, have the retail router vendors and the OpenWRT community "guaranteed" that no ISP (world-wide) will send unbleached traffic into their users' networks?  There must be huge problems with AC_BE traffic being starved by massive amounts of AC_VO and AC_VI.  Surely this would be a high priority (and easy) problem for them to solve, right? You've tried to argue in the past that NQB is different because it is anticipated that ISPs will be able to pass it through to their customer's networks. But, this is only true if A) the ISP is NQB-aware and disables bleaching for the NQB DSCP, and B) they actively re-mark the NQB DSCP (5) to the other value recommended in the draft (45), and C) they disregard the safeguards recommended in the draft.

    When the data
       rate of the access network segment is less than the expected data
       rate of the user network, this is unlikely to be an issue. 

    [SM] As expanded upon below, it is quite hard to estimate which rates will be achievable/expected in a leaf network's WiFi network... but I note it is such WiF networks that require the rate limiting safety mechanism most.

     However,
       if the access network rate exceeds the expected rate of the user
       network, the operator SHOULD deploy a policing function on NQB marked
       traffic that minimizes the potential for negative impacts on traffic
       marked Default, for example by limiting the rate of such traffic to a
       set fraction of the customer's service rate, with excess traffic
       either dropped or re-marked as Default.

    [SM] Which again does not help much, if say my AP-station link only achieves 100Mbps, but my access link is a full gigabit... (both conditions are IMHO not unrealistic).

       As an additional safeguard, and to prevent the inadvertent
       introduction of problematic traffic into unmanaged user networks,
       network equipment that is intended to deliver traffic into unmanaged
       user networks (e.g. an access network gateway for a residential ISP)
       MUST by default ensure that NQB traffic is marked with a DSCP that
       selects the WiFi "Best Effort" Access Category and the "Routine" IP
       Precedence level (i.e.  DSCP 0-7).  Such equipment MUST support the
       ability to configure the remapping, so that (when appropriate
       safeguards are in place) traffic can be delivered as NQB-marked."

    [SM] Sweet, except for the inconvenient fact that almost 100% of deployed network equipment does not do this, and the likelihood of gear being exchanged just to enable NQB seems pretty slim*. I have no objections to hoping for such a replacement strategy, but consider it to be overly optimistic to call something as unlikely to happen a "safeguard"....

[GW] Maybe this is a terminology issue.  "Access network gateway" was intended to mean the CMTS, OLT, DSLAM/BRAS, not the customer equipment.  AFAIK the vast majority of ISP network gear support this, and if it doesn't a router upstream of it almost certainly does.

    In short all is moot, because it requires 
    A) existence of a benevolent NQB-aware operator to gate all access to end user networks

    B) knowledge by the operator about the available bandwidth/rate/capacity in the leaf network.

    which both are not realistic assumptions, sorry.

    B) is especially relevant, because the expected negative side-effects only materialize on WiFi links, but WiFi is a variable rate link technology to beginn with and achievable rates depends on variables like distance between station and AP, RF noise in the selected channel, ... In short there is no way an operator can set an NQB rate that is guaranteed not to cause the unwanted side effects.

    As I stated in the past there is a simple solution to this conundrum, select a DSCP for NQB that by default maps to AC_BE so legacy NQB-unaware leaf networks will continue to work as well or as badly as today, while new gear ca opt-in to NQB's promises and gain its advantages.

    Insisting on using an DSCP that maps by default to AC_VI or AC_VO invalidates all apparent "safety" considerations in the draft as "safety theater"; there is a clear path forward that is safe by design, I fail to see why this draft still insists on proposing a path forward which clearly treats safety as a matter of wording in the RFC text and not in the engineering behind it.

[GW] It is incorrect (and you know it) to state that the draft recommends a DSCP that maps to AC_VO.  You've said this on several occasions and it is a blatant lie. Please stop. The draft recommends the value 45 for end devices.  As you've agreed in the past, that is ok since a) the NQB usage guidance is in-line with existing IETF recommendations for other DSCPs that map to AC_VI and b) there is no way to police what applications do anyway.  The draft recommends that NQB-aware operators re-mark that value to 5 prior to any interconnect, and that they re-mark it to 45 (subject to safeguards) prior to deliver to an end-user's network. As discussed above, NQB-unaware operators will almost certainly bleach it, and if they don't then they presumably have much bigger problems due to the 15 other codepoints that map to AC_VI and the 16 that map to AC_VO.  If the safeguards can't be realistically met, then per the guidance in the draft the DSCP for NQB stays at 5.  I fail to see what your concern is on that front.  

[GW] I'm not sure who is reading these threads besides you and me, so I'm not sure how much effort to put in responding.  I've heard all of your arguments before and you've heard mine. If anyone else is reading this thread and wants to see this debated more, please chime in. 

    Regards
    	Sebastian

    *) As an example when ISPs in Germany introduced Vectoring on their VDSL2 links they did not go and replaced legacy ADSL gear with vectoring-capable or vectoring-friendly devices, but they simply exempted the 2.2 MHz of spectrum used by ADSL from vectoring (in spite of vectoring actually could/should have been an improvement for the ADSL links); I expect the same approach for something much more obscure like NQB.

    P.S.: I have made similar arguments in the past, but as long as new revisions have the same obvious deficit I am going to object to them.

    > On Oct 26, 2021, at 01:05, [email-masked] wrote:
    > 
    > 
    > A New Internet-Draft is available from the on-line Internet-Drafts directories.
    > This draft is a work item of the Transport Area Working Group WG of the IETF.
    > 
    >        Title           : A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services
    >        Authors         : Greg White
    >                          Thomas Fossati
    > 	Filename        : draft-ietf-tsvwg-nqb-08.txt
    > 	Pages           : 20
    > 	Date            : 2021-10-25
    > 
    > Abstract:
    >   This document specifies properties and characteristics of a Non-
    >   Queue-Building Per-Hop Behavior (NQB PHB).  The purpose of this NQB
    >   PHB is to provide a separate queue that enables smooth, low-data-
    >   rate, application-limited traffic flows, which would ordinarily share
    >   a queue with bursty and capacity-seeking traffic, to avoid the
    >   latency, latency variation and loss caused by such traffic.  This PHB
    >   is implemented without prioritization and without rate policing,
    >   making it suitable for environments where the use of either these
    >   features may be restricted.  The NQB PHB has been developed primarily
    >   for use by access network segments, where queuing delays and queuing
    >   loss caused by Queue-Building protocols are manifested, but its use
    >   is not limited to such segments.  In particular, applications to
    >   cable broadband links, Wi-Fi links, and mobile network radio and core
    >   segments are discussed.  This document recommends a specific
    >   Differentiated Services Code Point (DSCP) to identify Non-Queue-
    >   Building flows.
    > 
    > 
    > The IETF datatracker status page for this draft is:
    > https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/
    > 
    > There is also an HTML version available at:
    > https://www.ietf.org/archive/id/draft-ietf-tsvwg-nqb-08.html
    > 
    > A diff from the previous version is available at:
    > https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-nqb-08
    > 
    > 
    > Internet-Drafts are also available by anonymous FTP at:
    > ftp://ftp.ietf.org/internet-drafts/
    > 
    >

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 6 Nov 2021 01:12:51 +0100

Hi Greg,

> On Nov 5, 2021, at 23:57, Greg White <[email-masked]> wrote:
> 
> Sebastian,

[SM2] Thanks.

> [GW] As I believe I've said before, AFAIK, current consumer-grade WiFi gear does not bleach the DSCP on inbound traffic from the WAN, nor does it provide the option for a consumer to do so.  I believe this is true both for factory firmware and third-party firmware.  So, in effect all residential leaf networks are both NQB-unaware and DiffServ-unaware in general and they will happily grant AC_VO or AC_VI to any traffic that arrives with the right IPP bits on their WAN port from the ISP. The DSCP re-marking feature apparently hasn't been needed because residential leaf networks world-wide have been able to rely on their ISP to "groom" the DSCP on their behalf (bleaching it appears to be the most common method). NQB does not change this.

	[SM2] I disagree:

qdisc cake 8032: dev ifb4pppoe-wan root refcnt 2 bandwidth 100Mbit diffserv3 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 34 mpu 88 
 Sent 5304626934 bytes 4251683 pkt (dropped 93351, overlimits 7478639 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 1574016b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       88 /    1526
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh       6250Kbit      100Mbit       25Mbit
  target          5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms
  pk_delay        105us         80us         68us
  av_delay         24us         12us          7us
  sp_delay          2us          3us          4us
  backlog            0b           0b           0b
  pkts            38015      4301629         5390
  bytes        29483573   5412467285       469132
  way_inds           39       120433           11
  way_miss          826        56366          217
  way_cols            0            0            0
  drops              23        93328            0
  marks               0            7            0
  ack_drop            0            0            0
  sp_flows            1            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len          1492         1492          192
  quantum           300         1514          762

here is cake statistics from my wan interface, see how I receive, CS1 (Bulk), as well as at least one of TOS4, VA, EF, CS6, CS7 (Voice)

See https://github.com/dtaht/sch_cake/blob/master/sch_cake.c for which DSCPs are mapped to which of the three "tins"
 *		Low Priority		(CS1)
 *		Best Effort
 *		Latency Sensitive	(TOS4, VA, EF, CS6, CS7)

I perform no DSCP marking on my ingress before traffic hits ifb4pppoe-wan, so this is nicely unbleached markings coming from my ISP (Telefonica/O2 so clearly an established incumbent ISP with loads of experience). Relying on non-tasked entities like ISPs without an SLA to fiddle with DSCP bits is not a robust and reliable solution as evidenced by the data from my ingress. My hypothesis why people do not complain about this is simply that very few end-users will be able to actually diagnose the symptoms of this issue.
What NQB would change however ist that stations in the leaf network will start to mark more aggressively with NQB as well from where they can easily monopolize large fractns of the available airtime.

>  Since this "problem" extends to all DSCPs, not just NQB, have the retail router vendors and the OpenWRT community "guaranteed" that no ISP (world-wide) will send unbleached traffic into their users' networks?  

	[SM] Well, in sqm-scripts (which often is the traffic-shaping/AQM solution of choice on OpenWrt) we leverage cake's ability to re-mark dscps to zero on ingress, because we made the experience that ISPs do not properly bleach DSCPs. I am running my network without that option and accept my ISPs DSCPs mostly because I want to see what reaches my network from the outside. So in a sense we have counter measures in place in OpenWrt for ingress, but these at the same time will also neuter NQB. (Which I am less and less concerned about, but initially i thought this to be a great concept). But that still leaves internal stations using NQB to the AP without any bleaching/or rate limiting help... 

> There must be huge problems with AC_BE traffic being starved by massive amounts of AC_VO and AC_VI.  Surely this would be a high priority (and easy) problem for them to solve, right?

	[SM] Well, I have argued before this is hard to diagnose in the first place...

> You've tried to argue in the past that NQB is different because it is anticipated that ISPs will be able to pass it through to their customer's networks. But, this is only true if A) the ISP is NQB-aware and disables bleaching

	[SM] Except there is no universal reliably bleaching of DSCPs in the wild. See my data above.... one can rely on that ingress DSCPs from an ISP will have no discernable meaning, but not that do not exist...

> for the NQB DSCP, and B) they actively re-mark the NQB DSCP (5) to the other value recommended in the draft (45), and C) they disregard the safeguards recommended in the draft.

	[SM] So the draft currently proposes:
Sender mark (45), NQB-aware operator re-mark (5), -> internet ->	NQB-aware operator remark to (45) if safe for leaf networks		
									NQB-unaware operator leave at (5)

which fails in the following situation:
Sender mark (45), NQB-unaware operator leave at (45),	-> internet ->	NQB-unaware operator leave at (45) -> potentially unsafe for both leaf networks if the endpoints use WiFi.

Compare this to my counter proposal:
Sender mark (5), operator leave at (5). 		-> internet -> 	operator leave at (5), receiver keep at (5)
		, unhelpful operator remark (!5). 	-> internet -> 	operator leave at (!5), receiver keep at (!5)

To gain NQB advantage over WiFi deploy NQB-aware APs that give (5) the desired special treatment (with the required rate limiting) of a secondary AC with the same parameters as AC_BE as proposed in the draft.

One of the two is considerably safer by design, the other one is in the draft...

>       rate of the user network, this is unlikely to be an issue. 

	[SM] Sure, but 1Gbps+ access links becoming more and more common, while WiFi networks that can sustain 940 Mbps goodput (to all stations) are IMHO less common. So WiFi being the bottleneck is mot an unlikely scenario.

> [GW] Maybe this is a terminology issue.  "Access network gateway" was intended to mean the CMTS, OLT, DSLAM/BRAS, not the customer equipment.  AFAIK the vast majority of ISP network gear support th…

	[SM] Ah, yes, I did not get your point. 

> [GW] It is incorrect (and you know it) to state that the draft recommends a DSCP that maps to AC_VO.  

	[SM] But that is not what I said above (or more precisely not what I wanted to convey). I just noted that both AC_VO and AC_VI have this specific issue (and admittedly AC_VO even more than AC_VI).

> You've said this on several occasions and it is a blatant lie. Please stop.

	[SM] ??? Again, I enumerate the problematic AC above and note that any of the two have the same safety issue, I make no claim about which one is currently applicable to the draft's proposed DSCP value. So I disagree that I am lying here, but I clearly was not formulating my point clear enough. 

> The draft recommends the value 45 for end devices.  As you've agreed in the past,

	[SM] Sure because AC_VI is less problematic than AC_VO (which in https://datatracker.ietf.org/doc/html/draft-white-tsvwg-nqb-02 was proposed as re-map target in rfc8325 compliant WiFi networks).

> that is ok since a) the NQB usage guidance is in-line with existing IETF recommendations for other DSCPs that map to AC_VI and b) there is no way to police what applications do anyway.

	[SM] Except there is... re-marking problematic DSCPs to less-problematic ones, or as proposed in the draft itself rate-limiting such packets/flows. And in WiFi an AP could even instruct stations to place any DSCP into any AC the AP wishes, which would not change what an application does but what consequences that action has.

>  The draft recommends that NQB-aware operators re-mark that value to 5 prior to any interconnect, and that they re-mark it to 45 (subject to safeguards) prior to deliver to an end-user's network.

	[SM] Yes, but since the sender should use (45), without that first re-marking to (5) we have an uncontrolled potential AC_VI user in the wild...

> As discussed above, NQB-unaware operators will almost certainly bleach it,

	[SM] That is I believe an incorrect assumption, and I base that on the observed data from my link above. It is possible that my link is an exception, but we had problems with unbleached DSCPs received from ISPs in the past in sqm development (I remember one cable ISP sending enough of CS1 packets to become a problem) that I will not place any stock in "universal" bleaching behavior of ISPs. What is the incentive for them to do so (outside of cleaning the few DSCPs they might use for their own purposes)?

> and if they don't then they presumably have much bigger problems due to the 15 other codepoints that map to AC_VI and the 16 that map to AC_VO.

	[SM] Well, as shown above they do not universally do that... and yes that can be a problem (but in my case the relative number of packets in the three classes make me not loose sleep yet) but as I said in the past it is a problem that just looks like a "bad wifi" day and hard to root cause to excessive use of specific ACs by some stations/applications. 

>  If the safeguards can't be realistically met, then per the guidance in the draft the DSCP for NQB stays at 5.  I fail to see what your concern is on that front.  

	[SM] The point where without a re-marking operator, the NQB DSCP stays at 45... and without such an operator the idea of rate-limiting NQB traffic as a safeguard also becomes unwieldy...

> 
> [GW] I'm not sure who is reading these threads besides you and me, so I'm not sure how much effort to put in responding.  I've heard all of your arguments before and you've heard mine.

	[SM] True, but silence is easily misunderstood as acceptance/agreement.

Sebastian

> If anyone else is reading this thread and wants to see this debated more, please chime in. 
> 
> 
> 
> 
>    Regards
>    	Sebastian
>

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Sat, 6 Nov 2021 08:10:32 -0700 (PDT)

Greg, Sebastian, tsvwg list,

Sorry for the top post, only responding to state *I* am
actively following this thread and it, nor any other thread,
IMHO, should be killed simply because a draft author feels
no one else is interested.   That SHOULD NOT be how the ietf
arrives at decisions, by squelching the messengers.

Regards,
Rod

> > ?On 10/26/21, 11:54 AM, "tsvwg on behalf of Sebastian Moeller" <[email-masked] on behalf of [email-masked]> wrote:
> > 
> >    Dear list,
> > 

-- 
Rod Grimes                                                 [email-masked]

---

From: Greg White <[email-masked]>
Date: Wed, 10 Nov 2021 23:59:04 +0000

On further reflection (and thanks Bob and others for off-list suggestions), here is an alternative:

“If the application's traffic exceeds more than a few packets per RTT, or on an instantaneous (inter-packet) basis exceeds 10% of the global average access link capacity at the time, the application SHOULD NOT mark its traffic with the NQB DSCP.  At the time of writing this document, the global average access link capacity is 63 Mbps down and 13 Mbps up for mobile networks, 113 Mbps down and 62 Mbps up for fixed networks [https://www.speedtest.net/global-index]. For a typical server application, this implies a 6.3 Mbps maximum instantaneous rate.  For a typical client application, this implies a 1.3 Mbps maximum instantaneous rate.”

This keeps the "a few packets per RTT" limit in addition to the rate limit.  It also provides an interpretation of the global average link capacity numbers that presumes that a "typical server" would not be sending traffic "up" an access network link (but a "typical client" might), and that a "typical" server or client app needs to assume that its traffic might transit a mobile access network link.

These rate numbers end up being higher than the limit we'd had before (1 Mbps).  While the "a few packets per RTT" limit would sometimes be the tighter bound, in many cases it isn't.   I wonder if 

Comments?

Along with the above, the other aspect that I'm considering is adding a statement that points out that bottleneck links having a data rate less than 10% of the global average link capacity would *really* be advised to implement traffic protection if they're going to implement NQB, because NQB marked applications would have a greater likelihood of exceeding their link rate.   This would especially be important in bottlenecks that also support L4S dualQ-coupled AQM, and thus would ordinarily classify NQB traffic into the low latency queue.  Since the congestion control used by (NotECT) NQB traffic is not L4S/Prague compliant it would not be expected to be responsive to L4S AQM feedback, and thus could potentially starve L4S applications if Traffic Protection isn't present.

-Greg

On 10/26/21, 9:47 AM, "tsvwg on behalf of Greg White" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Ruediger,

    Thanks for this. I wasn't aware of that rule of thumb.  But, I think I would probably want to be more cautious here and have this limit apply to the instantaneous (peak) rate as opposed to the average rate.  Given that there will be many networks that have bandwidth that is less than the 'global average access link capacity', I think it is warranted.

    So, maybe it's better as:

        "If the application's traffic on an instantaneous (inter-packet) basis exceeds 10% of the global average access link capacity at the time, the application SHOULD NOT mark its traffic with the NQB DSCP.  At the time of writing this document, the global average access link capacity is 63 Mbps down and 13 Mbps up for mobile networks, 113 Mbps down and 62 Mbps up for fixed networks [REF https://www.speedtest.net/global-index or some other reference]."

    -Greg

    On 10/26/21, 1:10 AM, "[email-masked]" <[email-masked]> wrote:

        Hi Greg,

        [RG] - Bob suggests 10% of the access bandwidth as a limit, so to say. A rule of thumb says the resource demand for flows below 10% interface bandwidth can be characterized by their average bandwidth, whereas above 10%, the peak bandwidth better characterizes the flows required resources. So above this boundary burstiness of such a flow impacts other flows (and therefore peak bandwidth should be used to define the required resources).

        To me the suggestion of Bob makes sense. In my recollection, I've read this rule of thumb in a publication of Gunnar Karlsson, a researcher who's been working on measurement based admission control.

        Regards,

        Ruediger

         ######### you wrote ###########  

        In looking back at the mailing list just now, I realized that I did not address Bob Briscoe's comment: 
        https://mailarchive.ietf.org/arch/msg/tsvwg/ntGanYqcGP8GqlgzCPSQ1W6rg1U/

        Apologies for that. 

        His suggestion was to replace:

        "If the application's traffic exceeds more than a few packets per RTT, or exceeds approximately 1 Mbps on an instantaneous (inter-packet) basis, the application SHOULD NOT mark its traffic with the NQB DSCP."

        With something along the lines of:

        "If the application's traffic exceeds a single digit percentage of the global average access link capacity at the time, the application SHOULD NOT mark its traffic with the NQB DSCP.  At the time of writing this document, the global average access link capacity is 63 Mbps down and 13 Mbps up for mobile networks, 113 Mbps down and 62 Mbps up for fixed networks [REF https://www.speedtest.net/global-index or some other reference]."

        His rationale can be found in the mailing list posting linked above.

        I'm interested in opinions on this, as well as suggestions on the wording.

        -Greg

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 11 Nov 2021 10:11:41 +0100

Dear list,

more below, prefixed [SM], I curated the CC list as all are subscribed, and discussion should happen on-list as much as possible IMHO.

> On Nov 11, 2021, at 00:59, Greg White <[email-masked]> wrote:
> 
> Along with the above, the other aspect that I'm considering is adding a statement that points out that bottleneck links having a data rate less than 10% of the global average link capacity would *really* be advised to implement traffic protection if they're going to implement NQB, because NQB marked applications would have a greater likelihood of exceeding their link rate.  

	[SM] I want to add a number of potential issues here, which I would love to discuss (or have "shot down" with explicit arguments (please) if deemed unreasonable):

A) As the proponents of L4S (and others) have convincingly argued the average is rarely an ideal measure to describe many real world-distributions, e.g. for latency-under-load the higher percentiles seem more informative about the to be expected quality degradation users will perceive than either the median or the average. I think the same logic applies to end-user internet access rates (and RTTs from end-users to the "internet" for that matter); maybe we should aim to also care for the unfortunate users well-below the global average.

B) I believe by selecting an DSCP value for NQB that by default maps to (higher priority than BE) AC_VI, essentially all existing WMM-APs/stations will be implementing some form of (special) NQB (treatment), but unbeknownst to their operators. Even for users with wired-access rates in excess of the global average, WiFi rate can and does drop below these access rates and that will effectively make the WiFi link the true bottleneck link (for some of the time). So this seems exactly the situation where you intend to advise the user to implement queue protection.

Question: 
1) How do you plan to actually get this information to the operators that would/should need to act on it? 
2) And what remedies do you propose for those operators to take (ideally not requiring to replace hardware/software, but by simple configuration possible with the existing tools)?*

> This would especially be important in bottlenecks that also support L4S dualQ-coupled AQM, and thus would ordinarily classify NQB traffic into the low latency queue.

	[SM] Question: as far as I understand the question non non-L4S flows to classify into the L-queue is left to operators (NQB is listed as an example):
A) Why do you assume NQB to ordinarily end up in the L-queue? 
B) And why does this matter for the concern of unresponsive NQB traffic (see below)?

>  Since the congestion control used by (NotECT) NQB traffic is not L4S/Prague compliant it would not be expected to be responsive to L4S AQM feedback, and thus could potentially starve L4S applications if Traffic Protection isn't present.

	[SM] If a packet does not carry ECT(1) but ends up in the L-queue (by operator choice) an L4S- AQM surely will resort to dropping, no? So L4S AQM feedback in this situation should be similar/identical to non-L4S AQM (or overloaded FIFO) feedback in that situation. If the NQB traffic is truly unresponsive it will do harm independent of AQM by crowding out more responsive traffic, no?
	So does it appear less of a problem to potentially starve/severely suppress non-L4S traffic? I venture a guess that users whose flows get severely depressed will be unhappy independent whether those flows used ECT(1)-marking or not. The relative importance of flows is really hard/impossible to judge from the information available to an intermediary AQM hop...**

Sebastian

*) I think it is established that:
A) Not all APs/stations offer easy ways to change the EDCA parameters for the different ACs or add additional ACs (e.g. OpenWrt does currently not offer that capability short of using a massively bespoke self-built firmware image, nor do the Speedport or Fritzbox WiFi-routers popular in the German market). Also, searching the manuals of all four of Germany's largest DOCSIS ISP (Vodafone) current WiFi-Modem-routers for EDCA, and WMM came up empty. While I do not have precise numbers, I assume that Telekom's Speedport's, Vodafone's Vodafone Stations and free-market AVM Fritzbox's together make up a considerable fraction (if not majority) of WiFi routers in the one country where I can speak from recent first hand experience. My conclusion is that modifying EDCA-parameters to make NQB traffic less dangerous is not an option available to many existing APs.

B) No router I have seen (I do not claim tp have seen more than a small set) offers explicit queue protection for NQB, and even OpenWrt routers with airtime fairness and fq_codel in the WiFi stack, will still give AC_VI the lion share of airtime/througput when in direct competition with AC_BE (based on testing on my own OpenWrt AP, so anecdotal, but based on real data). And again, many existing wifi routers will not even offer airtime fairness and/or wifi-fq.

C) No home-router I have seen (see above) offers DSCP re-marking to 0 on ingress, with the exception of OpenWrt, either with custom firewall rules or using SQM which offers DSCP "washing" as option. This indicates that even purposefully "neutering" NQBs on ingress to NQB incompatible networks is not going to be an option for many existing APs.

D) As posted in one of my last posts, currently (NQB-unaware) ISPs do not reliable re-map the DSCP field of packets sent to their users/customers to 0, and as shown by the traversal studys (e.g. https://www.simulamet.no/sites/default/files/publications/files/itc30.pdf), this lack of robust and reliable re-marking also seems true for the "inner" part of the internet (servers to data centers if I recall correctly). So we can not assume that NQB will only map to AC_VI if networks take active steps to make that happen. Nor can we assume that all ISPs of NQB marking end-points will be using access networks that know and follow the recommendation "networks that support NQB SHOULD remap NQB traffic (DSCP 45) to DSCP 5 prior to interconnection". Which to me indicates that NQB-unaware WiFi networks can and will encounter uncontrolled NQB-DSCP in the real internet, so assuring safe operation of NQB under these conditions seems a reasonable engineering goal. 

I you disagree with any of these points, I will listen, if you have references I would appreciate that a lot.

IMHO, there is a simple and elegant solution to NQB's WiFi woes, simply standardize on the arguably more internet-passable DSCP5 and recommend that NQB-aware networks re-map DSCP5 to AC_VI or a new AC_NQB. Ideally this mapping can be relegated to new NQB-aware APs, that then can easily implement the "NQB should not starve/severely depress other traffic" recommendation. Yes, this trades in ease of use/convenience and deployment incentives, but IMHO it does so while gaining much more in deployment safety than it looses in incentives. I would love to hear others' arguments against or for such a WiFi-side-ffect free DSCP recommendation.

**) This is IMHO the single strongest argument for per-flow queueing, while certainly never optimal (lacking reliable information about the optimal throughput per flow***), it does allow progress for every flow and will almost certainly never be pessimal (say by starving/throttling the most important flows excessively), and does not require any additional information about a flow's relative importance (as we need to assume that any such information, that is not at the same time essential for the flow reaching its target, can and will be gamed/abused). Using functionally essential information (like 5-tuples) flow queuing is hard to game at least for traffic that is intended to reach a destination.
	The one big failure mode of flow queueing, the lack of resilience against "sharding" (using multiple parallel flows between identical endpoints to gain a throughput advantage over other transfers), is also a failure mode for most other alternatives as TCP/TCP-friendly traffic is designed/expected to roughly converge on per-flow fair sharing of capacity.

***) Not sure whether on a massively multiplexing high speed link a single accepted optimal capacity split actually exists or rather can be discerned, given the likelihood of traffic of different users with different objectives and optimality criteria.

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Nov 2021 11:32:10 +0000

Greg,

> This keeps the "a few packets per RTT" limit in addition to the rate limit.  It also provides an interpretation of the global average link capacity numbers that presumes that a "typical server" wou…

[BB] I wonder if the process of getting this agreed would be easier if 
you said x% rather than 10%, and explained that you expect the WG to 
agree what x% should be, but this wording at least gives the form of 
words that is needed, but not the precise number (yet). x%=10% can be a 
strawman, but we can then determine whether anyone disagrees with even 
this form of constraint, separately from disagreeing with the number.

> These rate numbers end up being higher than the limit we'd had before (1 Mbps).  While the "a few packets per RTT" limit would sometimes be the tighter bound, in many cases it isn't.   I wonder if

[BB] I'll finish your sentence by saying how I would have finished it... 
I wonder if it would be useful to add an example where the RTT limit is 
a tighter bound.

> Along with the above, the other aspect that I'm considering is adding a statement that points out that bottleneck links having a data rate less than 10% of the global average link capacity would *r…

[BB] If there is >1 NQB app or >1 flow from that app at a time, it could 
still hit a link with capacity above 10% of the average, for which 
traffic protection is not as strongly advised. But I guess  you repeat 
the advice that traffic protection is probably needed more generally for 
NQB than just for low rate links (sorry I don't recall whether the draft 
already does?).

Are you talking here about per-flow traffic protection, or per-NQB-class?

Also, we need to be careful here not to give the impression that this is 
all about (instantaneous) rate, relative to a global average. It's only 
cast in those terms for NQB senders because they don't know the 
available rate at the bottleneck. But, once we start talking about 
protection at the link itself, it knows when actual queuing delay is 
occurring. So traffic protection ought to be conditioned on that, not 
just (instantaneous) rate.

Otherwise, on the flip side, per-NQB class protection based solely on 
rate would potentially ruin the NQB application(s) even if they were 
doing no harm - e.g. even if there is no other traffic in the link.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Greg White <[email-masked]>
Date: Thu, 11 Nov 2021 18:12:41 +0000

Thanks Bob,

On 11/11/21, 4:32 AM, "Bob Briscoe" <[email-masked]> wrote:

    Greg,

[GW] Yes, it occurred to me that 10% may or may not be the right value, which is what I was "wondering" when my typing trailed off previously ....

[GW] Ok, that isn't too hard, I could add that.

[GW] The draft already recommends Traffic Protection generally. What I was considering was a stronger statement in the case of low rate links (and in particular those that also implement L4S in a shared dualQ).

[GW] Per-flow.

[GW] There is only fairly high-level guidance given in the draft at the moment, with an example reference.   I can add some statements to address your concern. 

[GW] Agreed, the language around Traffic Protection refers specifically to identifying "flows" that are mismarked.  It doesn't mention any sort of per-class operation.  In fact there is an overall requirement (traffic protection aside) that says aggregate NQB traffic SHOULD NOT be rate limited or rate policed separately from the QB traffic.

    Bob

---

From: Martin Duke <[email-masked]>
Date: Fri, 19 Nov 2021 09:09:02 -0800

On Fri, Nov 5, 2021 at 3:58 PM Greg White <[email-masked]> wrote:

>
> [GW] It is incorrect (and you know it) to state that the draft recommends
> a DSCP that maps to AC_VO.  You've said this on several occasions and it is
> a blatant lie. Please stop. The draft recommends the value 45 for end
> devices.  As you've agreed in the past, that is ok since a) the NQB usage
> guidance is in-line with existing IETF recommendations for other DSCPs that
> map to AC_VI
>

Greg, I know it is frustrating to repeat yourself. Please make every effort
to give your correspondent the benefit of the doubt -- in this case,
Sebastian seems to have not articulated himself clearly enough.