Subject: [tsvwg] Traffic protection as a hard requirement for NQB (was: WG adoption of draft-white-tsvwg-nqb!)

From: Bob Briscoe <[email-masked]>
Date: Mon, 2 Sep 2019 15:47:04 +0100

David,

Thanks for your closing remarks on the NQB adoption call.
You say your last point is open for discussion, so I will dive straight 
in to start that discussion.

On 30/08/2019 16:40, Black, David wrote:
>
>  1. [snip]
>  2. The criticisms on this list of the “queue protection” requirement
>     in the draft are largely accurate.   The draft needs at least an
>     Editor’s Note that this material will be revised, as while the
>     DOCSIS mechanism is an example of how to do queue protection, it
>     is not appropriate to require implementation of that mechanism.  
>     A plausible plan that I have discussed with the authors is to
>     write a set of functional/behavioral requirements for NQB “traffic
>     protection” that can be satisfied by a “queue protection”
>     mechanism such as the DOCSIS mechanism, or by a suitably
>     configured FQ AQM implementation. [snip]
>
> In addition, related to item 2), my expectation (which is open to 
> further discussion) that “traffic protection” will be a “MUST” 
> requirement, perhaps with some well-specified exceptions (including 
> explanations of why the exceptions are ok).   This is because “traffic 
> protection” (e.g., “queue protection” or a suitably configured FQ AQM) 
> appears to be necessary in general to keep queue-building traffic out 
> of the NQB traffic aggregate, as allowing such traffic degrades the 
> properties of the NQB PHB.
>
I think we should be wary of making traffic protection a hard 
requirement at such an early stage in our knowledge of the NQB 
behaviour. I believe this is a case where the market, not the IETF, 
ought to decide whether protection is required.

The draft claims incentives can be aligned by an implementation being 
arranged to ensure that NQB traffic benefits from NQB marking and QB 
traffic benefits from QB marking. Incentives are hard to guess, so that 
may or may not be true. However, I don't think we (the tsvwg/IETF) can 
state categorically that it is not true.

The draft makes the point that, even if incentives are aligned, 
queue-building traffic could be mismarked as NQB, either accidentally or 
maliciously. That's a sound reason for an implementer to include traffic 
protection, but I don't think it's a good reason for us (the tsvwg/IETF) 
to require them to.

While there is no operational experience of NQB deployments, I think the 
market (i.e. most early-adopting operators) will want the warm feeling 
of some form of traffic protection. But as we get more experience we 
might find incentives really are aligned. And we might find accidents 
and malice are not a significant problem.

So I think the current 'SHOULD' is the right call. It could be beefed up 
with warnings on the risks of not providing protection - not least the 
risk no early adopter will want to use such an implementation.

As an analogy, when TCP congestion control was first developed, it was 
known that end-systems could run a subverted TCP algorithm or just use 
unresponsive UDP. At that time, the view could have been taken that 
per-flow scheduling would have to be a 'MUST' requirement for all 
Internet bottlenecks. As it has turned out, the Internet does have 
/per-user/ scheduling at bottlenecks, but there has been little need for 
/per-flow/ scheduling for capacity sharing (yes, FQ exists, but it's not 
needed for capacity sharing).

In the TCP case, it turned out that a delicate balance of incentives 
proved sufficient to allow most Internet equipment to be simpler and 
cheaper. There is a poorly understood balance of incentives in the NQB 
case. So let's not require equipment to be more complex than it might 
need to be, at least not yet.

Bob

> Thanks, --David (TSVWG co-chair, will be shepherd for NQB draft).
>
> ----------------------------------------------------------------
>
> David L. Black, Senior Distinguished Engineer
>
> Dell EMC, 176 South St., Hopkinton, MA 01748
>
> +1 (774) 350-9323 *New *   Mobile: +1 (978) 394-7754
>
> [email-masked] <mailto:[email-masked]>
>
> ----------------------------------------------------------------
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 3 Sep 2019 11:14:35 +0200

Dear Bob,

allow me to chime in.

> On Sep 2, 2019, at 16:47, Bob Briscoe <[email-masked]> wrote:
> 
> David,
> 
> You say your last point is open for discussion, so I will dive straight in to start that discussion.
> 
>> 	• [snip]
>> 	• The criticisms on this list of the “queue protection” requirement in the draft are largely accurate.   The draft needs at least an Editor’s Note that this material will be revised, as while the DOCSIS mechanism is an example of how to do queue protection, it is not appropriate to require implementation of that mechanism.   A plausible plan that I have discussed with the authors is to write a set of functional/behavioral requirements for NQB “traffic protection” that can be satisfied by a “queue protection” mechanism such as the DOCSIS mechanism, or by a suitably configured FQ AQM implementation. [snip]
>>  
>> In addition, related to item 2), my expectation (which is open to further discussion) that “traffic protection” will be a “MUST” requirement, perhaps with some well-specified exceptions (including explanations of why the exceptions are ok).   This is because “traffic protection” (e.g., “queue protection” or a suitably configured FQ AQM) appears to be necessary in general to keep queue-building traffic out of the NQB traffic aggregate, as allowing such traffic degrades the properties of the NQB PHB.
> I think we should be wary of making traffic protection a hard requirement at such an early stage in our knowledge of the NQB behaviour. I believe this is a case where the market, not the IETF, ought to decide whether protection is required. 

	[SM] The draft says "... it is worthwhile to note that the NQB designation and marking would be intended to convey verifiable traffic behavior, not needs or wants." in the light of this requirement it seems obvious that a hop willing to honor that DSCP should/must actually verify the traffic behavior, no? Requiring behavior to according to a set of requirements but not enforcing these requirements seems very very optimistic.

> 
> The draft claims incentives can be aligned by an implementation being arranged to ensure that NQB traffic benefits from NQB marking and QB traffic benefits from QB marking. Incentives are hard to guess, so that may or may not be true. However, I don't think we (the tsvwg/IETF) can state categorically that it is not true.

	[SM] I would have thought it would be on those claiming a laxer enforcement model to proof that that model is sufficient?

> 
> The draft makes the point that, even if incentives are aligned, queue-building traffic could be mismarked as NQB, either accidentally or maliciously. That's a sound reason for an implementer to include traffic protection, but I don't think it's a good reason for us (the tsvwg/IETF) to require them to.
> 
> While there is no operational experience of NQB deployments, I think the market (i.e. most early-adopting operators) will want the warm feeling of some form of traffic protection. But as we get more experience we might find incentives really are aligned. And we might find accidents and malice are not a significant problem.

	[SM] @ the chairs, how hard would it be to retroactively change from a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be nice, but if it is hard I would vote for a MUST as that seems to be the safer option.

> 
> So I think the current 'SHOULD' is the right call. It could be beefed up with warnings on the risks of not providing protection - not least the risk no early adopter will want to use such an implementation. 
> 
> 
> 
> As an analogy, when TCP congestion control was first developed, it was known that end-systems could run a subverted TCP algorithm or just use unresponsive UDP. At that time, the view could have been taken that per-flow scheduling would have to be a 'MUST' requirement for all Internet bottlenecks.
> As it has turned out, the Internet does have /per-user/ scheduling at bottlenecks,

	[SM] Question is this universally true? I am not 100% sure. My access link is limited by my contracted rate, but due to oversubscription there is no guarantee that on a congested link between my home and my ISP's traffic-shaper/the wider internet I get a share that reflects my "per-user" fraction of the shared capacity. 
	I fail to see any scheduler beyond my ISPs traffic shaper that has a wholistic-enough view to classify traffic based on "user" (think NATed IPv4 address, but variable length IPv6 prefixes, plus a router's ipv6/64), could you elaborate please? 

> but there has been little need for /per-flow/ scheduling for capacity sharing (yes, FQ exists, but it's not needed for capacity sharing).
> 
> In the TCP case, it turned out that a delicate balance of incentives proved sufficient to allow most Internet equipment to be simpler and cheaper. There is a poorly understood balance of incentives in the NQB case. So let's not require equipment to be more complex than it might need to be, at least not yet.

	[SM] I believe for malicious actors NQB will be a attractive DSCP as it promised to allow doing harm even at low bandwidth (just send a low average rate in lumpy bursts and I would expect an NQB-honoring L4S scheduler to get into trouble*, without requiring a large offensive traffic load, keeping it cheap and hard to detect yet sufficiently disruptive to rob the low latency queue of its intended functionality).

*) This is a hypothesis I have not confirmed in any way, but it seems to be in line with how I understand dual queue aqm to work (so it might be more of a reflection of my level of understanding and not so much of dual queue aqm). Please correct me if this seems wrong.

> u
>> +1 (774) 350-9323 New    Mobile: +1 (978) 394-7754
>> [email-masked]

---

From: "Black, David" <[email-masked]>
Date: Tue, 3 Sep 2019 21:46:25 +0000

[Writing as an individual]

The core of my current views is this:

> ... “traffic protection” (e.g., “queue
> protection” or a suitably configured FQ AQM) appears to be necessary in
> general to keep queue-building traffic out of the NQB traffic aggregate, as
> allowing such traffic degrades the properties of the NQB PHB.

I see serious potential here for a "tragedy of the commons" outcome where lots of QB traffic is sent marked NQB because the resulting latency is (at least initially) lower.

> > I think we should be wary of making traffic protection a hard requirement
> at such an early stage in our knowledge of the NQB behaviour. I believe this
> is a case where the market, not the IETF, ought to decide whether protection
> is required.

I want to see clear and convincing evidence that the "tragedy of the commons" outcome is highly unlikely, which this text ...

> traffic benefits from QB marking. Incentives are hard to guess, so that may or
> may not be true. However, I don't think we (the tsvwg/IETF) can state
> categorically that it is not true.

... is not.

The above text not only reverses the burden of proof but also changes the question from whether a "tragedy of the commons" is possible to whether it will always (categorically) happen - the resulting straw proposition is then easy to demolish.  Nice try, no sale ... or as Sebastian writes:

> 	[SM] I would have thought it would be on those claiming a laxer
> enforcement model to proof that that model is sufficient?

I agree.  As for requirements language ...

> market (i.e. most early-adopting operators) will want the warm feeling of
> some form of traffic protection. But as we get more experience we might
> find incentives really are aligned. And we might find accidents and malice are
> not a significant problem.
> 
> 	[SM] @ the chairs, how hard would it be to retroactively change from
> a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be
> nice, but if it is hard I would vote for a MUST as that seems to be the safer
> option.

... a common IETF approach to this sort of uncertain situation is "MUST implement, SHOULD use" with discussion of when the "SHOULD" may not apply and/or possible consequences of ignoring the "SHOULD."  The underlying rationale is that the functionality will be available for use if/when the need for it is discovered in a fashion that surprises the network operator.  It is much easier to relax "MUST implement" based on implementation experience than it is to upgrade "SHOULD implement" as the former doesn't have the potential to break any "running code" implementations.

> unresponsive UDP. At that time, the view could have been taken that per-
> flow scheduling would have to be a 'MUST' requirement for all Internet
> bottlenecks.   As it has turned out, the Internet does have /per-user/
> scheduling at bottlenecks,

I understand the TCP congestion control history, and the even-more-relevant ECN for TCP history, where there was a serious potential for a "tragedy of the commons" outcome - ignoring ECN congestion indications was a potential new way to cause congestion collapse of the Internet (fortunately, that did not happen).  There are at least a couple of aspects that seem rather different in the current situation:
- At the time, the IETF community had strong influence over the important TCP stacks. 
	The network stacks have gotten much more diverse since then, and a lot of NQB traffic
	can be expected to come from applications that have a much weaker relationship to the IETF community.
- Switching away from TCP to UDP was a high hurdle, e.g., as TCP uses stream sockets, whereas UDP uses datagram sockets, and ECN functionality was in the core of TCP implementations.
	The hurdle here is much lower, particularly for applications that already use UDP.

> > In the TCP case, it turned out that a delicate balance of incentives proved
> sufficient to allow most Internet equipment to be simpler and cheaper.
> There is a poorly understood balance of incentives in the NQB case. So let's
> not require equipment to be more complex than it might need to be, at least
> not yet.

I concur with Sebastian that the onus is on the proposers of the complexity/cost savings to demonstrate that the resulting designs are safe for the Internet.  I'm reminded by analogy of something that my software engineering professor said back when I was in grad school - "I can make the code run arbitrarily fast if it doesn't have to be correct." [Mary Shaw]

Thanks, --David

> -----Original Message-----
> From: Sebastian Moeller <[email-masked]>
> Sent: Tuesday, September 3, 2019 5:15 AM
> To: Bob Briscoe
> Cc: Black, David; [email-masked]
> Subject: Re: [tsvwg] Traffic protection as a hard requirement for NQB (was:
> WG adoption of draft-white-tsvwg-nqb!)
> 
> 
> [EXTERNAL EMAIL]
> 
> Dear Bob,
> 
> allow me to chime in.
> 
> >> 	• The criticisms on this list of the “queue protection” requirement in
> the draft are largely accurate.   The draft needs at least an Editor’s Note that
> this material will be revised, as while the DOCSIS mechanism is an example of
> how to do queue protection, it is not appropriate to require implementation
> of that mechanism.   A plausible plan that I have discussed with the authors is
> to write a set of functional/behavioral requirements for NQB “traffic
> protection” that can be satisfied by a “queue protection” mechanism such as
> the DOCSIS mechanism, or by a suitably configured FQ AQM implementation.
> [snip]
> >>
> >> In addition, related to item 2), my expectation (which is open to further
> discussion) that “traffic protection” will be a “MUST” requirement, perhaps
> with some well-specified exceptions (including explanations of why the
> exceptions are ok).   This is because “traffic protection” (e.g., “queue
> protection” or a suitably configured FQ AQM) appears to be necessary in
> general to keep queue-building traffic out of the NQB traffic aggregate, as
> allowing such traffic degrades the properties of the NQB PHB.
> > I think we should be wary of making traffic protection a hard requirement
> at such an early stage in our knowledge of the NQB behaviour. I believe this
> is a case where the market, not the IETF, ought to decide whether protection
> is required.
> 
> 	[SM] The draft says "... it is worthwhile to note that the NQB
> designation and marking would be intended to convey verifiable traffic
> behavior, not needs or wants." in the light of this requirement it seems
> obvious that a hop willing to honor that DSCP should/must actually verify the
> traffic behavior, no? Requiring behavior to according to a set of requirements
> but not enforcing these requirements seems very very optimistic.
> 
> 
> >
> traffic benefits from QB marking. Incentives are hard to guess, so that may or
> may not be true. However, I don't think we (the tsvwg/IETF) can state
> categorically that it is not true.
> 
> 	[SM] I would have thought it would be on those claiming a laxer
> enforcement model to proof that that model is sufficient?
> 
> >
> > The draft makes the point that, even if incentives are aligned, queue-
> building traffic could be mismarked as NQB, either accidentally or maliciously.
> That's a sound reason for an implementer to include traffic protection, but I
> don't think it's a good reason for us (the tsvwg/IETF) to require them to.
> >
> market (i.e. most early-adopting operators) will want the warm feeling of
> some form of traffic protection. But as we get more experience we might
> find incentives really are aligned. And we might find accidents and malice are
> not a significant problem.
> 
> 	[SM] @ the chairs, how hard would it be to retroactively change from
> a SHOULD to  a MUST? If it is easy I agree with Bob that a SHOULD would be
> nice, but if it is hard I would vote for a MUST as that seems to be the safer
> option.
> 
> >
> > So I think the current 'SHOULD' is the right call. It could be beefed up with
> warnings on the risks of not providing protection - not least the risk no early
> adopter will want to use such an implementation.
> >
> >
> >
> unresponsive UDP. At that time, the view could have been taken that per-
> flow scheduling would have to be a 'MUST' requirement for all Internet
> bottlenecks.
> 	[SM] Question is this universally true? I am not 100% sure. My access
> link is limited by my contracted rate, but due to oversubscription there is no
> guarantee that on a congested link between my home and my ISP's traffic-
> shaper/the wider internet I get a share that reflects my "per-user" fraction of
> the shared capacity.
> 	I fail to see any scheduler beyond my ISPs traffic shaper that has a
> wholistic-enough view to classify traffic based on "user" (think NATed IPv4
> address, but variable length IPv6 prefixes, plus a router's ipv6/64), could you
> elaborate please?
> 
> > but there has been little need for /per-flow/ scheduling for capacity sharing
> (yes, FQ exists, but it's not needed for capacity sharing).
> >
> > In the TCP case, it turned out that a delicate balance of incentives proved
> sufficient to allow most Internet equipment to be simpler and cheaper.
> There is a poorly understood balance of incentives in the NQB case. So let's
> not require equipment to be more complex than it might need to be, at least
> not yet.
> 
> 	[SM] I believe for malicious actors NQB will be a attractive DSCP as it
> promised to allow doing harm even at low bandwidth (just send a low
> average rate in lumpy bursts and I would expect an NQB-honoring L4S
> scheduler to get into trouble*, without requiring a large offensive traffic
> load, keeping it cheap and hard to detect yet sufficiently disruptive to rob the
> low latency queue of its intended functionality).
> 
> 
> *) This is a hypothesis I have not confirmed in any way, but it seems to be in
> line with how I understand dual queue aqm to work (so it might be more of a
> reflection of my level of understanding and not so much of dual queue aqm).
> Please correct me if this seems wrong.
> 
> >
> > u
> >
> > Bob
> >
> >
> >>

---

From: Greg White <[email-masked]>
Date: Wed, 4 Sep 2019 23:01:47 +0000

Thanks David.

On your item 3, could you clarify what you mean?   RFC 8325 does not discuss a “non queue building” service class, and so I don’t see a technical inconsistency between the NQB draft and RFC8325.  There has been some discussion on the list as to whether IETF should recommend that NQB be mapped to UP_6/AC_VO or UP_5/AC_VI, but in at least one of those cases where it was suggested to change the draft, the rationale for doing so was based on an incorrect assumption that NQB-marked traffic would by definition include capacity-seeking flows (e.g. L4S).  As I stated on list, I don’t have a problem with changing it to UP_5/AC_VI if that is the consensus.  But, the reason for mapping NQB to UP_6/AC_VO is so that both RFC8325 and “default mapping” 802.11 devices map EF, VA and NQB together to the same UP/AC, and so that both types of devices map these traffic types one AC level “higher” than elastic traffic sources (e.g. AF3x).  I’d like to see a more reasoned argument as to why NQB should be grouped with AF3x traffic as opposed to EF & VA before making a change.

-Greg

From: tsvwg <[email-masked]> on behalf of David Black <[email-masked]>
Date: Friday, August 30, 2019 at 9:41 AM
To: "[email-masked]" <[email-masked]>
Subject: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

In the Montreal TSVWG meeting, there was a strong sense of the room that the TSVWG WG should adopt draft-white-tsvwg-nqb as a starting point for work on an NQB PHB.   The WG call for adoption on this list has been open for over a week, since August 21, and having seen only one objection on the list to adoption of the draft (from Dave Taht), the WG chairs (Gorry, Wes and David) have concluded that the WG rough consensus is to adopt this draft.  It’s important to emphasize that the draft is adopted as a *starting point* for TSVWG work, i.e., the current draft content does not represent the rough consensus of the WG.  Significant work will be required to produce an actual PHB spec that is suitable for implementation – see the recent RFC 8622 on the LE PHB (https://datatracker.ietf.org/doc/rfc8622/) for an example of what a PHB spec looks like.

There are a few items that will need attention before the initial -00 WG version of the NQB draft is submitted – these are to avoid confusion about what the WG intends to do:

1)      The draft needs to be clear that the use of the 0x2A DSCP value as the default for this PHB is a *suggestion by the authors that is subject to change*.  Whether to use that DSCP or a different one is a WG decision; the plan is to discuss and select the default DSCP value starting (and hopefully concluding) in September.

2)      The criticisms on this list of the “queue protection” requirement in the draft are largely accurate.   The draft needs at least an Editor’s Note that this material will be revised, as while the DOCSIS mechanism is an example of how to do queue protection, it is not appropriate to require implementation of that mechanism.   A plausible plan that I have discussed with the authors is to write a set of functional/behavioral requirements for NQB “traffic protection” that can be satisfied by a “queue protection” mechanism such as the DOCSIS mechanism, or by a suitably configured FQ AQM implementation.

3)      RFC 8325 reflects the IETF consensus on how to map between Diffserv and WiFi QoS, hence the 8.3 section of the NQB draft needs to be modified to be consistent with RFC 8325.

4)      Similarly, the 8.2 section of the NQB draft needs to be modified to reflect the conclusion of discussion on this topic in Montreal.
Those 4 changes are necessary in the -00 WG version of the NQB draft.

[email-masked]<mailto:[email-masked]>

---

From: <[email-masked]>
Date: Thu, 5 Sep 2019 08:02:51 +0000

Hi Greg,

the policy of the company I work for is: EF marked traffic received on a peering interface is remarked. DiffServ traffic destined to a retail access is subject to admission control, if that traffic is not expected to experience congestion.

The latter might change with high-speed access links. If the users and the above traffic have a hard time congesting these access links, the admission control requirement may not be strictly required.

If NQB traffic is supposed to be sent to third parties without an SLA at peering links being in place, proper selection of the DSCP matters. Without a deeper thought, a DSCP 000xx1 may be a proper choice in that case. Without an SLA being in place, the forwarding to be expected is default forwarding. If NQB is supposed to define a default-like behavior for a serious traffic share, it should be marked by a DSCP in the range 000 xxx at interconnection.

If an SLA is expected to be in place at peerings, any traffic without admission control should face a higher packet drop probability if destined to a retail access, than admission controlled traffic. If I assume the admission controlled traffic to be marked AFx1, NQB traffic in the same queue should be marked AFx2 (or AFx3). I’m open to discuss queue depths and expected traffic properties for a suitable AF class. RFC 8100 proposes to specify DSCPs only at interconnection. That may be a starting point, should SLAs on NQB at peerings be within commercial scope (that’s a non-technical issue, of course).

The choice of a DSCP for traffic which is created and terminated within a single domain doesn’t need to specified – here the PHB specification is important and the behavior of that traffic in the presence of traffic from other PHBs. I’d again opt for AFx2 or AFx3 (hoping that the NQB PHB specification allows for a useful implementation suiting AFx1 and NQB traffic in the same queue, expecting both to expect no congestion, AFx1 traffic to be admission controlled if present and NQB having features to reduce bandwidth if congestion is imminent).

I’m not sure what is expected from a backbone network supporting NQB (and other DiffServ PHBs). MPLS supports 8 traffic classes or codepoints, respectively. This coding space includes the codepoints required for ECN (codepoints, not bits).

I appreciate more standard DSCPs based on a clear and consented definition of the corresponding PHBs. Default marking, LE, EF and Network Management are good examples.

My take – sorry if I misunderstood some of your design aims, should that be the case.

Regards,

Ruediger

Von: tsvwg <[email-masked]> Im Auftrag von Greg White
Gesendet: Donnerstag, 5. September 2019 01:02
An: Black, David <[email-masked]>; [email-masked]
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Thanks David.

-Greg

From: tsvwg <[email-masked]<mailto:[email-masked]>> on behalf of David Black <[email-masked]<mailto:[email-masked]>>
Date: Friday, August 30, 2019 at 9:41 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 5 Sep 2019 10:26:05 +0200

Hi Greg,

let me chime in...

> On Sep 5, 2019, at 01:01, Greg White <[email-masked]> wrote:
> 
> Thanks David.
>  

Have a look at figure 1 on page 19-20 RFC 8325 (https://tools.ietf.org/html/rfc8325#page-19):

+-------------------------------------------------------------------+
  | IETF Diffserv | PHB  |Reference |         IEEE 802.11              |
  | Service Class |      |   RFC    |User Priority|  Access Category   |
  |===============+======+==========+=============+====================|
  |               |      |          |     7       |    AC_VO (Voice)   |
  |Network Control| CS7  | 
RFC 2474
 |            OR                    |
  |(reserved for  |      |          |     0       | AC_BE (Best Effort)|
  | future use)   |      |          |See Security Considerations-Sec.8 |
  +---------------+------+----------+-------------+--------------------+
  |               |      |          |     7       |    AC_VO (Voice)   |
  |Network Control| CS6  | 
RFC 2474
 |            OR                    |
  |               |      |          |     0       | AC_BE (Best Effort)|
  |               |      |          |    See Security Considerations   |
  +---------------+------+----------+-------------+--------------------+
  |   Telephony   |  EF  | 
RFC 3246
 |     6       |    AC_VO (Voice)   |
  +---------------+------+----------+-------------+--------------------+
  |  VOICE-ADMIT  |  VA  | 
RFC 5865
 |     6       |    AC_VO (Voice)   |
  |               |      |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |   Signaling   | CS5  | 
RFC 2474
 |     5       |    AC_VI (Video)   |
  +---------------+------+----------+-------------+--------------------+
  |   Multimedia  | AF41 |          |             |                    |
  | Conferencing  | AF42 | 
RFC 2597
 |     4       |    AC_VI (Video)   |
  |               | AF43 |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |   Real-Time   | CS4  | 
RFC 2474
 |     4       |    AC_VI (Video)   |
  |  Interactive  |      |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |  Multimedia   | AF31 |          |             |                    |
  |  Streaming    | AF32 | 
RFC 2597
 |     4       |    AC_VI (Video)   |
  |               | AF33 |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |Broadcast Video| CS3  | 
RFC 2474
 |     4       |    AC_VI (Video)   |
  +---------------+------+----------+-------------+--------------------+
  |    Low-       | AF21 |          |             |                    |
  |    Latency    | AF22 | 
RFC 2597
 |     3       | AC_BE (Best Effort)|
  |    Data       | AF23 |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |     OAM       | CS2  | 
RFC 2474
 |     0       | AC_BE (Best Effort)|
  +---------------+------+----------+-------------+--------------------+
  |    High-      | AF11 |          |             |                    |
  |  Throughput   | AF12 | 
RFC 2597
 |     0       | AC_BE (Best Effort)|
  |    Data       | AF13 |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |   Standard    | DF   | 
RFC 2474
 |     0       | AC_BE (Best Effort)|
  +---------------+------+----------+-------------+--------------------+
  | Low-Priority  | CS1  | 
RFC 3662
 |     1       | AC_BK (Background) |
  |     Data      |      |          |             |                    |
  +--------------------------------------------------------------------+

  Note: All unused codepoints are RECOMMENDED to be mapped to UP 0
  (See Security Considerations below)

       Figure 1: Summary of Mapping Recommendations from Downstream
                       DSCP to IEEE 802.11 UP and AC

draft-white-tsvwg-nqb defines the following characteristics for flows to qualify for NQB:

"There are many applications that send traffic at relatively low data rates and/or in a fairly smooth and consistent manner such that they are highly unlikely to exceed the available capacity of the network path between source and sink."
[...]
"These Non-queue-building (NQB) flows are typically UDP flows that send traffic at a lower data rate and don't seek the capacity of the link (examples: online games, voice chat, DNS lookups)"

To me this does not rule out relatively high bandwidth applications like (paced) video delivery over UDP.

Anyway the issue I see is that NQB lumps quite a few different traffic types into one category, and when mapping to AC's it seems prudent to not only look at the best case, but also at the average and worst cases as well.

So with video streams as valid NQB flows (@Greg if you believe streaming video not to qualify for NQB then the draft should mention that explicitly) the highest AC defensible according to figure 1 above is AC_VI. So can we please reduce the discussion to the only open question, does NQB qualify for AC_VI or AC_BE?

I argue that NQB will be used for "low latency data" flows, whether the current draft acknowledges that or not (in the end NQB has all the right trimmings for being the "L4S" dscp and I consider it highly unlikely that it will not be used as a cheap entry into the fast lane). And in that case, I believe the aggregate that is marked by NQB should only be marked AC_BE. 
Especially considering that in the egress direction stations will send NQB-marked packets and hence potentially hog air-time slots if using AC_VI/VO, before the queue protection of the router has any possibility to keep things in check.

	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 5 Sep 2019 19:23:46 +0100

David & Sebastien,

[These responses cover come of Sebastien''s points that were common to 
David's reply as well - I'll pick out the remaining ones specific to 
Sebastien's email in a subsequent response. Sorry for the long response, 
but I've tried hard to cover this rigorously.]

> I see serious potential here for a "tragedy of the commons" outcome where lots of QB traffic is sent marked NQB because the resulting latency is (at least initially) lower.

      ===Misunderstanding?==

Using a term like 'tragedy of the commons', implies your argument might 
be based on a different understanding of what NQB is to mine. I'm trying 
to unpick what you think NQB is from your sentence "the resulting 
latency is (at least initially) lower" for a QB flow that marks itself NQB?

Please confirm that you (both) understand that, with NQB, the low 
latency is not provided by scheduling in the network. It is primarily a 
property of NQB traffic sources, and the only necessary property of the 
network's per-hop behaviour for NQB is to isolate NQB from QB traffic 
(i.e. in a separate queue for the NQB class). There is no priority 
scheduling requirement.

[A comment I would make to the authors about the draft: it needs to give 
some example approaches to scheduling between NQB and QB, or at least 
talk about this scheduling.]

>>> The draft claims incentives can be aligned by an implementation being arranged to ensure that NQB traffic benefits from NQB marking and QB
> The above text not only reverses the burden of proof but also changes the question from whether a "tragedy of the commons" is possible to whether it will always (categorically) happen - the resulting straw proposition is then easy to demolish.  Nice try, no sale ...
[BB] OK, I'll turn the burden of proof back round again...

As this insertion is long, I've used sub-headings.

      ==Incentive Alignment?==

To judge whether there's any tragedy of the commons (incentive 
alignment), I'll put up a straw-man NQB configuration that complies with 
the following requirement in the draft:

    a useful property of nodes that support separate queues for NQB and QB
    flows would be that for NQB flows, the NQB queue provides better
    performance (considering latency, loss and throughput) than the QB
    queue; and for QB flows, the QB queue provides better performance
    (considering latency, loss and throughput) than the NQB queue.

_Background_: NQB has become possible largely because Internet access 
link rates have typically become fast enough that the serialization 
delay of a packet can be sub-millisecond, and therefore a queue of a few 
packets introduces delay that is small relative to other non-optional 
sources of delay like propagation. In these cases we no longer need 
priority scheduling for low delay._

Config_:

  * Scheduler:
      o WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at
        least 60Mb/s for NQB flows.
      o Scheduler quantum: 1500B.
  * Buffering:
      o The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
      o The QB buffer is deeper (say 200ms) with an AQM target delay of
        say 10ms.

_Traffic_: Let's introduce some example NQB traffic:

  * 2 paced flows of VoIP datagrams, each avg 50B payload plus 58B of
    Eth/IP/UDP/RTP headers at 33pkt/s
      o bit-rate: 29kb/s /flow
      o serialization delay: 7.2us / pkt
  * 2 streams of 1000B game sync datagrams at 30pkt/s =>
      o bit-rate: 240kb/s /flow
      o serialization delay: 67us / pkt
  * plus occasional DNSSec datagrams, worst case 1500B
      o serialization delay: 100us / pkt
  * Perhaps 540kb/s in all, which is about 0.9% of the min NQB capacity.

_Worst-case NQB queuing delay calculation_ for the above traffic model:
Each NQB flow paces out its own packets, but one might happen to arrive 
while a packet from any of the other NQB flows is already queued. Worst 
case n-1 = 4 other NQB packets queued, where n is the number of 
application flows. And if there's traffic in the QB queue, each NQB 
packet will have to wait for one quantum (100us) while the other queue 
is served. Worst-case NQB delay is then:

    (67us * 2 + 7.2us + 100us) + (100us * 4) = 641us,

It's v unlikely that this worst-case would arise, but it gives an 
indication of where the tail of the delay distribution will lie.

_Tragedy?_
Now, why do you think "the resulting latency is (at least initially) 
lower" for a QB flow that marks itself NQB? Why do you think incentives 
are misaligned (i.e. a tragedy of the commons)?

Certainly, as the draft says, there is a danger of QB being mismarked 
due to accidents or malice. However, there's no /incentive/ to mismark 
to /get/ lower latency, in the strawman config above.

  * If a QB flow mismarked as NQB causes higher latency, it will cause
    it to itself.
  * If a long-running flow behaves in a QB way, one assumes it needs to
    build a queue of perhaps 10ms for full utilization, so in a 3ms
    buffer it will experience high loss and badly under-utilize the link.
  * If a short QB flow mismarked itself as NQB, it would experience its
    self-induced latency during start-up. Nonetheless, If there already
    happens to be a long-running QB flow in the QB queue, it would
    indeed get lower delay than if it marked itself truthfully and added
    to the QB queue. But, with the shallow NQB buffer, the mismarked
    flow would find it very hard to avoid high loss, especially tail loss.

This last point is what I was thinking of when I said "Incentives are 
hard to guess". Even if this was tried experimentally, it would be hard 
to know whether the harm from high risk of loss would dominate the 
benefit from the reduction in delay. But if one accepts that a loss is 
usually equivalent to more than 1RTT of delay, it seems there is no 
incentive for QB flows to mismark.

> or as Sebastian writes:
>>> While there is no operational experience of NQB deployments, I think the market (i.e. most early-adopting operators) will want the warm feeling of
[BB] As this insertion is long, I've used sub-headings...

      ==Incentive Alignment was the point of NQB==

As I understand it, the latest NQB draft was a merge of:

  * NQB from the DOCSIS community
  * and LoLa from the GSMA community.

The whole point of LoLa was to trade off loss and latency incentives to 
avoid the complexities of Diffserv policing. The point of NQB also 
concerned incentives, but avoided the LoLa tradeoff between lower 
latency and higher loss. Instead, both QB and NQB got better performance 
by marking themselves truthfully than they did by mismarking.

1/ Incentives are a first order concern. If incentives are not aligned, 
that affects all typical cases, because every party can be assumed to be 
acting in its own interests.

2/ The NQB draft introduces traffic protection in the context of 
accidents and malice. They are second order concerns, i.e. not expected 
to be the common case, but still potentially of concern.

I might be more sympathetic to "MUST implement" if incentives were not 
aligned. But even if incentives were misaligned, I don't think it would 
be normal or appropriate for the IETF to mandate protection...

      ==MUST is not for Nannying==

Quoting the bible on the use of MUST [RFC2119]:

    Imperatives of the type defined in this memo must be used with care
    and sparingly.  In particular, they MUST only be used where it is
    actually required for interoperation or to limit behavior which has
    potential for causing harm (e.g., limiting retransmissions)  For
    example, they must not be used to try to impose a particular method
    on implementors where the method is not required for
    interoperability.

a) There is certainly no interoperability concern here.

b) "Limiting behaviour which has potential to cause harm" means limiting 
harm to others; it does not mean it is appropriate to mandate that an 
NQB implementation MUST limit harm to itself. That is nannying. 
Implementers are perfectly able to make that decision for themselves.

For instance, regular Diffserv with prioritization has much stronger 
protection concerns than NQB because, not only is there potential for 
accidents and malice, but incentives are not aligned. Nonetheless, 
Diffserv was divided into standards track RFC2474 about the DS field 
(interop), and informational RFC2475 about domain structure, policing, 
etc. There is no normative text in RFC2475 and I don't think the IETF 
ever mandated that a border router MUST implement Diffserv policing.

      ==Continual Run-time Protection is Unnecessary==

Even if we did want to mandate protection, it would be inappropriate to 
mandate continual run-time protection. Because it is common for 
accidents and malice to be weeded out before run-time.

For instance, in the upstream of a mobile network, the RAN scheduler 
only protects one user's traffic from another's. It would not protect 
one NQB app of a user from another NQB app of the same user. To do that, 
a step could be added to the app-store validation process to verify 
correct use of NQB by an app.

Similarly, on other platforms virus checkers weed out malicious software 
at installation time.

      ==Mandating a Layering Violation?==

Saying "MUST implement" some form of traffic protection would turn NQB 
from a simple L2 or L3 class-based queue isolation scheme (with a 
"SHOULD" recommendation to access L4) into an approach that is 
non-compliant unless it accesses L4 headers.

Low latency is a big deal in 5G. The 3GPP complied with normal network 
layering principles when it required the 5G RLC layer to compress and 
encrypt its payload (the DSCP at L3 gets translated into a 5QI in the 
RLC header at L2, which is equivalent of a QCI in LTE).

How will it look to the 3GPP if the IETF introduces a low latency DSCP 
that mandates access to L4 headers in order to use it?

> 	The hurdle here is much lower, particularly for applications that already use UDP.
[BB] That's all true. Yes, it increases the risk, particularly of 
accidents and bugs.

Don't get me wrong, I'm not trying to argue down the risk here. I'm 
saying it's not the IETF's role to mandate what to do about the risk (no 
matter how high it is). The arguments above against nannying and 
mandating self-protection all still apply.

> I concur with Sebastian that the onus is on the proposers of the complexity/cost savings to demonstrate that the resulting designs are safe for the Internet.  I'm reminded by analogy of something t…
[BB] I hope I've explained that, even if risks are high (of 
self-interest, accidents or malice), it is not the IETF's role to 
mandate that implementers protect against these risks.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 5 Sep 2019 22:52:41 +0300

> On 5 Sep, 2019, at 9:23 pm, Bob Briscoe <[email-masked]> wrote:
> 
> Config: 
> 	• Scheduler: 
> 		• WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least 60Mb/s for NQB flows. 
> 		• Scheduler quantum: 1500B. 
> 	• Buffering:
> 		• The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
> 		• The QB buffer is deeper (say 200ms) with an AQM target delay of say 10ms.

This seems like a reasonable implementation, on the face of it.

> Now, why do you think "the resulting latency is (at least initially) lower" for a QB flow that marks itself NQB? Why do you think incentives are misaligned (i.e. a tragedy of the commons)?

Okay, let's extend your traffic scenario a bit by assuming there's a significant amount of QB traffic, perhaps from a number of 50GB game updates in progress to various PCs and consoles in the household.  This could be as many as a couple of dozen bulk, saturating flows at a time, with the load being more-or-less continuous for an hour or so.  This traffic originates from gaming-related companies, so they know better than to risk interfering with actual gaming traffic carrying the NQB marking.

The above PHB template does a good job of isolating the NQB traffic from the BE traffic in this case, in which everyone behaves nicely.

But now let's introduce an adversary; let's call them Netflix' Unscrupulous Rival (NUR).  They are not in the gaming (interactive entertainment) business, but Video On Demand (passive entertainment).  Their USP is that they get the complete video file onto the subscriber's PC in the minimum possible time; this incidentally also removes the load from their servers as early as possible.  In short, they have chugged the entire cask of Flow Completion Time koolaid, and their flows are multiple gigabytes each.

In service to this goal, NUR have selected BBRv2 as their CC algorithm because, lacking the traditional TCP sawtooth, it achieves higher throughput than most, and without needing to open multiple connections (which means reduced server load and client software complexity).  And, because they're unscrupulous and literally don't care about competing traffic - how important can it be if they're busy watching our video? - they've increased the packet loss threshold to 10% from the default of 1%; they don't mind retransmitting a few packets if it gets the job done faster.  Since we may assume that the AQM in this PHB implements RFC-3168 ECN, this tweak doesn't actually have much effect on this particular case, but it does elsewhere, on the many networks still using dumb FIFOs.

What NUR now notices is that, at some times and with some particular subscribers, their throughput is maybe a tenth of what they expect.  This is unacceptable, so they investigate.  And then they switch on NQB marking to see what happens.  After all, BBRv2 is advertised as not building a queue, so it's allowed, right?

This gives them almost uncontended access to the 60Mbps reserved for NQB traffic, instead of having to share the 120Mbps total pipe with a couple of dozen other bulk flows.  NUR is ecstatic and rolls it out across their entire system.

And there's your incentive to mis-mark as NQB.

Now, what effect does this have on the actual NQB traffic?  Well, BBRv2 spends most of its time pacing at just below the detected path capacity, but periodically probes for additional bandwidth by pacing at a higher rate for an RTT or so.  Usually this will exceed the capacity allocated to NQB and start queuing, which imposes some delay both on itself and on other NQB traffic sharing the same queue.  This will cap out at 6ms (30 packets at 60Mbps) - ten times the worst-case delay you calculated for NQB traffic alone - at which point packet loss will begin to occur.  Because the NUR flow is very long and not application-limited, tail loss is not a factor for most of its lifetime, and they have configured BBRv2 to be exceptionally tolerant of loss before initiating congestive backoff.

In short, NQB traffic may periodically experience an additional 6ms delay and 10% packet loss due to this mis-marked NUR flow, depending on some other factors.

As for the "tragedy of the commons", NUR then proudly publishes an investor report and a white paper explaining how they quintupled their throughput with this one weird trick.  Other unscrupulous Internet companies take note of this and try it themselves, usually with less competence and without measuring the actual extent to which they can expect improvements in practice.  After all, there's no law against it, and it worked for NUR didn't it?  They're a big famous Internet company so they must be right!

Consequently the NQB queue becomes increasingly full of QB traffic.  The latency advantage of the NQB marking is thus eroded, with 6ms delays and significant packet loss becoming the rule rather than the exception, not all that much better than the 10ms target delays on the other side of the scheduler.

Do you now see?

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 5 Sep 2019 22:25:19 +0200

Hi Bob,

> On Sep 5, 2019, at 20:23, Bob Briscoe <[email-masked]> wrote:
> 
> David & Sebastien,
> 
> [These responses cover come of Sebastien''s points that were common to David's reply as well - I'll pick out the remaining ones specific to Sebastien's email in a subsequent response. Sorry for the long response, but I've tried hard to cover this rigorously.]
> 
> Using a term like 'tragedy of the commons', implies your argument might be based on a different understanding of what NQB is to mine. I'm trying to unpick what you think NQB is from your sentence "the resulting latency is (at least initially) lower" for a QB flow that marks itself NQB?
> 
> Please confirm that you (both) understand that, with NQB, the low latency is not provided by scheduling in the network. It is primarily a property of NQB traffic sources, and the only necessary property of the network's per-hop behaviour for NQB is to isolate NQB from QB traffic (i.e. in a separate queue for the NQB class). There is no priority scheduling requirement. 

	[SM] Assuming our NQB endpoints employ reasonably competent oracles to actually make that feasible.... especially in the light of transient "cross-traffic". Also the NQB senders will have to actually time their transmissions such that their packets arrive almost perfectly "desynchronized" (spread out such that each of their packets just hit our non-scheduling bottleneck hop just before the "egress" time-slot assigned to each flow). This is not going to work without some form of feed-back from the bottleneck... In my understanding the non-queue build hence requires the bottlenecks interaction, a process I would naively call scheduling; and yes since the only node that has enough information to actually synchronize the end-points I would call this scheduling in the network. Either that or our NQB flows will build up (transient) queues at the bottleneck. What a I missing here?

> 
> [A comment I would make to the authors about the draft: it needs to give some example approaches to scheduling between NQB and QB, or at least talk about this scheduling.]
> 
> To judge whether there's any tragedy of the commons (incentive alignment), I'll put up a straw-man NQB configuration that complies with the following requirement in the draft:
> Background: NQB has become possible largely because Internet access link rates have typically become fast enough that the serialization delay of a packet can be sub-millisecond, and therefore a queue of a few packets introduces delay that is small relative to other non-optional sources of delay like propagation. In these cases we no longer need priority scheduling for low delay.
> 
> Config: 
> 	• Scheduler: 
> Traffic: Let's introduce some example NQB traffic:
> 	• 2 paced flows of VoIP datagrams, each avg 50B payload plus 58B of Eth/IP/UDP/RTP headers at 33pkt/s 
> 		• bit-rate: 29kb/s /flow
> 		• serialization delay: 7.2us / pkt
> 	• 2 streams of 1000B game sync datagrams at 30pkt/s => 
> 		• bit-rate: 240kb/s /flow
> 		• serialization delay: 67us / pkt
> 	• plus occasional DNSSec datagrams, worst case 1500B
> 		• serialization delay: 100us / pkt
> 	• Perhaps 540kb/s in all, which is about 0.9% of the min NQB capacity.
> Worst-case NQB queuing delay calculation for the above traffic model: 
> Each NQB flow paces out its own packets, but one might happen to arrive while a packet from any of the other NQB flows is already queued. Worst case n-1 = 4 other NQB packets queued, where n is the number of application flows. And if there's traffic in the QB queue, each NQB packet will have to wait for one quantum (100us) while the other queue is served. Worst-case NQB delay is then:
> It's v unlikely that this worst-case would arise, but it gives an indication of where the tail of the delay distribution will lie.
> 
> Tragedy?
> Certainly, as the draft says, there is a danger of QB being mismarked due to accidents or malice. However, there's no /incentive/ to mismark to /get/ lower latency, in the strawman config above.

	[SM] But this also gives a great incentive for latency sensitive capacity seeking traffic to mis-mark itself into the NQB treatment, 0.6ms delay versus >=10ms. Why is that not an incentive, I fail to see. If a flow is sufficiently short (and say for browsing many are done in iw10) or bursty this delay advantage will be real...

> 	• If a QB flow mismarked as NQB causes higher latency, it will cause it to itself.
> 	• If a long-running flow behaves in a QB way, one assumes it needs to build a queue of perhaps 10ms for full utilization, so in a 3ms buffer it will experience high loss and badly under-utilize the link.
> 	• If a short QB flow mismarked itself as NQB, it would experience its self-induced latency during start-up. 
> Nonetheless, If there already happens to be a long-running QB flow in the QB queue, it would indeed get lower delay than if it marked itself truthfully and added to the QB queue. But, with the shallow NQB buffer, the mismarked flow would find it very hard to avoid high loss, especially tail loss.

	[SM] But at 30 packets buffer depth tail loss is hardly that much worse than head drop, no?

> 
> This last point is what I was thinking of when I said "Incentives are hard to guess". Even if this was tried experimentally,

	[SM] What bothers me personally a bit, is the "if this was tried" bit in the above instead of actually running a real experiment.

> it would be hard to know whether the harm from high risk of loss would dominate the benefit from the reduction in delay. But if one accepts that a loss is usually equivalent to more than 1RTT of delay, it seems there is no incentive for QB flows to mismark.

	[SM] Well, my hypothetical bursty web flow will just be fine as long as it does not send bursts larger than 3 ms.

> 	• NQB from the DOCSIS community 
> 	• and LoLa from the GSMA community. 
> The whole point of LoLa was to trade off loss and latency incentives to avoid the complexities of Diffserv policing.

	[SM] This seems a sane approach, make low-latency costly in another dimension and users will think hard what they actually require.

> The point of NQB also concerned incentives, but avoided the LoLa tradeoff between lower latency and higher loss. Instead, both QB and NQB got better performance by marking themselves truthfully than they did by mismarking.

	[SM Now, show that this holds true for conditions instead of one example...

> 
> 1/ Incentives are a first order concern. If incentives are not aligned, that affects all typical cases, because every party can be assumed to be acting in its own interests.
> 
> 2/ The NQB draft introduces traffic protection in the context of accidents and malice. They are second order concerns, i.e. not expected to be the common case, but still potentially of concern.

	[SM] You realize that DDOS as a service is a thing people pay for over the internet? I would say malice is much more common than you make it sound.

> 
> I might be more sympathetic to "MUST implement" if incentives were not aligned. But even if incentives were misaligned, I don't think it would be normal or appropriate for the IETF to mandate protection...
> 
> b) "Limiting behaviour which has potential to cause harm" means limiting harm to others; it does not mean it is appropriate to mandate that an NQB implementation MUST limit harm to itself. That is nannying. Implementers are perfectly able to make that decision for themselves.
> 
> For instance, regular Diffserv with prioritization has much stronger protection concerns than NQB because, not only is there potential for accidents and malice, but incentives are not aligned. Nonetheless, Diffserv was divided into standards track RFC2474 about the DS field (interop), and informational RFC2475 about domain structure, policing, etc. There is no normative text in RFC2475 and I don't think the IETF ever mandated that a border router MUST implement Diffserv policing.

	[SM] But unlike NQB "normal" standard DSCPs were not supposed to survive passage though the internet e2e.

> Even if we did want to mandate protection, it would be inappropriate to mandate continual run-time protection. Because it is common for accidents and malice to be weeded out before run-time.

	[SM] This seems quite optimistic in light of giving gaming traffic as a user for the NQB marking, and building a system in which relatively small periodic bursts are sufficient to degrade NQB performance. If countering malice is only an after thought it is not going to work.

> 
> For instance, in the upstream of a mobile network, the RAN scheduler only protects one user's traffic from another's. It would not protect one NQB app of a user from another NQB app of the same user. To do that, a step could be added to the app-store validation process to verify correct use of NQB by an app.

	[SM] That puts an awful lot of trust into the walled-garden vetting processes.

> 
> Similarly, on other platforms virus checkers weed out malicious software at installation time. 
> 
> Saying "MUST implement" some form of traffic protection would turn NQB from a simple L2 or L3 class-based queue isolation scheme (with a "SHOULD" recommendation to access L4) into an approach that is non-compliant unless it accesses L4 headers.

	[SM] Then require at least a pure src/dst IP address tuple for your "flow" definition to at least protect different end points from each user, and for IPv6 also include the floe label (just declare IPv4 to be out of scope), if layering violation are of such concern. IMHO the fact that (to my knowledge) most internet traffic is either UDP or TCP indicates that "port" addresses really are useful, and maybe the "layering violation" is that the IP header omits these fields, but I digress.

> 
> Low latency is a big deal in 5G.

	[SM] Which so far means in 5G marketing...

> The 3GPP complied with normal network layering principles when it required the 5G RLC layer to compress and encrypt its payload (the DSCP at L3 gets translated into a 5QI in the RLC header at L2, which is equivalent of a QCI in LTE).
> 
> How will it look to the 3GPP if the IETF introduces a low latency DSCP that mandates access to L4 headers in order to use it? 
> 
> 
>> 
> [BB] That's all true. Yes, it increases the risk, particularly of accidents and bugs.
> 
> Don't get me wrong, I'm not trying to argue down the risk here. I'm saying it's not the IETF's role to mandate what to do about the risk (no matter how high it is). The arguments above against nannying and mandating self-protection all still apply. 
> 
>> 
> [BB] I hope I've explained that, even if risks are high (of self-interest, accidents or malice), it is not the IETF's role to mandate that implementers protect against these risks.

	[SM] The whole premise of the NQB draft is that NQB-ness is obeservable data:
"In an unmanaged
   environment, how can networks trust endpoint marking, and why
   wouldn't all applications mark their packets as NQB?

   As an answer the last question, it is worthwhile to note that the NQB
   designation and marking would be intended to convey verifiable
   traffic behavior, not needs or wants."

Requiring "verifiable behavior" and then not actually verifying it indicates that either the requirement or the enforcement policy is wrong and should be changed, you seem to argue for changing the NQB requirement then?

	Sebastian

>>>> Bob Briscoe
>>>> http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 5 Sep 2019 22:46:27 +0100

Sebastian, (sorry, I mis-spelled your name previously), inline...

> 	[SM] The draft says "... it is worthwhile to note that the NQB designation and marking would be intended to convey verifiable traffic behavior, not needs or wants." in the light of this requiremen…
[BB] Just because the behaviour is verifiable does not mean it always 
has to be verified. For instance, if someone claims a car is 17 years 
old, you might trust it is likely to be truthful because the speaker 
knows you could verify it. But it doesn't mean you have to actually look 
up the engine and chassis serial numbers to check the date of manufacture.

In the case of NQB, there is less need to verify behaviour because 
applications or users have no incentive to mismark - they would be worse 
off.

However, there is still the risk of accidents or malice. Nonetheless, 
the harm from any accidental or malicious NQB mismarking will be 
confined to other applications of the same customer {Note 1}.

Given there's no incentive to mismark, and the effect of accidents or 
malice are contained, an operator might decide that traffic protection 
is unnecessary - more cost than benefit.

The point about verifiability is merely that, if an operator does choose 
to protect flows from each other, an implementation can objectively 
determine which flows are out of compliance, because NQB behaviour is 
verifiable. Objective verifiability is neutral, which is important in 
jurisdictions with net neutrality legislation.

{Note 1}: The draft says NQB is targeted at access links. That implies 
NQB scheduling is between the classes of one customer and that a higher 
level in the network scheduling hierarchy isolates customers from each 
other.

> 	[SM] I would have thought it would be on those claiming a laxer enforcement model to proof that that model is sufficient?
[BB] See previous email to you and David Black.
> 	I fail to see any scheduler beyond my ISPs traffic shaper that has a wholistic-enough view to classify traffic based on "user" (think NATed IPv4 address, but variable length IPv6 prefixes, plus a…
[BB] In DSL, DOCSIS, mobile or satellite access networks, a scheduler 
both limits a customer to their contracted access rate and shares out 
the capacity during over-subscription periods. How these schedulers 
actually do that is secret sauce in many cases {Note 2}, but they all do 
it somehow.

If another link deeper into the network becomes the bottleneck (e.g. a 
core or peering link), then there is typically no scheduler arbitrating 
between Internet customers {Note 3}. Then it's down to TCP between 
flows, regardless of which customer they are from. However, a core 
bottleneck rarely happens by design - usually it would be due to a screw up.

The main exception is campus networks (e.g. Universities, corporates), 
where the access link with the Internet doesn't usually schedule between 
users. Solutions range from a simple FIFO (i.e. the transport layers 
will be determining shares per flow, not per user) to complex often 
DPI-based prioritization solutions that might recognize users and/or 
applications.

{Note 2}: When I worked for BT, we described BT's wholesale solution here:
https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf#10
(it had already been published elsewhere, due to the regulatory need for 
transparency in the UK market).

{Note 3}: But one customer cannot get a huge share of the core capacity 
by opening loads of TCP flows, because each customer is still capped by 
their access scheduler as well.

> 	[SM] I believe for malicious actors NQB will be a attractive DSCP as it promised to allow doing harm even at low bandwidth (just send a low average rate in lumpy bursts and I would expect an NQB-h…
[BB] Yes, if a malicious actor could get control of either end, it would 
be fairly easy for them to cause more queuing. On the other hand it 
would be confined within one customer's service, so not a particularly 
high profile attack for a script kiddie to boast about.

As I said in my other mail to you & David, I'm not saying there's not a 
risk. I'm saying it's not the IETF's place to mandate that implementers 
MUST protect against this risk. It's not a standards matter. 
Implementers will provide protection if operators want it. And if some 
operators don't want it, there will be a market for lower cost solutions 
without protection. Implementers are perfectly able to make these 
decisions themselves. The IETF has no mandate to meddle in this market.

> *) This is a hypothesis I have not confirmed in any way, but it seems to be in line with how I understand dual queue aqm to work (so it might be more of a reflection of my level of understanding an…
Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 6 Sep 2019 00:13:45 +0100

Jonathan,

On 05/09/2019 20:52, Jonathan Morton wrote:
> Now, what effect does this have on the actual NQB traffic?  Well, BBRv2 spends most of its time pacing at just below the detected path capacity, but periodically probes for additional bandwidth by pacing at a higher rate for an RTT or so.  Usually this will exceed the capacity allocated to NQB and start queuing, which imposes some delay both on itself and on other NQB traffic sharing the same queue.  This will cap out at 6ms (30 packets at 60Mbps) - ten times the worst-case delay you calculated for NQB traffic alone - at which point packet loss will begin to occur.
[BB]: Picky point: I said NQB has 3ms of physical buffer.
> Because the NUR flow is very long and not application-limited, tail loss is not a factor for most of its lifetime, and they have configured BBRv2 to be exceptionally tolerant of loss before initiating congestive backoff.
>
Yes, this is a possible attack, altho some picky details are 
questionable, e.g. I think BBRv2 will run significantly under 60Mb/s 
because of the interaction between its bandwidth probes and the very 
shallow buffer (that can be tried experimentally). But in general I 
admit there will be a gain to the attacker here.

However, you cannot put this solely down to incentive misalignment. It 
is an active attack, because NUR has tweaked BBRv2's loss tolerance from 
1% to 10%. That in itself would give a QB flow significantly higher 
throughput than other competing QB flows (whether Cubic or BBRv2).

And you're basing NUR's attack on BBRv2's exploitation of a min loss 
threshold (for whatever value of threshold is chosen), as if BBRv2 
cannot be the cause of the tragedy of the commons because Google asserts 
it is not, then applying it to an NQB queue allows you to blame NQB for 
the tragedy. Rather a disingenuous argument isn't it?

Wouldn't it be sufficient to use a Cubic flow or even Reno with NQB, 
which would get about 75% of 60Mb/s, which would be higher than getting 
1/24 of 120Mb/s with QB. It would even be higher than 33% of 120Mb/s if 
there were only two long-running flows in the QB queue.

Nonetheless, you have not considered the question of how often NUR's 
tweak will make throughput worse by using NQB, in all the cases where 
the QB queue has 0 or 1 long-running flow in it (likely the more 
prevalent cases for most users). Wouldn't users report this in the 
forums which would put others off trying it out?

More generally, attempts to take more throughput at the expense of other 
applications of the same customer, have come and gone over the years, 
and so far none has taken hold. Again, probably because users report the 
negative effects on your own other apps in the forums.

I'm not saying none will ever take hold. Just as I'm not trying to 
downplay the risks of attacks against NQB.

My argument is that implementers can decide whether traffic protection 
is worthwhile, and it's not the IETF's place to tell them to.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 6 Sep 2019 03:12:04 +0300

> On 6 Sep, 2019, at 2:13 am, Bob Briscoe <[email-masked]> wrote:
> 

Which you explicitly defined at a 120Mbps drain rate.  But with both BE and NQB queues saturated, each will drain at only 60Mbps, so the same amount of NQB buffer works out to 6ms in that case.

It would only remain at 3ms if there was no BE traffic competing while the NQB queue was saturated - a somewhat unlikely scenario if the NQB marking is used as intended.

> Yes, this is a possible attack, altho some picky details are questionable, e.g. I think BBRv2 will run significantly under 60Mb/s because of the interaction between its bandwidth probes and the very shallow buffer (that can be tried experimentally). But in general I admit there will be a gain to the attacker here.
> 
> However, you cannot put this solely down to incentive misalignment.

The gain to the attacker *is* the incentive - which I did my best to illustrate through the narrative device of NUR, their motivations, and the evolution of their methods of maximising throughput for themselves.

Another form of gain would be if an attacker wanted to *degrade* service in the NQB category specifically.  This does not require that they gain throughput for themselves, only that latency and/or packet loss become unacceptable for legitimate NQB traffic.  This could be done with a DDoS in which most (but not all) of the flood traffic is marked NQB.  This would ensure that the BE queue is saturated (reducing NQB's share to 50% and pushing the latency up to 6ms) and force NQB traffic to share the drop rate required to fit the NQB-marked flood traffic down the remaining half.

> And you're basing NUR's attack on BBRv2's exploitation of a min loss threshold (for whatever value of threshold is chosen), as if BBRv2 cannot be the cause of the tragedy of the commons because Google asserts it is not, then applying it to an NQB queue allows you to blame NQB for the tragedy. Rather a disingenuous argument isn't it?

I have not yet had the opportunity to try out BBRv2 in practice and in competition with other CC algos.  However, it is a reasonable choice for NUR simply because it achieves near-perfect link utilisation when uncontended.

I specifically conceived NUR as a plausible RFC-ignorant actor, concerned only with their own performance.  But they are starting with established leading-edge technology and "tuning" it to their needs, rather than simply blasting packets without CC at all.  The latter would probably break their own CDN in short order - it's possible they tried it, and backed off when they saw the problems it caused themselves.

NUR's subsequent tweak to accept 10% loss is *not* attributable to Google - they chose a 1% default for good reason - but is simply NUR's own sociopathy showing itself when they started to encounter contention on some last-mile links.  Because BBRv2 understands ECN (to some extent, anyway), it would not run into significant loss when encountering the NQB-aware qdisc you described while still marked BE.

Because it is the sender that chooses what marking to apply and which CC algo to employ, users at the receiving end have little choice but to accept it - aside from discarding the rest of their paid-for subscriptions and moving to a competitor.

Meanwhile, the tragedy of the commons mainly occurs when other people start copying NUR's example.  Until then, the bad effects are confined to the relatively short periods when one of NUR's videos is being downloaded in the same household, *and* the periodic bursts in which BBRv2 is conducting a bandwidth probe.

> Wouldn't it be sufficient to use a Cubic flow or even Reno with NQB, which would get about 75% of 60Mb/s, which would be higher than getting 1/24 of 120Mb/s with QB. It would even be higher than 33% of 120Mb/s if there were only two long-running flows in the QB queue.

Yes, and that is what later movers following NUR's example might notice, if they bother to actually measure the effect, which most might not - they're just blindly copying some tweak they don't really understand.  I'm not sure that it really helps your argument.

> Nonetheless, you have not considered the question of how often NUR's tweak will make throughput worse by using NQB, in all the cases where the QB queue has 0 or 1 long-running flow in it (likely the more prevalent cases for most users). Wouldn't users report this in the forums which would put others off trying it out?

If there's 0-1 BE flows contending for the link, then NUR will get similar throughput in either case when using BBRv2, because WRR allows either child queue to grow into capacity left unused by the other, and BBR tries to keep the bottleneck queue empty on average.  This might be less true if some other CC algo was used, but I note that the 6ms capacity of the NQB side is not very different from the 10ms target of the BE side, unless the path latency is exceptionally low.

> More generally, attempts to take more throughput at the expense of other applications of the same customer, have come and gone over the years, and so far none has taken hold. Again, probably because users report the negative effects on your own other apps in the forums.

As I noted, NUR's attitude here is "who cares about other traffic while they're watching our videos; they can get to it when they're done with us".  Their customer support hotline is outsourced to Bangalore, and involves a rigid script being followed in a very heavy accent, and no access to any means of actually solving the customer's problem.

I'm sure you've encountered a similar setup in the real world.

> My argument is that implementers can decide whether traffic protection is worthwhile, and it's not the IETF's place to tell them to.

But the IETF gets to say whether your specification is approved for publication as an RFC.  As part of that decision, they must consider the possible negative effects on the Internet as well as the benefits.  That's the process we're going through right now, and which I am attempting to help inform.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 6 Sep 2019 11:51:52 +0200

> On Sep 5, 2019, at 23:46, Bob Briscoe <[email-masked]> wrote:
> 
> [BB] Just because the behaviour is verifiable does not mean it always has to be verified.

> For instance, if someone claims a car is 17 years old, you might trust it is likely to be truthful because the speaker knows you could verify it. But it doesn't mean you have to actually look up the engine and chassis serial numbers to check the date of manufacture.

	[SM] I fail to see how this is a fitting analogy to this issue at hand. The NQB draft defines NQB-ness as " the NQB designation and marking would be intended to convey verifiable traffic behavior, not needs or wants." IMHO either get rid of this section or mandate the verification.

> 
> In the case of NQB, there is less need to verify behaviour because applications or users have no incentive to mismark - they would be worse off.

	[SM] I still do not believe that to be true, and no amount of Gedankenexperimente is going to change that, as this is rather a data question. 
As I wrote in another mail, any application that can gracefully deal with the shallow buffer of the NQB queue will be fine to misclassify itself into it even if its behaviors would be verifiably capacity-seeking. The only argument for misalignment of incentives to mismark is the shallowness of the buffer, once that is solved (by following Jonathan's BBR example) the argument that "for NQB flows, the NQB queue provides better performance (considering latency, loss and throughput) than the QB queue; and for QB flows, the QB queue provides better performance (considering latency, loss and throughput) than the NQB queue." is shown to be overly optimistic unless backed by queue protection.

> 
> However, there is still the risk of accidents or malice.

	[SM] With malice actually being a market place with its own economics. IMHO that factor alone should make queue protection mandatory, as this proposal will allow denial of service attacks much cheaper (no need to clogg the full pipe as long as one can drive the NQB queue into dropping mode often enough and that, in your example requires at most 31 full-MTU packets...)

> Nonetheless, the harm from any accidental or malicious NQB mismarking will be confined to other applications of the same customer {Note 1}.

	[SM] The draft mentions access links, it does not restrict itself to access links though, so these need to be covered as well, no? But in the DOS case it is not an application of the customer that degrades the NQB queue but the attacker's.

> 
> Given there's no incentive to mismark,

	[SM] Which I am not convinced to be true yet.

> and the effect of accidents or malice are contained, an operator might decide that traffic protection is unnecessary - more cost than benefit.

	[SM] Sure, but in that case our hypothetical operator will have no problem whatsoever to simply ignore the RFC's MUST/SHOULD language. But making it SHOULD means the operator can do so and actually crow about it to its end customers.

> 
> The point about verifiability is merely that, if an operator does choose to protect flows from each other, an implementation can objectively determine which flows are out of compliance, because NQB behaviour is verifiable. Objective verifiability is neutral, which is important in jurisdictions with net neutrality legislation.

	[SM] If a tree falls in a forrest and no one is there to witness it does it still make a sound? Only the bottleneck is guaranteed to have sufficient information to actually perform the verification, if that bottleneck does not do this it is game over; with the exception of the very narrow model of NQB-scheduling only on the access link in which case that observable behavior at the end-user will be strongly correlated with the behavior at the bottleneck (but even there it is not exactly clear which packets were dropped by the upstream NQB-scheduler and why).

> 
> 
> {Note 1}: The draft says NQB is targeted at access links. That implies NQB scheduling is between the classes of one customer and that a higher level in the network scheduling hierarchy isolates customers from each other.

	[SM] Given your explanation, the draft maybe should clarify that it only targets access links, otherwise this argument is not very convincing.

> [BB] See previous email to you and David Black.

	[SM] Still not convinced.

> [BB] In DSL, DOCSIS, mobile or satellite access networks, a scheduler both limits a customer to their contracted access rate and shares out the capacity during over-subscription periods. How these schedulers actually do that is secret sauce in many cases {Note 2}, but they all do it somehow.

	[SM] in DSL networks the "scheduler" might actually just be the fact that the sync rate between CPE and DSLAM sets an upper limit and that the max sync rate can be programmed from the ISPs side. Such configurations are hopefully rare though.

> 
> If another link deeper into the network becomes the bottleneck (e.g. a core or peering link), then there is typically no scheduler arbitrating between Internet customers {Note 3}. Then it's down to TCP between flows, regardless of which customer they are from. However, a core bottleneck rarely happens by design - usually it would be due to a screw up.

	[SM] Well, in my reality such deeper bottlenecks are not such rare unicorns, there often are predictable (time of day dependent) capacity exhaustions on the transit links to/from an ISP's network that introduce unwanted packet loss and delay. But as I predicted, beyond the access link's upstream end (loosely defined) no network node knows of user directly (albeit using IPv4 dst addresses and IPv6 dst prefixes should give more remote hops also some handle on endpoints/endpoint-networks). But I admit that this is somewhat tangential to our topic.

> 
> The main exception is campus networks (e.g. Universities, corporates), where the access link with the Internet doesn't usually schedule between users. Solutions range from a simple FIFO (i.e. the transport layers will be determining shares per flow, not per user) to complex often DPI-based prioritization solutions that might recognize users and/or applications.
> 
> 
> (it had already been published elsewhere, due to the regulatory need for transparency in the UK market).

	[SM] Thanks this seems to confirm my intuition derived from observations of my own access link properties.

> 
> {Note 3}: But one customer cannot get a huge share of the core capacity by opening loads of TCP flows, because each customer is still capped by their access scheduler as well.
> 
> 
>> 
> [BB] Yes, if a malicious actor could get control of either end, it would be fairly easy for them to cause more queuing.

	[SM] Why would the malicious actor need control over either end to just send bursts of NQB-marked packets to one of the endpoints'  IP-addresses? All needed seems to be the IP-address of the end-point and the low bandwidth DOS game is on... (this actually promises to take the (D) out of (D)DOS again as the required traffic burst rates to drive the NQB queue into tail drop seem rather low).

> On the other hand it would be confined within one customer's service, so not a particularly high profile attack for a script kiddie to boast about.

	[SM] On-line gamers do exactly that, they order and pay for (D)DOS attacks on individual other users they want to disadvantage during game-play, as far as I know. The fact that DDOS as a service exists indicates to me, that there exists sufficient demand for malicious action, to predict that if NQB gets adopted by on-line games NQB queues will get targeted. In a sense the proposed NQB example from your other mail, serves the game-traffic on a silver platter in a simple attack sensitive queue that has a convenient way to be addressed from any outside entity, without having to flood the whole segment link, potentially allowing the attack to continue under the ISP's "radar".

> 
> As I said in my other mail to you & David, I'm not saying there's not a risk.

	[SM] The discussion is about the level of risk and how to mitigate this as as far as I see the proposed NQB scheme will increase the attack surface of users behind and NQB-honoring scheduler.

> I'm saying it's not the IETF's place to mandate that implementers MUST protect against this risk. It's not a standards matter. Implementers will provide protection if operators want it. And if some operators don't want it, there will be a market for lower cost solutions without protection. Implementers are perfectly able to make these decisions themselves. The IETF has no mandate to meddle in this market.

	[SM] David's proposal was MUST implement SHOULD use which seems like it would please you. I believe it should be a categorical MUST for implementation and use, as the more I think about it the more it becomes clear that this thing is dangerously allowing targeted attacks on important services like VoIP in a convenient way.

Best Regards
	Sebastian

---

From: Greg White <[email-masked]>
Date: Fri, 6 Sep 2019 17:18:21 +0000

Hi Ruediger,

Many networks can (and do) configure the DSCP to PHB mappings and policies in accordance with their own intended usage.  In particular, DOCSIS links and LTE links (likely others as well) can be configured in a manner that complies with the PHB definition.  In other cases, it may be either necessary or sufficient (or both) to support NQB traffic without fully complying with the PHB (of course this isn’t new with NQB).

One of the high level goals is to ensure that the benefits of differentiating NQB traffic are not lost when that traffic traverses a WiFi link (since in many of the target use cases it does).  WiFi gear will likely not support all of the aspects of the PHB definition (at least current equipment will not).  We are essentially limited to 4 queues (BK, BE, VI, VO) and their associated CSMA parameters.  An additional constraint that we’re faced with is that in many cases the mapping from DSCP to AC is essentially fixed – it is either operationally hard or impossible to change.

Of the WiFi APs that we’ve tested (a sample of current generation .11ac devices, covering several chipsets) 100% of them use the “default mapping” of DSCP to AC as follows.

DSCP
WMM Access Category
001xxx, 010xxx
Background (AC_BK)
000xxx, 011xxx
Best Effort (AC_BE)
10xxxx
Video (AC_VI)
11xxxx
Voice (AC_VO)

Living within this constraint, and trying to make the best of it, we’ve selected as a default DSCP the 0b101010 value, which would get mapped into the Video Access Category (along with CS4, AF41, AF42, AF43, CS5, VA, EF).  This would ensure that, in large part, NQB traffic is handled in a separate queue from the queue-building bulk data traffic (the vast majority of which uses AC_BE or AC_BK).

I would not expect that it would be important for NQB marked traffic to be differentiated in core network switches or routers, which in many cases do not provide deep (10’s to 100’s of ms) buffers.  So, in these switches NQB would likely be treated the same as Default (or whichever local use codepoint is used for best effort).

A characteristic of the NQB PHB as implemented in DOCSIS is that it provides low latency, low latency variation, and low loss for an NQB marked flow as long as the flow is “well behaved”.  If an NQB-marked flow is not well behaved then it experiences greater latency variation, greater loss, and potential re-ordering.   A well behaved flow is one that does not materially cause queue build-up in the network, i.e. its instantaneous rate does not materially exceed the instantaneous available bandwidth of the path.

In managed networks, or in cases where SLAs are provided, it seems to me that an operator should determine whether traffic that is already in a defined service class would benefit from being further differentiated based on QB vs NQB behavior (like, as you suggested using various AF marks), or whether certain existing service classes can be grouped into QB vs NQB meta-classes, depending on the technology limitations at each hop (MPLS, Ethernet, WiFi, LTE, DOCSIS, etc).  The result of this determination could require the use of non-default (i.e. non-standard) marking of NQB traffic in some cases.

-Greg

From: "[email-masked]" <[email-masked]>
Date: Thursday, September 5, 2019 at 2:03 AM
To: Greg White <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: AW: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Hi Greg,

Regards,

Ruediger

Thanks David.

-Greg

---

From: Steven Blake <[email-masked]>
Date: Fri, 06 Sep 2019 14:36:01 -0400

On Thu, 2019-09-05 at 19:23 +0100, Bob Briscoe wrote:
> alignment), I'll put up a straw-man NQB configuration that complies
> with the following requirement in the draft:
> Background: NQB has become possible largely because Internet access
> delay of a packet can be sub-millisecond, and therefore a queue of a
> few packets introduces delay that is small relative to other non-
> optional sources of delay like propagation. In these cases we no
> longer need priority scheduling for low delay.
> 
> Config: 
> Scheduler: 
> WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least
> 60Mb/s for NQB flows. 
> Scheduler quantum: 1500B.
> Buffering:
> The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
> The QB buffer is deeper (say 200ms) with an AQM target delay of say
> 10ms.
> 
> 2 paced flows of VoIP datagrams, each avg 50B payload plus 58B of
> bit-rate: 29kb/s /flow
> serialization delay: 7.2us / pkt
> 2 streams of 1000B game sync datagrams at 30pkt/s => 
> bit-rate: 240kb/s /flow
> serialization delay: 67us / pkt
> plus occasional DNSSec datagrams, worst case 1500B
> serialization delay: 100us / pkt
> Perhaps 540kb/s in all, which is about 0.9% of the min NQB capacity.
> Each NQB flow paces out its own packets, but one might happen to
> arrive while a packet from any of the other NQB flows is already
> queued. Worst case n-1 = 4 other NQB packets queued, where n is the
> number of application flows. And if there's traffic in the QB queue,
> each NQB packet will have to wait for one quantum (100us) while the
> other queue is served. Worst-case NQB delay is then:

The objective as I understand it is to offer low queueing latency to non-admissioned-controlled/non-capacity-seeking traffic (NQB). First, this is only feasible if NQB traffic is naturally self-limiting to a small fraction of total traffic (e.g., < 5%). Second, existence of this NQB service should not noticeably impact the QoS of capacity-seeking (QB) traffic. Third, implementations of this NQB service should try to ensure that capacity-seeking traffic trying to utilize the NQB service gets worse service with very high probability (so that it doesn't even bother to probe).

Given this, using a WFQ-ish scheduler with a high NQB scheduler weight to achieve low latency is the wrong hammer to hit this nail, IMHO. A small queue by itself is insufficient to prevent capacity-seeking traffic from trying to grab that large & isolated share of capacity. If I were implementing this (and only this) I would consider putting NQB traffic in a bounded strict priority queue, feeding it priority credits at say 2-4% of link capacity. I would also put a policer in front of that queue set at a rate of 4-6% link capacity, with a large enough token bucket to let small NQB bursts through. That is a very simple implementation, available in lots of existing hardware, that achieves the stated objectives. 

DoS-ing NQB traffic in this implemention is "easier" than DoS-ing it in your proposed scheduling approach, but a targeted NQB DoS attack will have less impact on capacity-seeking traffic, and DoS-ing residential access links using only best-effort service is already a thing, so I'm not sure it really matters.

Regards,

// Steve

---

From: Greg White <[email-masked]>
Date: Fri, 6 Sep 2019 19:53:50 +0000

[GW] comments below

From: tsvwg <[email-masked]> on behalf of Steven Blake <[email-masked]>

[GW] If the WRR weight is 0.5 for each queue, then it is feasible for the NQB PHB to provide low queueing latency to NQB traffic up to 50% of the total link capacity (and more than 50% when QB traffic is not filling up the other 50%).

[GW] On the second point, you are correct, the existence of the NQB service will not noticeably impact the QoS of capacity-seeking QB traffic.  For example, if the NQB traffic did happen to be smooth traffic (CBR) at exactly 50% of link capacity, then with NQB, the QB traffic flows have 50% of link capacity to compete with each other for. Let’s say that competition results in (e.g.) 0.1% packet drop/mark.  On the other hand, without the NQB service, and in the presence of the same CBR flow, the capacity-seeking traffic flows similarly compete for the remaining 50% of link capacity, and settle in to the same 0.1% packet drop/mark rate (yet the CBR flow is unfortunately also subject to high queuing delay as well as 0.1% drop if the link doesn’t support ECN).  Well, being picky, if the link didn’t support ECN then I guess the capacity seeking traffic would compete for 50.1% of link capacity without the NQB PHB, as opposed to 50% with the NQB PHB, but I would argue that isn’t a “noticeable impact”.

[GW] On your third point, I agree in concept, and it is an aspect that we’ve discussed privately several times.  At the end of the day, we could not convince ourselves of an appropriate “punishment” for mismarked QB traffic, other than high packet loss in the case where QP is not implemented or potential for out-of-order delivery when QP is implemented.

[GW] But priority isn’t necessary, and it results in the need for limitations on NQB bandwidth.  For example, in your implementation, what would happen when the arrival rate at the link is only 20% of link capacity but all of it is NQB?  I guess the policer would unnecessarily discard a large percentage of that traffic.

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 7 Sep 2019 12:48:01 +0200

Hi Greg,

> On Sep 6, 2019, at 19:18, Greg White <[email-masked]> wrote:
> 
> Hi Ruediger,
>  

	[SM] To summarize, if we can be sure that NQB traffic does NOT contain any of the following traffic types

 |    Low-       | AF21 |          |             |                    |
  |    Latency    | AF22 | RFC 2597 |     3       | AC_BE (Best Effort)|
  |    Data       | AF23 |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |     OAM       | CS2  | RFC 2474 |     0       | AC_BE (Best Effort)|
  +---------------+------+----------+-------------+--------------------+
  |    High-      | AF11 |          |             |                    |
  |  Throughput   | AF12 | RFC 2597 |     0       | AC_BE (Best Effort)|
  |    Data       | AF13 |          |             |                    |
  +---------------+------+----------+-------------+--------------------+
  |   Standard    | DF   | RFC 2474 |     0       | AC_BE (Best Effort)|
  +---------------+------+----------+-------------+--------------------+
  | Low-Priority  | CS1  | RFC 3662 |     1       | AC_BK (Background) |
  |     Data      |      |          |             |                    |
  +--------------------------------------------------------------------+

then AC_VI can be considered in agreement with https://www.rfc-editor.org/rfc/rfc8325.txt, otherwise your only option is AC_BE. 
Really, if you decide to label a set of diverse flows with one identifier, then for wifi scheduling purposes you must restrict yourself to the lowest priority access class of any of the possible members of your set. If you dislike this, define and enforce your set, such that all members fulfill the rfc8325 requirements.
Just to be clear, e.g. DNS traffic which you seem to give as a candidate for NQB "(examples: online games, voice chat, DNS lookups)" falls explicitly into the OAM category in rfc8325 and hence does not justify AC_VI or AC_VO, so either exclude DNS from NQB or restrict yourself to AC_BE.

In the light of the default dscp to AC mapping this directly implies that the selected codepoint 0x2A is NOT SUITABLE as dscp for NQB, you need to restrict yourself to a dscp out of the AC_BE set (000xxx, 011xxx)

Sidenote, since rfc8325 are the normative IETF recommendations for dscp to ieee 802.11 ACs recommending both AC_VI amd AC_VO in section 8.3 of the draft makes little to no sense.

@chairs, do you agree with my interpretation of the rfc8325 requirements?

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 7 Sep 2019 17:48:53 +0200

Dear Bob,

> It's v unlikely that this worst-case would arise, but it gives an indication of where the tail of the delay distribution will lie.

	[SM] I just re-thought this section, and believe it to be a strawman argument. 
In my reality the NQB draft states:
"7.  Relationship to L4S
   The dual-queue mechanism described in this draft is intended to be compatible with [I-D.ietf-tsvwg-l4s-arch]."

In turn [I-D.ietf-tsvwg-l4s-arch], drags in I-D.ietf-tsvwg-aqm-dualq-coupled which sizes the shared dualQ buffer to allow for 250ms at the egress bandwidth ("limit = MAX_LINK_RATE * 250 ms               % Dual buffer size"), and not 3 ms.
In the example above it is ONLY the extremely shallow buffer of 3 ms that "aligns" the incentives such that QB traffic gets unhappy in the NQB queue. So either  -D.ietf-tsvwg-aqm-dualq-coupled needs to change, or your argument above needs adaption to 250ms. And even if we would allow 50% of that for the QB queue, that still leaves 125ms. Since you are heavily involved in the other L4S drafts this is a rather surprising lack of candidness about how the NQB queue is actually supposed to be operated. 
Feel free to demonstrate again how with a 125 ms buffer (that is 0.125/((1500*8)/(120*1000^2)) = 1250 packets or 1250*1500/1000^2 = 1.875 MB, a whole lot of flows are one with their business long before they transmitted 1,8 MB)  there are no incentives for QB flows to mark themselves as NQB unless there is a stringent monitor and enforcement regime operated at the bottleneck NQB queue operating element to weed out misbehaving flows. But please plug in the numbers that the l4S family of drafts actually recommend to use, so it becomes relevant for this draft.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 7 Sep 2019 21:14:34 +0200

> On Sep 6, 2019, at 21:53, Greg White <[email-masked]> wrote:
> 
> [GW] comments below
>  

	[SM] The NQB PHB is not going to provide low queueing latency to NQB traffic, the whole "trick" behind the low queueing latency, IMHO effectively is to mandate pacing and ideally synchronize all active senders such that each of their packet's arrive only shortly before its egress timeslot at the bottleneck becomes available. For Bob's examples like VoIP that trick will only work partially, as the VoIP senders will pick their sending slots based on acquisition start time and sampling rate and will not adapt to what would be ideal for the bottleneck's scheduler (VoIP being UDP there is not even a viable path from the bottleneck AQM back to the sender to synchronize them). Using NTP as a proxy for assessing synchronisation quality and jitter it seems to me that <1ms synchronization over the open internet is going to be hard given the transient nature of most flows, but hard does not equal impossible.

>  
> [GW] On the second point, you are correct, the existence of the NQB service will not noticeably impact the QoS of capacity-seeking QB traffic.  
> For example, if the NQB traffic did happen to be smooth traffic (CBR) at exactly 50% of link capacity, then with NQB, the QB traffic flows have 50% of link capacity to compete with each other for. Let’s say that competition results in (e.g.) 0.1% packet drop/mark.  On the other hand, without the NQB service, and in the presence of the same CBR flow, the capacity-seeking traffic flows similarly compete for the remaining 50% of link capacity, and settle in to the same 0.1% packet drop/mark rate (yet the CBR flow is unfortunately also subject to high queuing delay as well as 0.1% drop if the link doesn’t support ECN).  Well, being picky, if the link didn’t support ECN then I guess the capacity seeking traffic would compete for 50.1% of link capacity without the NQB PHB, as opposed to 50% with the NQB PHB, but I would argue that isn’t a “noticeable impact”.

	[SM] And now entertain the idea of a flow fair queueing system on the bottleneck which nicely would isolate the effect of CBR traffic exceeding its permitted rates from the rest of the flows...

> [GW] On your third point, I agree in concept, and it is an aspect that we’ve discussed privately several times.  At the end of the day, we could not convince ourselves of an appropriate “punishment…

	[SM] With the actual numbers from the DualQ RFC it seems that a QB flow will be able to deposit at least up to 125ms worth of packets before hitting the overload-limit, at Bob's 120 Mbps that is roughly 1.8 MB of data, quite a number of TCP flows will be done with their business before hitting that, reaping potential benefit from mis-marking themselves, without encountering negative effects from doing so.

---

From: <[email-masked]>
Date: Mon, 9 Sep 2019 06:56:52 +0000

Hi Greg,

to me that reads as if you are mainly interested in a specified NQB DSCP for the last mile. A packet received at interconnection with a DSCP which hasn’t been agreed between sending and receiving domain will be re-written (speaking for the part of the network I work with).

That’s ok, as DiffServ involves policy management anyway.

Regards,

Ruediger

Von: Greg White <[email-masked]>
Gesendet: Freitag, 6. September 2019 19:18
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Hi Ruediger,

DSCP
WMM Access Category
001xxx, 010xxx
Background (AC_BK)
000xxx, 011xxx
Best Effort (AC_BE)
10xxxx
Video (AC_VI)
11xxxx
Voice (AC_VO)

-Greg

From: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date: Thursday, September 5, 2019 at 2:03 AM
To: Greg White <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: AW: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Hi Greg,

Regards,

Ruediger

Von: tsvwg <[email-masked]<mailto:[email-masked]>> Im Auftrag von Greg White
Gesendet: Donnerstag, 5. September 2019 01:02
An: Black, David <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Thanks David.

-Greg

---

From: Steven Blake <[email-masked]>
Date: Mon, 09 Sep 2019 12:31:30 -0400

Resending because for some reason this didn't reach the tsvwg list.

// Steve

Subject: Re: [tsvwg] Traffic protection as a hard requirement for NQB
From: Steven Blake <[email-masked]>
To: Greg White <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Date: Sun, 08 Sep 2019 11:49:58 -0400

> [GW] comments below
Greg,
I don't know how to reconcile any of this below with the text of your
draft (specifically Secs. 1 & 3), which talks about low-bandwidth
applications, specifically those over UDP. Are you contending that the
sum of these adds up to a significant fraction of traffic on any link?
> 
> 
> From: > tsvwg <[email-masked]> on behalf of Steven Blake <[email-masked]>
> 

> 
> 
> The objective as I understand it is to offer low queueing latency to non-admissioned-controlled/non-capacity-seeking traffic (NQB). First, this is only feasible if NQB traffic is naturally self-limiting to a small
>  fraction of total traffic (e.g., < 5%). Second, existence of this NQB service should not noticeably impact the QoS of capacity-seeking (QB) traffic. Third, implementations of this NQB service should try to ensure that capacity-seeking traffic trying to utilize
>  the NQB service gets worse service with very high probability (so that it doesn't even bother to probe).
> 

> [GW] If the WRR weight is 0.5 for each queue, then it is feasible for the NQB PHB to provide low queueing latency to NQB traffic up to 50% of the total link capacity (and more than 50% when QB traf…

That claim is technically false. You cannot achieve 100% utilization on
any link (or link slice) at low queueing delay for uncoordinated, non-
admissioned controlled traffic. I'll bet you cannot find a carrier
anywhere that allocates 50% (or even 20%) of capacity on their metro or
long-haul networks to admissioned-controlled & tarriffed EF traffic. So
I don't see them doing so for non-admissioned controlled/non-tarriffed
traffic.

> 
>  
> 
> [GW] On the second point, you are correct, the existence of the NQB
> service will not noticeably impact the QoS of capacity-seeking QB
> traffic.  For example, if the NQB traffic did happen to be smooth
> traffic (CBR) at exactly 50% of link
>  capacity, then with NQB, the QB traffic flows have 50% of link
> capacity to compete with each other for. Let’s say that competition
> results in (e.g.) 0.1% packet drop/mark.  On the other hand, without
> the NQB service, and in the presence of the same CBR flow,
>  the capacity-seeking traffic flows similarly compete for the
> remaining 50% of link capacity, and settle in to the same 0.1% packet
> drop/mark rate (yet the CBR flow is unfortunately also subject to
> high queuing delay as well as 0.1% drop if the link doesn’t
>  support ECN).  Well, being picky, if the link didn’t support ECN
> then I guess the capacity seeking traffic would compete for 50.1% of
> link capacity without the NQB PHB, as opposed to 50% with the NQB
> PHB, but I would argue that isn’t a “noticeable impact”.
If I am writing a capacity seeking application, and I believe that
there is a lightly utilized link slice of 20%-50% of capacity where I
can grab more than my fair share, I would be highly tempted to setup
two connections, one-NQB marked and one not, and let them each grab as
much capacity as possible. Which benefits my app until everyone else
does the same, and then we are back to square one.
> 
> [GW] On your third point, I agree in concept, and it is an aspect that we’ve discussed privately several times.  At the end of the day, we could not convince ourselves of an appropriate “punishment” for mismarked QB traffic, other than
>  high packet loss in the case where QP is not implemented or potential for out-of-order delivery when QP is implemented.
> 
>  
> 
> Given this, using a WFQ-ish scheduler with a high NQB scheduler weight to achieve low latency is the wrong hammer to hit this nail, IMHO. A small queue by itself is insufficient to prevent capacity-seeking traffic
>  from trying to grab that large & isolated share of capacity. If I were implementing this (and only this) I would consider putting NQB traffic in a bounded strict priority queue, feeding it priority credits at say 2-4% of link capacity. I would also put a policer
>  in front of that queue set at a rate of 4-6% link capacity, with a large enough token bucket to let small NQB bursts through. That is a very simple implementation, available in lots of existing hardware, that achieves the stated objectives. 
> 

> 
> 
>  
> 
> [GW] But priority isn’t necessary, and it results in the need for limitations on NQB bandwidth.  For example, in your implementation, what would happen when the arrival rate at the link is only 20% of link capacity but all of it is NQB? 
>  I guess the policer would unnecessarily discard a large percentage of that traffic. 
> 

> 
> 
> DoS-ing NQB traffic in this implemention is "easier" than DoS-ing it in your proposed scheduling approach, but a targeted NQB DoS attack will have less impact on capacity-seeking traffic, and DoS-ing residential
>  access links using only best-effort service is already a thing, so I'm not sure it really matters.
> 

How do you estimate how much NQB traffic there mi

Regards,

// Steve

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 9 Sep 2019 23:29:11 +0300

> On 9 Sep, 2019, at 7:31 pm, Steven Blake <[email-masked]> wrote:
> 
> How do you estimate how much NQB traffic there might be? How would you tune this? My assumption is that this whole scheme only survives if the amount of NQB traffic is always so small relative to any link's capacity (< 5%) that you just allocation a small fraction of link capacity to NQB and forget about it. In which case, WRR scheduling doesn't give you good latency because your HOL NQB packet might have to wait for >= 20 MTU/link rate to get serviced.

I think we are also dancing around a more general problem here.  If we are having to use straw-man implementations to argue about the merits of the PHB, that implies that we don't have a real implementation to experiment with.

I would advise the authors to come up with a real implementation that has the behaviour they want and adequate robustness, and then we can re-open the discussion with "running code" to work from.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Sep 2019 00:24:31 +0100

Jonathan,

> It would only remain at 3ms if there was no BE traffic competing while the NQB queue was saturated - a somewhat unlikely scenario if the NQB marking is used as intended.

[BB] You're right that I said it like this, and I was wrong.

A better way of implementing this would be a 3ms limit on sojourn time, 
which would remain at 3ms whatever the drain-rate.
> The gain to the attacker *is* the incentive - which I did my best to illustrate through the narrative device of NUR, their motivations, and the evolution of their methods of maximising throughput f…
[BB] Yes, I've said you are right that there is an incentive. But here 
I'm saying you have constructed motives that are not /solely/ due to 
self-interest.

With bandwidth sharing (and in game theory and economics more generally) 
it's normal to deal with self-interest separately from malice, because 
self-interest is pervasive, whereas setting out to deprive others 
irrespective of gain for yourself is not so common.

For instance, if A gets 10Mb/s more than B, any rational individual is 
expected to derive value from the extra 10Mb/s they got. However, it's 
less common for someone to also derive additional value from the mere 
fact that B got less. That's the preserve of sociopathic individuals or 
where A and B happen to be business competitors at a higher layer (in 
addition to the basic lower layer contention for network resources).

The NUR scenario you've constructed contains a mix of self-interest and 
malice. I'm not saying that makes the scenario invalid, just that the 
draft seems to be trying to follow the more usual approach of separating 
these the two types of motivation.
> I have not yet had the opportunity to try out BBRv2 in practice and in competition with other CC algos.  However, it is a reasonable choice for NUR simply because it achieves near-perfect link util…
[BB] That is rather starry eyed. I believe that would only work with a 
deeper buffer and stable conditions. We'd have to try it, but if the 
bandwidth probes hit a shallow buffer (experiencing high loss), I 
believe BBRv2 is designed to run at lower utilization than with a deep 
buffer.
> NUR's subsequent tweak to accept 10% loss is *not* attributable to Google
[BB] I didn't say that. I said that starting from the assumption that 
loss below some level can be ignored, is attributable to Google.
> - they chose a 1% default for good reason - but is simply NUR's own sociopathy showing itself when they started to encounter contention on some last-mile links.  Because BBRv2 understands ECN (to some extent, anyway),
[BB] My strawman NQB qdisc does not support any form of AQM, let alone 
ECN. Why do you assume that NQB gives out ECN signals?

I am trying to check that NQB stands on its own merits, with no 
dependency on L4S.

> it would not run into significant loss when encountering the NQB-aware qdisc you described while still marked BE.
[BB] Eh? How does BBRv2 get classified into the NQB queue if its DSCP is 
BE?
> Yes, and that is what later movers following NUR's example might notice, if they bother to actually measure the effect, which most might not - they're just blindly copying some tweak they don't rea…
My argument is against saying NQB MUST implement traffic protection. I'm 
happy to help improve understanding of the incentives around NQB, but my 
argument concerns IETF procedure, irrespective of whether there are 
effective attacks or incentives.

You've not addressed my argument yet. I was hoping you might at some point.
> If there's 0-1 BE flows contending for the link, then NUR will get similar throughput in either case when using BBRv2, because WRR allows either child queue to grow into capacity left unused by the other, and BBR tries to keep the bottleneck queue empty on average.
[BB] In your "starry-eyed" view of BBRv2, it perfectly utilizes 
bandwidth no matter how shallow the buffer. Given your argument depends 
on this, you ought to check. I think you'll find it underutilizes when 
the buffer is shallow, which will make throughput worse in the common 
cases with 0 or 1 long-running QB flow.

> This might be less true if some other CC algo was used, but I note that the 6ms capacity of the NQB side is not very different from the 10ms target of the BE side, unless the path latency is exceptionally low.
[BB] You can note that 6ms is close to 10ms if you want, but you're 
comparing apples with mashed potatoes. The strawman configuration was as 
follows:

                     NQB    QB
Buffer limit  3ms    200ms
AQM target  n/a     10ms

A buffer limit (absolute cap to any variations) is not comparable to an 
AQM target (long-running convergence).

Only the 200ms is dependent on the drain rate. 10ms is absolute, and so 
is 3ms now I've suggested it would be a sojourn limit.

Your scenario was potentially enlightening, but can I ask that you 
reconstruct it, this time using our new mutual understanding of the 
strawman qdisc. Please also separate out the self-interest and malice 
parts of the motivations and please consider a range of likely 
background traffic, not just the one that makes your scenario work.
> I'm sure you've encountered a similar setup in the real world.
[BB] Similar setups have come and gone, but none has endured so far in 
the real world. I also used to continually push this 'project fear' 
line, but I decided to take time out to consider why none of these 
setups had endured.

I suspect it's for two reasons: i) network schedulers control 
inter-customer capacity; ii) flows have two ends.

Because of i), setups like NUR only alter intra-customer shares of 
capacity.
Because of ii), altho service providers choose congestion controllers, 
end-customers (at the other end) choose service providers.

whenever a service provider takes a larger share, it can only be at the 
expense of other service providers being used by the same end-customer. 
So, end-customers have no reason to prefer the NURs of this world, which 
wither and die.

> But the IETF gets to say whether your specification is approved for publication as an RFC.  As part of that decision, they must consider the possible negative effects on the Internet as well as the…
[BB] Understood and thank you. However, the IETF tries not to 
over-constrain an implementation, beyond the needs of interoperability. 
That's a major difference between the roles of IETF and Linux. Mandating 
security over-constrains implementers wherever the security concern is 
not universal.

Regards

Bob

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 10 Sep 2019 11:38:38 +0200

> On Sep 10, 2019, at 01:24, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
> 
> A better way of implementing this would be a 3ms limit on sojourn time, which would remain at 3ms whatever the drain-rate.

	[SM] Under the assumption that the 3ms have any bearing on the actual intended use for the NQB mark that is. As written in another post, in reality we are rather talking about 125 to 250 milliseconds worst case buffering.

> [BB] Yes, I've said you are right that there is an incentive. But here I'm saying you have constructed motives that are not /solely/ due to self-interest.

	[SM] Does this really matter, why the incentives are not aligned unless we can fix them?

> 
> With bandwidth sharing (and in game theory and economics more generally) it's normal to deal with self-interest separately from malice, because self-interest is pervasive, whereas setting out to deprive others irrespective of gain for yourself is not so common.

	[SM] Please do not forget the DDOS as a service crowd (or rather network stress-testing as a service), they will game NQB out of self-interest to reduce the cost for their operations, the malice part is restricted to their customers.

> 
> For instance, if A gets 10Mb/s more than B, any rational individual is expected to derive value from the extra 10Mb/s they got. However, it's less common for someone to also derive additional value from the mere fact that B got less. That's the preserve of sociopathic individuals or where A and B happen to be business competitors at a higher layer (in addition to the basic lower layer contention for network resources).
> 
> The NUR scenario you've constructed contains a mix of self-interest and malice. I'm not saying that makes the scenario invalid, just that the draft seems to be trying to follow the more usual approach of separating these the two types of motivation.
>> 
> [BB] That is rather starry eyed. I believe that would only work with a deeper buffer and stable conditions. We'd have to try it, but if the bandwidth probes hit a shallow buffer (experiencing high loss), I believe BBRv2 is designed to run at lower utilization than with a deep buffer.
>> 
> [BB] I didn't say that. I said that starting from the assumption that loss below some level can be ignored, is attributable to Google.

	[SM] This is a stance BBR takes that is quite problematic in itself, given that it purposefully ignores drops intended to signal slow-down...

> [BB] My strawman NQB qdisc does not support any form of AQM, let alone ECN. Why do you assume that NQB gives out ECN signals?

	[SM] I would very much prefer we would discuss the actual intended use-cases for NQB instead of carefully constructed strawman configurations.

> 
> I am trying to check that NQB stands on its own merits, with no dependency on L4S.

	[SM] The draft cautiously argues:
"7.  Relationship to L4S
   The dual-queue mechanism described in this draft is intended to be compatible with [I-D.ietf-tsvwg-l4s-arch]."

But it does not mention any other queueing design at all, so I took this as a reserved way of saying, this is mainly designed to complement L4S.

> [BB] Eh? How does BBRv2 get classified into the NQB queue if its DSCP is BE?

	[SM] if a flow accepts Jonathan's 1-% loss before slowing down, it will also encounter the same loss on the QB queue, as by design it will go up against its ceiling, being a TCP and all?

> My argument is against saying NQB MUST implement traffic protection.

	[SM] Yes, but once you do this the whole based-on-observable-behavior justification for NQB disintegrates. 

> I'm happy to help improve understanding of the incentives around NQB,

	[SM] But you just admitted above that in reality the incentives for QB flows to avoid the NQB flow are dubious, which, if you still want to salvage the whole NQB concept, indicates that you need to leverage the observable behavior angle, no?

> but my argument concerns IETF procedure, irrespective of whether there are effective attacks or incentives.

	[SM] Unless the IETF is in the business of building better DDOS attack vectors, I fail to see how queue protection can be anything but mandatory?

> [BB] In your "starry-eyed" view of BBRv2, it perfectly utilizes bandwidth no matter how shallow the buffer. Given your argument depends on this, you ought to check. I think you'll find it underutilizes when the buffer is shallow, which will make throughput worse in the common cases with 0 or 1 long-running QB flow.

	[SM] Jonathan proposed a heavily tuned TCP based on BBRv2 not the canonical BBR.

> [BB] You can note that 6ms is close to 10ms if you want, but you're comparing apples with mashed potatoes. The strawman configuration was as follows:
> 
>                     NQB    QB
> Buffer limit  3ms    200ms
> AQM target  n/a     10ms
> 
> A buffer limit (absolute cap to any variations) is not comparable to an AQM target (long-running convergence).
> 
> Only the 200ms is dependent on the drain rate. 10ms is absolute, and so is 3ms now I've suggested it would be a sojourn limit.
> 
> 
> Your scenario was potentially enlightening, but can I ask that you reconstruct it, this time using our new mutual understanding of the strawman qdisc. Please also separate out the self-interest and malice parts of the motivations and please consider a range of likely background traffic, not just the one that makes your scenario work.

	[SM] Would it not be a better use of everybody's time, if we would shelf the strawman qdisc discussion, and actually discuss these issues in the light of the intended real AQM configurations for real users of this feature instead? As far as I can see, and I happily repeat myself until ACKd/NACKd, with the 125-250ms as proposed in the dualQ draft, this whole sub-discussion changes considerably, even for shorter flows.

> [BB] Similar setups have come and gone, but none has endured so far in the real world. I also used to continually push this 'project fear' line, but I decided to take time out to consider why none of these setups had endured.

	[SM] As an aside, quite a number of CDNs are willing to use non-standards TCP-behavior (like iw >> 10) to improve their perceived value proposition and are still doing this (see https://www.researchgate.net/publication/328519615_Demystifying_TCP_Initial_Window_Configurations_of_Content_Distribution_Networks). This is not as extreme a modification as what Jonathan proposes, but seems to endure over time...

> 
> I suspect it's for two reasons: i) network schedulers control inter-customer capacity; ii) flows have two ends.
> 
> Because of i), setups like NUR only alter intra-customer shares of capacity.
> Because of ii), altho service providers choose congestion controllers, end-customers (at the other end) choose service providers.
> 
> whenever a service provider takes a larger share, it can only be at the expense of other service providers being used by the same end-customer. So, end-customers have no reason to prefer the NURs of this world, which wither and die. 

	[SM] I would expect if two typical end-customer's use two competing video sources and one of them stutters in concurrent use, many will dump the stuttering one (even if this is caused by reckless behavior of the other). I think that folding back the consequence of reckless behavior onto the mischievous actor might have a faster change of behavior...

> [BB] Understood and thank you. However, the IETF tries not to over-constrain an implementation, beyond the needs of interoperability.

	[SM] I venture a guess here, not being an IETF member, that the IETF will constrain an implementation if not doing so can have drastic consequences of other flows.

> That's a major difference between the roles of IETF and Linux. Mandating security over-constrains implementers wherever the security concern is not universal.

	[SM] This sounds great, but in effect you are risking to establish a new cheaper DOS vector, which has some security implications. Unless your argument is, that with the easy to target and disrupt NQB queue DOS attacks will require significantly less traffic and hence the rest of the internet is better off?

Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Sep 2019 16:13:35 +0100

Sebastian,

> Feel free to demonstrate again how with a 125 ms buffer (that is 0.125/((1500*8)/(120*1000^2)) = 1250 packets or 1250*1500/1000^2 = 1.875 MB, a whole lot of flows are one with their business long b…
You're right that, when the NQB draft says it is compatible with 
l4s-arch, it means it is compatible with the structure of the 
architecture, not necessarily the specific recommended numbers in one of 
the example implementations in aqm-dualq-coupled. Indeed, in Appendix A 
of aqm-dualq-coupled, note b says that two separate buffers might be 
preferred to a shared 250ms buffer [and the Low Latency buffer would not 
need to be more than a few ms].

That said, here's a strawman (i.e. not tested, but for you to bash) for 
adding NQB support to the DualPI2 example in appendix A of 
aqm-dualq-coupled:

* Classify NQB packets as well as L4S ECN into the Low Latency queue
* Don't subject NQB packets to the L4S AQM (unless their ECN field is 
also L4S).
* During overload, subject all packets to the same drop prob (Classic, 
L4S and NQB)
* add a 3ms sojourn limit for packets marked NQB (but not L4S), after 
which they are dropped
* It might be necessary to tune down the L4S scheduling weight to 
bolster the incentive against mismarking, e.g. 3/4 not 15/16.

The alternative specified for DOCSIS is to use a 10ms low latency buffer 
but also to implement per-flow queue protection (which can be disabled 
by the operator).

HTH

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Sep 2019 17:55:29 +0100

Sebastian,

On 07/09/2019 20:14, Sebastian Moeller wrote:
> 	[SM] The NQB PHB is not going to provide low queueing latency to NQB traffic, the whole "trick" behind the low queueing latency, IMHO effectively is to mandate pacing and ideally synchronize all active senders such that each of their packet's arrive only shortly before its egress timeslot at the bottleneck becomes available.
The whole trick is the first part (pacing), but the second (ideally 
synchronize) is not part of the trick and not necessary.
> For Bob's examples like VoIP that trick will only work partially, as the VoIP senders will pick their sending slots based on acquisition start time and sampling rate and will not adapt to what would be ideal for the bottleneck's scheduler (VoIP being UDP there is not even a viable path from the bottleneck AQM back to the sender to synchronize them). Using NTP as a proxy for assessing synchronisation quality and jitter it seems to me that <1ms synchronization over the open internet is going to be hard given the transient nature of most flows, but hard does not equal impossible.
Randomness is our friend. For a certain NQB utilization, r, if there are 
two equal but uncoordinated NQB flows the probability of a queue of 2 
packets is r/2. But if there are n equal but uncoordinated flows, the 
probability of a queue of n is vanishingly small ((r/n)^(n-1), I think).

That's why no-one attempts to get VoIP flows or flows of game-sync 
datagrams to shift their phase to reduce overall delay. Randomness is 
sufficient.

The calculations I showed with 5 flows gave roughly 600us worst-case 
queuing delay without any attempt at desynchronizing each of the flows. 
And the probability of even that much would be tiny.

> 	[SM] And now entertain the idea of a flow fair queueing system on the bottleneck which nicely would isolate the effect of CBR traffic exceeding its permitted rates from the rest of the flows...
[BB] I believe Greg's point was that the CBR flow at 50% of the capacity 
might have been desired by the user.

> 	[SM] With the actual numbers from the DualQ RFC it seems that a QB flow will be able to deposit at least up to 125ms worth of packets before hitting the overload-limit, at Bob's 120 Mbps that is r…
[BB] Already addressed in response to your other email.

Bob

---

From: Greg White <[email-masked]>
Date: Tue, 10 Sep 2019 18:29:32 +0000

Hi Ruediger,

Yes, my focus (both from the perspective of the systems that I work on, and from the perspective of where I think NQB differentiation is most valuable) is on last mile and WiFi.  In terms of interconnection, there could be multiple ways to handle this.  My view is that the endpoint application has the best a priori knowledge of its sending behavior, so it is good to leverage that (and either validate it or not, depending on what is most appropriate for the situation).   This would argue for networks to adopt a policy of not bleaching the NQB value on traffic arriving from interconnections, and utilizing queue protection if necessary to validate NQB behavior.   That said, there are other ways that network operators can identify NQB traffic without solely relying on endpoint marking (and DSCP preservation across interconnects).  I know some operators are investigating such techniques.

-Greg

Date: Monday, September 9, 2019 at 12:57 AM

Hi Greg,

Regards,

Ruediger

Hi Ruediger,

DSCP
WMM Access Category
001xxx, 010xxx
Background (AC_BK)
000xxx, 011xxx
Best Effort (AC_BE)
10xxxx
Video (AC_VI)
11xxxx
Voice (AC_VO)

-Greg

Hi Greg,

Regards,

Ruediger

Thanks David.

-Greg

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 10 Sep 2019 21:23:17 +0200

Hi Greg,

> On Sep 10, 2019, at 20:29, Greg White <[email-masked]> wrote:
> 
> Hi Ruediger,
>  
> Yes, my focus (both from the perspective of the systems that I work on, and from the perspective of where I think NQB differentiation is most valuable) is on last mile and WiFi.  

	[SM] Except with the current very loose definition of which flows qualify for NQB marking, NQB seems to only merit AC_BE, which is already the default, so is NQB actually going to be valuable for wifi?

> In terms of interconnection, there could be multiple ways to handle this.  My view is that the endpoint application has the best a priori knowledge of its sending behavior, so it is good to leverage that (and either validate it or not, depending on what is most appropriate for the situation).  

	[SM] not validating the behavior but giving special powers (like the originally proposed AC-VI/AC_VO on wifi) seems like a genuinely optimistic ideas. 

> This would argue for networks to adopt a policy of not bleaching the NQB value on traffic arriving from interconnections, and utilizing queue protection if necessary to validate NQB behavior.   That said, there are other ways that network operators can identify NQB traffic without solely relying on endpoint marking (and DSCP preservation across interconnects).  I know some operators are investigating such techniques.  

	[SM] if the name has any relevance, one could treat each flow as non-queue-building until there is enough evidence to change that classification (like fq_codel treats all new flows as sparse initially until their observed bahavior merits a different classification).

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Tue, 10 Sep 2019 22:16:57 +0200

Dear Bob,

> On Sep 10, 2019, at 17:13, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> You're right that, when the NQB draft says it is compatible with l4s-arch, it means it is compatible with the structure of the architecture, not necessarily the specific recommended numbers in one of the example implementations in aqm-dualq-coupled.

	[SM] Well, as far as I can tell these are the only numbers around.

> Indeed, in Appendix A of aqm-dualq-coupled, note b says that two separate buffers might be preferred to a shared 250ms buffer [and the Low Latency buffer would not need to be more than a few ms].

	[SM] Indeed, except I can not find the part in brackets in that draft. Setting the Low Latency buffer to only a few milliseconds, will then make the whole low latency queue susceptible of the attack scenario I described before, is that really an improvement (and helping your cause])? 

> 
> That said, here's a strawman (i.e. not tested, but for you to bash) for adding NQB support to the DualPI2 example in appendix A of aqm-dualq-coupled:
> 
> * Don't subject NQB packets to the L4S AQM (unless their ECN field is also L4S).

	[SM] You are talking about the ECN marking component only here?

> * During overload, subject all packets to the same drop prob (Classic, L4S and NQB)

	[SM] Is there any other option, unless one wants to effectively starve all other traffic "classes"?

> * add a 3ms sojourn limit for packets marked NQB (but not L4S), after which they are dropped

	[SM] Great idea, looking forward seeing this in the next dualQ draft! Until then I will assume 250ms/2 as the likely default value. I also note that this adds another complication to dualQ, basically multiplexing a dropping regime into an ECN based system.

> * It might be necessary to tune down the L4S scheduling weight to bolster the incentive against mismarking, e.g. 3/4 not 15/16.

	[SM] You mean the part in the dualQ draft that says " if L4S traffic is over-aggressive or unresponsive, the scheduler weight for Classic traffic will at least be large enough to ensure it does not
      starve."? I would not actually direct attention to this part of the dualQ rfc, as for a system that claims to share between L4S and standard-compliant flows only giving 1/16th to the standard-compliant flows is a problem.

> The alternative specified for DOCSIS is to use a 10ms low latency buffer but also to implement per-flow queue protection (which can be disabled by the operator).

	[SM] If we are talking about [DOCSIS-MULPIv3.1]  Cable Television Laboratories, Inc., "MAC and Upper Layer Protocols Interface Specification, CM-SP-MULPIv3.1-I18-190422", April 22, 2019,
              <https://specification-search.cablelabs.com/ CM-SP-MULPIv3.1>.

In Annex N I read: I read: 
BUFFER_SIZE                // The size of the buffer for the LL service flow [B]
                           //  (see Section C.2.2.7.11.5). A value of 10 ms multiplied
                           // by the Maximum Sustained Rate (MAX RATE) is recommended

Naively, I would say this is only 10 ms if MAX RATE equals 1, no? What am I missing.

Best Regards
	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Sep 2019 22:25:29 +0100

Sebastian,

On 06/09/2019 10:51, Sebastian Moeller wrote:
> 	[SM] I fail to see how this is a fitting analogy to this issue at hand.
[BB] It is a highly fitting analogy, so if you fail to see it, please 
try harder. My wife just read this, and said, "What's not to understand? 
That's an extremely relevant analogy."

In life, most trust is based on verifiability, but rarely actually verified.
> The NQB draft defines NQB-ness as " the NQB designation and marking would be intended to convey verifiable traffic behavior, not needs or wants." IMHO either get rid of this section or mandate the verification.
No, it's important that it is verifiable. It's also important that it 
doesn't have to be verified, for reasons I've given in other emails.

For example, the case of a mobile uplink where:
a) apps are verified before being allowed into the app-store
b) and the only place where the operator could apply verification prior 
to the bottleneck radio link would be on the UE

Here continual run-time verification is both complex (b) and unnecessary 
(a). So not appropriate to mandate it.

I'm afraid I cannot continue spending time on your responses if you just 
snip the text of my arguments that you haven't argued against and then 
continue to argue against the conclusion I've drawn from those arguments 
with no rationale, just IYHO assertions.

> As I wrote in another mail, any application that can gracefully deal with the shallow buffer of the NQB queue will be fine to misclassify itself into it even if its behaviors would be verifiably ca…
[BB] Well, let's see what Jonathan comes up with in response to my 
request to reconstruct his argument, given he had misunderstood the config.

But remember that, even if the incentives are not clear-cut, I am saying 
that it is still not appropriate for the IETF to mandate a protection 
mechanism.

> 	[SM] With malice actually being a market place with its own economics. IMHO that factor alone should make queue protection mandatory, as this proposal will allow denial of service attacks much che…
[BB] You seem to be treating me as if the economics of DDoS and traffic 
security is not my area of expertise.

I assume you are aware that nearly every RFC is vulnerable to DDoS. So 
now please explain why each RFC does not mandate protection against DDoS?

It's not an oversight. It's deliberate, because one doesn't have to 
implement protection at every point in a network and at all times (think 
of the example of app-store verification above).

> 	[SM] The draft mentions access links, it does not restrict itself to access links though, so these need to be covered as well, no?
I guess so. But that's not relevant to whether 'MUST protect' is necessary.

To prove MUST is necessary, you have to prove protection is necessary in 
every scenario. That means you have to argue against the cases where I 
have shown it's /not/ necessary. Unless you have done that, there's no 
point you introducing more cases where you think protection /is/ necessary.

> But in the DOS case it is not an application of the customer that degrades the NQB queue but the attacker's.
[BB] My sentence is about the application that is harmed, not the 
application doing the harm.
> 	[SM] Sure, but in that case our hypothetical operator will have no problem whatsoever to simply ignore the RFC's MUST/SHOULD language. But making it SHOULD means the operator can do so and actuall…
[BB] The argument right from the start has been about SHOULD/MUST 
implement, not SHOULD/MUST enable. But it's similar. If it's SHOULD 
implement, the implementer can crow about it. But there's still a market 
for NQB without protection (e.g. for a mobile uplink).
> 	[SM] If a tree falls in a forrest and no one is there to witness it does it still make a sound? Only the bottleneck is guaranteed to have sufficient information to actually perform the verification, if that bottleneck does not do this it is game over;
[BB] Yes.

That is still not an argument for 'MUST protect', which would force 
implementations of NQB to protect in order to claim RFC compliance, even 
if protection is not appropriate for the deployment scenario of the 
particular NQB implementation.

> with the exception of the very narrow model of NQB-scheduling only on the access link in which case that observable behavior at the end-user will be strongly correlated with the behavior at the bottleneck (but even there it is not exactly clear which packets were dropped by the upstream NQB-scheduler and why).
[BB] Access links are the primary and often expected to be the only 
point on a path through the Internet where NQB would be deployed. See 
earlier response about how operators design their networks to ensure the 
bottleneck is usually at a single point that they can easily control.
> 	[SM] Given your explanation, the draft maybe should clarify that it only targets access links, otherwise this argument is not very convincing.
[BB] At present the draft concerns the bottleneck link of a path, which 
it says is frequently the access network. I think an NQB 
per-hop-behaviour could be applicable to any bottleneck, but the 
discussion of incentives and protection in the draft is mostly specific 
to cases where the bottleneck is an access network link (as in the three 
use-cases given).

So if you mean that the authors could say that the protection discussion 
applies primarily to access networks, I would agree.

Regards

Bob

---

From: Greg White <[email-masked]>
Date: Tue, 10 Sep 2019 21:28:21 +0000

Hi Sebastian, 

See below marked [GW].

On 9/10/19, 1:23 PM, "Sebastian Moeller" <[email-masked]> wrote:
    Hi Greg,

[GW] What validation is done today for applications that mark their packets as AC_VO or AC_VI?  I think that WMM is another one of those cases where the tragedy of the commons seems imminently possible, yet it largely hasn't occurred.  But, I hear you about being more clear about which flows are intended to be marked NQB. We certainly don't want to recommend something that creates the tragedy. Perhaps we've left it too ambiguous in the draft, so that needs some review.  It was not the intent that capacity-seeking flows (even L4S) mark their packets as NQB.  The target flows would be ones where the sender has a degree of confidence that it will not exceed the available capacity of the path.  Also, I think validation of NQB behavior is a very useful tool, and it is in fact mandatory to implement for DOCSIS links.  Other links (including WiFi) could implement it as well.  For example, WiFi APs or Stations could monitor queue depth or queue latency for the AC_VI (or AC_VO, whichever NQB maps to) queue and re-direct NQB traffic to AC_BE, either in a flow-aware way (probably preferred) or possibly even in a flow-blind way.  In other situations it may not be necessary (e.g. in controlled environments or on links that support FQ), or it may not be feasible.  

[GW] Exactly.  That is one mechanism that has been discussed.  A possible downside to that approach is that once the flow has demonstrated that it is QB (by building a queue in the NQB buffer) and subsequent packets are classified to the QB buffer, it is possible that the QB buffer was empty when those packets arrived, and they end up being transmitted before the tail packets in the NQB buffer, i.e. out of order delivery.   Flow startup for traditional TCP flows (even those that support RACK) can be sensitive to out of order delivery, so if this happens frequently it could be an issue.

    Best Regards
    	Sebastian

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Sep 2019 23:49:41 +0100

Sebastian,

On 10/09/2019 21:16, Sebastian Moeller wrote:
> 	[SM] Indeed, except I can not find the part in brackets in that draft.
[BB] That's the reason it's in brackets.
> Setting the Low Latency buffer to only a few milliseconds, will then make the whole low latency queue susceptible of the attack scenario I described before, is that really an improvement (and helping your cause])?
[BB] It is intended to align incentives. If someone is minded to attack 
a queue, they always have numerous approaches available in their 
armoury. Per-flow traffic protection is designed to address that.

You keep repeating the same arguments about vulnerability and keep 
ignoring the argument that vulnerability is not a reason for the IETF to 
mandate protection.

Every RFC is vulnerable to such attacks (including other Diffserv PHBs). 
None make protection against such attacks mandatory. Because in the IETF 
MUSTs are standardization language, not general importance language.

> 	[SM] You are talking about the ECN marking component only here?
[BB] Yup, There's a special case in the aqm-dualq-coupled draft at 
S.2.5.1.1. 
<https://tools.ietf.org/html/draft-ietf-tsvwg-aqm-dualq-coupled-10#section-2.5.1.1> 
for Not-ECN packets classified into the L queue in a 'no protection' case:

             the L AQM
             SHOULD apply drop using a drop probability appropriate to
             Classic congestion control and appropriate to the target
             delay in the L queue

In DOCSIS, if the operator disables protection, it implements this 
special case using tail drop in the shallow buffer I mentioned. 
Alternatively, for a DualPI2 implementation without QProt, its L4S AQM 
could drop Not-ECT NQB packets with the square of the probability output 
by the L4S AQM.

> 	[SM] Is there any other option, unless one wants to effectively starve all other traffic "classes"?
[BB] Access links generally do not include anything to handle overload 
attacks on their current (FIFO) links. Some ISPs provide DDoS detection, 
redirection and scrubbing services in their core. Access links are 
usually rudimentary, simple things though.

One could apply per-flow protection. However, here we're talking about 
options if the implementer has chosen not to implement traffic protection.
> 	[SM] Great idea, looking forward seeing this in the next dualQ draft!
[BB] I think you're misunderstanding the purpose of the appendix in the 
draft. It's an example for tutorial purposes.

I have no intention of adding NQB support to that tutorial example, 
which needs to focus on explaining the Coupled DualQ, not get cluttered 
up with extraneous details like optional extra DSCPs, and corner case 
pseudocode.

But we are planning to add NQB support to the Linux reference 
implementation of DualPI2.

> Until then I will assume 250ms/2 as the likely default value.
[BB] That would be a very poor assumption. The shared buffer is 250ms to 
absorb Classic bursts, and the L4S queue just shares a part of that 
buffer that is typically tiny.

If an implementation used separate buffers, I would advise physical 
buffer of 250ms at the link rate for Classic and perhaps 4ms at the link 
rate for L4S, but with this 3ms sojourn limit applied within that.

> I also note that this adds another complication to dualQ, basically multiplexing a dropping regime into an ECN based system.
[BB] There would be an extra 'if' per packet (which is why we just used 
tail drop in a shallow buffer in DOCSIS).
>        starve."? I would not actually direct attention to this part of the dualQ rfc, as for a system that claims to share between L4S and standard-compliant flows only giving 1/16th to the standar…
[BB] You've completely missed the point of the coupling and how it 
overrides the scheduler. Please read the explanation in the draft 
(search for the three places where '1/16' is mentioned). It's no wonder 
you're not impressed with the design if you don't understand how it works.

> Naively, I would say this is only 10 ms if MAX RATE equals 1, no? What am I missing.
[BB] That means the buffer size in bytes (indicated by '[B]') is 10ms 
multiplied by MAX_RATE given in the appropriate units (B/ms).

Regards

Bob

---

From: "Jerome Henry (jerhenry)" <[email-masked]>
Date: Tue, 10 Sep 2019 23:33:42 +0000

Hi Greg,

I must say that Sebastien's argument also resonate with me (and I may be old school). The traffic types listed in the draft as NQB (interactive voice and video, gaming, machine to machine applications, then further also voice chat and DNS lookup) are all defined in various other RFCs (RFC8325 for 802.11 of course, but also RFC 4594 for most of them), with their associated recommended DSCP value. One element that they have in common is that they may individually be innocuous, but a large amount of them would obviously generate congestion. In the case of Wi-Fi, there may be several queues (usually 4 to 5 or 8 to 9), but in the end, there is only one radio, and any traffic will compete for this limited resource. Maybe what stops us (at least me) is a poor understanding of what exactly would be these NQB flows. I am debating with myself to decide if they are flows that have not been classified or categorized before, of if they are flows that are already classified, and that the NQB draft suggest should be classified differently (thus requiring an update of several other documents).
The case of 'interactive voice' or 'voice chat' is case in point. As I understand these flows (again, I may be old school), they would require a very different treatment from DNS requests. And at the same time, I am lost as to why delaying a DNS lookup (to a reasonable extend) would be damaging to the user experience or the DNS service on the client or server. Would you help me understand these NQB traffic flows more in details?
Also, I read in the thread what appears to me as a contradiction, that we are required to work with the current state of implementations where there is a default mapping of the first 3 bits of DSCP to UP (I would suggest trying other brands, I can give you quite a few names __), and that we are therefore stuck with this mapping and logic, but yet these same products should implement a new queue structure. Why would we be stuck with mapping and not with the queue structure?

Best regards,
Jerome 

On 9/10/19, 5:29 PM, "tsvwg on behalf of Greg White" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Sebastian, 
    
    See below marked [GW].

    

    
    
        Best Regards
        	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 11 Sep 2019 12:31:28 +0200

Hi Bob,

> On Sep 10, 2019, at 23:25, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
> [BB] It is a highly fitting analogy, so if you fail to see it, please try harder. My wife just read this, and said, "What's not to understand? That's an extremely relevant analogy."

	[SM] Okay, I tried to avoid this unnecessary detour, but since you seem to insist:
NQB, according to this draft is more than a simple claim of a property, it is rather a "promise" to behave in a specific way. In turn this draft proposes to give something special to NQB-marked flows as an exchange for that specific behavior. 

IMHO this is not exactly the same as me claiming my car to be 17 years old, which by itself will most likely have zero effect on anything, except maybe the verbosity of my story (and hence will not require verification). Now, let's see what happens if we actually make the age of the car relevant:
	Say, there is a specific tax exemption for cars older than 16 years (or a special tax increase, the direction does not really matter); and, say I want to buy the car you claim to be 17 years old, do you really expect me to not verify your claim? (And how could I actually not realize the age of the car, given that the car's papers will state the manufacturing date explicitly (at least here in Germany they do)? It sees to me that once the age of a car actually has actionable meaning verification will in all likelihood happen. Otherwise, I have a 17 year old car to sell to you...
	Now, unlike real-world persistent object like the car above, where we can do the verification post-hoc and drag each other to court if the given information was inaccurate yet relevant, in the NQB case there really only is a single entity which can verify the behavior, that is also is the same entity that gives special treatment to NQB marked flows. 
	Excuse me, if I still believe your car analogy (at least in its original form) to be ill fitting to the topic at hand. Sure I agree that there are verifiable facts that do not need to constantly be verified, but NQB-ness is not one of them.

> In life, most trust is based on verifiability, but rarely actually verified.

	[SM] Sure, but for important things we do verify and we have post-hoc measures to ensure compliance (search for penal law). But that is not applicable to transient flows on the internet, you either enforce compliance on the spot or the horse has left the barn. There is a reason why "trust, but verify" and "speak softly, but carry a big stick" entered the political dialog, enforcement of relevant facts is much less rare than you might believe.

> No, it's important that it is verifiable. It's also important that it doesn't have to be verified, for reasons I've given in other emails.
> 

	[SM] That does not help, as the NQB draft so eloquently observed non-que-buildingness is only partly a function of each individual sender, it effectively depends on the aggregate ingress traffic (in the NQB class) versus the available egress rate, if the former exceeds the latter all NQB-classified flows are doing what they claimed they will not, namely building a queue. It is the bottleneck that sees the queue first and is in the best position to react, no amount of app-store pre-verification is going to change that, and that is under the optimistic assumption that app-store verification is perfect.... (Just see the examples of root kits distibuted vi)

> b) and the only place where the operator could apply verification prior to the bottleneck radio link would be on the UE

	[SM] UE = User Equipment? Which still runs a baseband process that the user has exactly zero influence on (this is a stronghold of the telco). Surely that baseband software should do the verification in the upstream direction? In addition the fact that for the upstream direction it is the CPE that should do the NQB handling is also true for DSL and DOCSIS. What is so special about mobile in this regard?

> 
> Here continual run-time verification is both complex (b) and unnecessary (a). So not appropriate to mandate it.

	[SM] Well, then simply do not propose a scheme that is complex in the first place? That it is necessary has been argued multiple times in this thread (Jonathan's NUR example, my DDOD as a service examples).

> 
> I'm afraid I cannot continue spending time on your responses if you just snip the text of my arguments that you haven't argued against and then continue to argue against the conclusion I've drawn from those arguments with no rationale, just IYHO assertions.

	[SM] ATM we are both exchanging assertions, and I would have thought that you would simply shut me up with real-world data, instead of strawman constructs and Gedankenexperimenten.

> [BB] Well, let's see what Jonathan comes up with in response to my request to reconstruct his argument, given he had misunderstood the config.
> 
> But remember that, even if the incentives are not clear-cut, I am saying that it is still not appropriate for the IETF to mandate a protection mechanism.

	[SM] Yes, I get that, you are convinced a MUST for queue protection must be avoided, that much is clear, what is lacking to me is a clear and understandable rationale for that position that is more elaborate than "this might rule out a few desirable configurations" since I believe you underestimate the likely "undesirable configurations" that will come along from not enforcing what ever behavior a NQB flow needs to fulfill. Conceding that the incentives are not as clearly aligned as you initially posited makes this enforcement even more important IMHO. I think it was your argument that enforcement can be lax, since there is no incentive to game the NQB marking. Once you agree that the incentive situation is not that clear what else makes abuse improbable?

> [BB] You seem to be treating me as if the economics of DDoS and traffic security is not my area of expertise.

	[SM] Well, then why did I have to bring this up? 

> 
> I assume you are aware that nearly every RFC is vulnerable to DDoS. So now please explain why each RFC does not mandate protection against DDoS?

	[SM] And why is your only defense for advocating for more attack surface to state that everybody else is not perfect either? But we are discussing the merits of this RFC draft and not all other RFCs so, lets's leave these out unless specifically related.

> 
> It's not an oversight. It's deliberate, because one doesn't have to implement protection at every point in a network and at all times (think of the example of app-store verification above).

	[SM] Deliberately, introducing a new cheaper DDOS attack vector that nicely isolates easy to disrupt traffic on a silver platter does not make it any better than doing it accidentally, rather the opposite. Requirng app-store verification basically proposes to make commercial entities the back-stop safety valve for a potentially overly optimistic RFC, is that the IETF's typical approach?.
 
> To prove MUST is necessary, you have to prove protection is necessary in every scenario.

	[SM] Do I? From https://www.ietf.org/rfc/rfc2119.txt:

"Imperatives of the type defined in this memo must be used with care and sparingly.  In particular, they MUST only be used where it is actually required for interoperation or to limit behavior which has potential for causing harm (e.g., limiting retransmisssions)  For example, they must not be used to try to impose a particular method on implementors where the method is not required for interoperability."

"limit behavior which has potential for causing harm" is a far cry from your interpretation that as long as there exists one scenario without "potential for causing harm" a MUST is not merited. 
@chairs, is my reading of rfc2119 in agreement with the official stance, or is Bob's interpretation correct?

I believe that your proposed level of absoluteness is not what engineering is about, the question is not the existence of scenarios that do or do not require behavioral verification, but rather the IETF will have to make a judgement call how much potential damage to risk for how much potential gain.
 
	And David Black already indicated he leans towards a MUST with a set of explicitly enumerated exceptions, so why not start enumerating those cases that should not be required to activate queue protection, instead of boxing yourself into a corner?

> That means you have to argue against the cases where I have shown it's /not/ necessary.

	[SM] No, I do not need to do this, as I do not believe that your assumption "necessary in every scenario" is in compliance with rfc2119's definition of the term.

> Unless you have done that, there's no point you introducing more cases where you think protection /is/ necessary.

	[SM] Well played, well played, to demonstrate why this line of dogmatic argumentation will lead to absurd conclusions, how about:
Not all motorcyclist are not involved in accidents and crashes and hence should not be required to wear a helmet because not all of them need that protection! 
Not all TCP's need congestion control, since as long as the window size restricts the max achievable bandwidth for the flow below the available bandwidth for the path no "collapse" will happen. 

I could go on, but I hope I made my point.

> [BB] My sentence is about the application that is harmed, not the application doing the harm.

	[SM] Ah, sorry, I misunderstood.

> [BB] The argument right from the start has been about SHOULD/MUST implement, not SHOULD/MUST enable. But it's similar. If it's SHOULD implement, the implementer can crow about it. But there's still a market for NQB without protection (e.g. for a mobile uplink).

	Erm, the last proposal was MUST implement SHOULD enable, at least that was the last proposal I remember. 
	Why does is a mobile uplink better off without queue protection than with? Really an honest question, as the owner of a mobile device I would be happy if the system would handle misbehaving flows gracefully and localized instead of spreading the pain to all flows sharing the queue. Look, I belong to those folks that consciously run fq AQMs on both upstream and downstream direction of my fixed link and that would happily do so for my mobile devices as well, and I would pay extra to be able to do so, so there might also be a market for full flow isolation on mobile uplinks as well, for all I know (maybe with a customer pool of one).

> That is still not an argument for 'MUST protect', which would force implementations of NQB to protect in order to claim RFC compliance, even if protection is not appropriate for the deployment scenario of the particular NQB implementation.

	[SM] And that is why it was proposed to go for MUST implement SHOULD enable instead of the stricter MUST implement & enable. So is your point that you see the implementation cost as too high so it will hinder deployment? In that case I would argue the fact that cable labs both mandates (implementation) of queue protection and champions the NQB codepoint, indicates that your sentiment is not shared universally, no?
	But back to the MUST, I see clearly a "potential for causing harm" from unprotected NQB queues, so IMHO a MUST protect is perfectly justified by rfc2119, no? (Note how interoperability is not the only clause that justifies a MUST or SHOULD, and how the "not required for interoperability" sentence is only given as an example).

> [BB] Access links are the primary and often expected to be the only point on a path through the Internet where NQB would be deployed.

	[SM] Sure, unless those cases when they are not... For wifi with its variable rates see my rationale for mapping NQB not higher than AC_BE for example. And for end-users more often then desired bottlenecks appear in transit/peering links between ISPs (which admittedly are not likely to special-case NQB anytime soon).

> See earlier response about how operators design their networks to ensure the bottleneck is usually at a single point that they can easily control.

	[SM] Some do and some do not, see e.g. the discussion about peering conflicts between eyeball-T1s and "pure" T1-ISPs, where it is implied that keeping capacity too low for peak traffic is used as a tool to sell direct access links for content providers affected by the rush-hour congestion, but I digress.
 
> [BB] At present the draft concerns the bottleneck link of a path, which it says is frequently the access network. I think an NQB per-hop-behaviour could be applicable to any bottleneck, but the discussion of incentives and protection in the draft is mostly specific to cases where the bottleneck is an access network link (as in the three use-cases given).
> 
> So if you mean that the authors could say that the protection discussion applies primarily to access networks, I would agree.

	[SM] I would prefer to either fully restrict the scope of NQB to access links or also discuss the implications for other link types, yes.

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 11 Sep 2019 13:43:59 +0200

Hi Greg,

thanks for your response, comments prefixed with [SM]

> On Sep 10, 2019, at 23:28, Greg White <[email-masked]> wrote:
> 
> Hi Sebastian, 
> 
> See below marked [GW].
> 
> 
> 
> [GW] What validation is done today for applications that mark their packets as AC_VO or AC_VI?  

	[SM] As far as I know almost no validation, but do to the traditional DSCP bleaching over the internet and the reluctance of most applications to twiddle with DSCP bits, most traffic will be in the AC_BE, but admittedly rather by chance than by design. In addition the IEEE's default DSCP to AC mappings are quite conservative (like mapping EF to AC_VI not AC_VO) making high volume traffic in the higher ACs relatively unlikely with such DSCPs that might survive internet passage. But with NQB's desirable approach to be transmitted e2e, that is going to change.

> I think that WMM is another one of those cases where the tragedy of the commons seems imminently possible, yet it largely hasn't occurred.  

	[SM] Mmmh, with 2.4GHz radios in a dense AP-concentration situation, we are close (and this is by beacon traffic alone).

> But, I hear you about being more clear about which flows are intended to be marked NQB. We certainly don't want to recommend something that creates the tragedy.

	[SM] Great, I think we can all agree on this ;)

> Perhaps we've left it too ambiguous in the draft, so that needs some review.  It was not the intent that capacity-seeking flows (even L4S) mark their packets as NQB.

	[SM] Great, maybe this can be made more explicit?

>  The target flows would be ones where the sender has a degree of confidence that it will not exceed the available capacity of the path.  

	[SM] With a variable bandwidth/rate path like wifi, the sender's confidence might not be the best measure? I would guess the hops operating the RF link would be in a better position to predict the path capacity in the immediate future?

> Also, I think validation of NQB behavior is a very useful tool, and it is in fact mandatory to implement for DOCSIS links.

	[SM] +1.

>  Other links (including WiFi) could implement it as well.  

	[SM] I fear I am less optimistic, and would argue that without this implemented wifi should be very careful to prioritize NQB (unless we can agree on NQB qualifying behavior that makes harm very unlikely).

> For example, WiFi APs or Stations could monitor queue depth or queue latency for the AC_VI (or AC_VO, whichever NQB maps to) queue and re-direct NQB traffic to AC_BE, either in a flow-aware way (probably preferred) or possibly even in a flow-blind way.  In other situations it may not be necessary (e.g. in controlled environments or on links that support FQ), or it may not be feasible.  

	[SM] All good ideas; the question to me is more in light of the lack of such mechanisms would it not be safer to default to AC_BE and only selectively elevate NQB traffic if the radio controller/AP is NQB aware? 
	I ask because I just did a flent rrul_CS8 (that is bi-directional greedy TCP traffic with one flow dscp-marked for each CS) test-run on my home net, and saw my macbook's sent CS6 and CS7 flows almost starve all other traffic, including CS6/CS7 marked traffic from the AP; I am not confident, that the hardware out there is ready for significant traffic volumes on AC_VO/AC_VI yet... (happy to share the flent data file on request).
	Now, I realize that NQB is not intended for bulk flows, but as long as the marking is not actively verified (and maybe even bleached on detected mis-marking) the intention does not matter that much, NQB-mismarking will give a considerable bandwidth and latency improvement. With access links getting faster and faster, wifi is becoming more and more a (transient) bottleneck making this issue sensitive.

> [GW] Exactly.  That is one mechanism that has been discussed.  A possible downside to that approach is that once the flow has demonstrated that it is QB (by building a queue in the NQB buffer) and…

	Interestingly it was a similar observation that made me initially think, DOCSIS queue protection would track each flow's packets in the queue so that on data-driven change of classification all queued packets of that flow could be moved from one queue to the other (I obviously did not consider the cost of tracking very deeply). Even though at that point it might be easier to directly go for stochastic flow queueing, as in that configuration re-ordering is not an issue and it should be much easier to verify compliance to required behavior if the flows are isolated... 

Best Regards
	Sebastian

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 11 Sep 2019 15:18:55 +0300

> On 11 Sep, 2019, at 1:31 pm, Sebastian Moeller <[email-masked]> wrote:
> 

In the UK, there are quite a few exemptions from modern regulations that would otherwise apply to vintage cars.  Emissions, safety equipment, safety-related aspects of design, styles of registration plate, and so on.  I think the cutoff date for many of these exemptions is somewhere in the early 1970s, and is defined by the "date of first registration".  The exemptions do not, AFAIK, apply to newly built replicas of these vintage designs.

Conversely, in the UK the date of registration is encoded into the registration number, in a way that's quite easy for an interested party to read, and it is forbidden to apply a registration number *newer* than that corresponding to the date of first registration of a particular vehicle (though you *are* allowed to transfer an older plate to a newer vehicle).  This is a regulation mostly aimed at sales of relatively new cars, in which the precise age of the vehicle has a strong correlation with its depreciation, in much the same way as odometer regulations.

So the age of a car is a fact that is often worthwhile to verify, in either direction.  But it tends to be relatively easy to do so in most cases, by referring to paperwork, checking the registration number, and/or looking up the VIN in a database.

By contrast, imagine you're trying to buy a 6502 CPU.  Sometimes you need one to repair a vintage computer - but you might also be trying to build a new machine as a hobby project.  Now, 6502s are in fact still being made (by WDC) with a speed rating of 14MHz and quite a few features that early models lacked; you can buy one for €7, if you can find a distributor willing to sell just one to you, as opposed to a batch of ten or more.  There are also several dozen combinations of older manufacturers and models, all the way back to MOS' original 1976 production run in which one of the instructions was faulty (and consequently omitted from contemporary documentation).

Your average Chinese eBay or AliExpress listing of 6502s will advertise a very cheap source of Rockwell 4MHz or WDC 10MHz parts.  The markings are very neatly printed, and seem in remarkably good condition for chips of such age - production of those models ended in the 1990s.  And you only have to pay €2 with free shipping!  Okay, it'll take a few weeks on the boat, but it'll take you that long to finish assembling the rest of your prototype anyway.  I'll buy three.

When it arrives, however, it doesn't work.  After some experimentation, you might discover that it is in fact a 6502 - but a circa-1980 NMOS model, with very different electrical and performance characteristics to the 1990s CMOS model you thought you were getting.  This wouldn't matter if you were repairing some ancient Apple II which was designed around those characteristics, but your project has much more stringent performance requirements, and now you're stuck.

And while the three examples you have in hand have identical markings on top, you now notice that they have slightly different cases, and different markings on the bottom - so they were not all made by Rockwell or WDC, but by three different manufacturers from the several that made 6502s back in the day.  If you're *lucky*, one of them was *actually* a Rockwell product.  And the date-codes are from the 2010s, long after the purported models went out of production.

The Chinese e-waste recycler has simply taken all the 6502s he can find, scraped the original markings off the top, straightened the pins, and printed a standard marking resembling that of one of the more desirable versions of the chip.  This costs him basically nothing to do and increases his income significantly, so that's good business to him.  He has your money - that's the important part.  And he's sold you a set of working CPUs, so his conscience is clear.  The fact that there are different models of 6502 is simply lost on him.  Why should it matter?

But it matters to you, the buyer.

 - Jonathan Morton

---

From: <[email-masked]>
Date: Wed, 11 Sep 2019 12:29:59 +0000

Hi Greg,

backbone routers handle bulk traffic to the extent that I can tell. Finer grained traffic treatment may be found at network edges terminating consumer access. That is to say: a peer or a customer should agree which DSCPs are exchanged at interconnection or expect re-marking.

And to add (as that has been discussed too): don’t expect per flow treatment at a congested backbone link. That’s to say, that I don’t expect NQB support by a backbone router in the same way, as by an edge router. The next generation interfaces are n*100 Gbit/s and I’m not sure how these perform trying to identify the “least NQB like flow” in the case of congestion.

Regards, Ruediger

Von: Greg White <[email-masked]>
Gesendet: Dienstag, 10. September 2019 20:30
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Hi Ruediger,

-Greg

Hi Greg,

Regards,

Ruediger

Von: Greg White <[email-masked]<mailto:[email-masked]>>
Gesendet: Freitag, 6. September 2019 19:18
An: Geib, Rüdiger <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Hi Ruediger,

DSCP
WMM Access Category
001xxx, 010xxx
Background (AC_BK)
000xxx, 011xxx
Best Effort (AC_BE)
10xxxx
Video (AC_VI)
11xxxx
Voice (AC_VO)

-Greg

Hi Greg,

Regards,

Ruediger

Thanks David.

-Greg

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 11 Sep 2019 16:02:41 +0200

Dear Bob,

> On Sep 11, 2019, at 00:49, Bob Briscoe <[email-masked]> wrote:
> 
> Sebastian,
> 
>>>> [...]

	[SM] As a reader of an RFC I often like such details explicitly spelled out, especially with the rationale behind such a choice.

> [BB] It is intended to align incentives. If someone is minded to attack a queue, they always have numerous approaches available in their armoury. Per-flow traffic protection is designed to address that. 

	[SM] So why fight against making that kind of protection mandatory then?

> 
> You keep repeating the same arguments about vulnerability and keep ignoring the argument that vulnerability is not a reason for the IETF to mandate protection.

	[SM] Not ignoring, rather (so far) silently rejecting. RFC2119 states as justification for the use of imperatives "they MUST only be used where it is actually required for interoperation or to limit behavior which has potential for causing harm". And here I see the " limit behavior which has potential for causing harm" clause triggered. You might disagree with this assessment, but arguing that there is no justification for mandating SHOULD/MUST implement/enable queue protection seems a bit extreme.

> 
> Every RFC is vulnerable to such attacks (including other Diffserv PHBs). None make protection against such attacks mandatory. Because in the IETF MUSTs are standardization language, not general     importance language.

	No, I disagree, not all RFCs define new methods that can be used for targeted attacks in such a convenient way that an unprotected NQB-queue will. I am happy to discuss specific relevant other RFC's but argueing over the whole set of RFCs seems to be a waste of time.

> [BB] Yup, There's a special case in the aqm-dualq-coupled draft at S.2.5.1.1. for Not-ECN packets classified into the L queue in a 'no protection' case:
>             the L AQM 
> In DOCSIS, if the operator disables protection, it implements this special case using tail drop in the shallow buffer I mentioned. Alternatively, for a DualPI2 implementation without QProt, its L4S AQM could drop Not-ECT NQB packets with the square of the probability output by the L4S AQM.

	[SM]Thanks for clarifying!

> [BB] Access links generally do not include anything to handle overload attacks on their current (FIFO) links. Some ISPs provide DDoS detection, redirection and scrubbing services in their core. Access links are usually rudimentary, simple things though.
> 
> One could apply per-flow protection. However, here we're talking about options if the implementer has chosen not to implement traffic protection. 
>> 
> [BB] I think you're misunderstanding the purpose of the appendix in the draft. It's an example for tutorial purposes.

	[SM] But this recommendation you gave for NQB is in violation with the dualQ draft recommendation:
"if separate queue protection is not provided, the L AQM SHOULD apply drop using a drop probability appropriate to Classic congestion control and appropriate to the target delay in the L queue"
A "target delay in the L queue" is not given explicitly, but looking at the pseudocode that might be this value would be around 1ms for L4S traffic (pi2: minTh+range = 475 + 525µs = 1 ms, curvy RED: T = 1ms), so without special casing for 3ms sojourn time the draft recommends 1ms sojourn time. What am I missing?

> 
> I have no intention of adding NQB support to that tutorial example, which needs to focus on explaining the Coupled DualQ, not get cluttered up with extraneous details like optional extra DSCPs, and corner case pseudocode.

	[SM] Sure, that draft is on the longish side already. How about then also removing the following section:

"In addition, an operator could use other identifiers to classify certain additional packet types into the L queue that it deems will not risk harm to the L4S service.  For instance addresses of specific
   applications or hosts (see [I-D.ietf-tsvwg-ecn-l4s-id]), specific Diffserv codepoints such as EF (Expedited Forwarding) and Voice-Admit service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]) or certain protocols (e.g.  ARP, DNS).  Note that the mechanism only reads these identifiers.  [I-D.ietf-tsvwg-ecn-l4s-id] says it "MUST NOT alter these non-ECN identifiers"."

As this set seems to have a large intersection with the NQB set and hence would ideally have the same treatment?

> 
> But we are planning to add NQB support to the Linux reference implementation of DualPI2.
> 
> [BB] That would be a very poor assumption. The shared buffer is 250ms to absorb Classic bursts, and the L4S queue just shares a part of that buffer that is typically tiny.

	[SM] Sure typically, but worst case is actually 250ms. I am not much interested in discussing best or average cases here, I want to talk about the pathological cases, or rather those in which queue-protection could potentially help.

> 
> If an implementation used separate buffers, I would advise physical buffer of 250ms at the link rate for Classic and perhaps 4ms at the link rate for L4S, but with this 3ms sojourn limit applied within that.
> 
> [BB] There would be an extra 'if' per packet (which is why we just used tail drop in a shallow buffer in DOCSIS).
>> 
> [BB] You've completely missed the point of the coupling and how it overrides the scheduler. Please read the explanation in the draft (search for the three places where '1/16' is mentioned). It's no wonder you're not impressed with the design if you don't understand how it works.

	[SM] Again, I am not interested in discussing best case scenarios here. The quoted text explicitly allows for a scenario where "L4S traffic is over-aggressive or unresponsive" (which in theory could be avoided by mandatory queue protection?) which will lead to starvation of the standard-compliant queue, so you assign 1/16 of the bandwidth for standard-compliant traffic in that case. Which is a tad below the naively expected 1/1for equal sharing. Now this might be an unlikely scenario, but it certainly is not one I invented, but one from the dualQ RFC that I consider problematic.
	Note how this discussion is unrelated to the normal sharing between TCP-friendly flows and L4S-flows caused by the coupling?

> [BB] That means the buffer size in bytes (indicated by '[B]') is 10ms multiplied by MAX_RATE given in the appropriate units (B/ms).\

	[SM] Thanks, my bad, I mis-interpreted that (glad I cited the text though to at least indicate where I screwed up).

---

From: Greg White <[email-masked]>
Date: Wed, 11 Sep 2019 16:21:03 +0000

Hi Steven,

See my responses (marked [GW2]).

-Greg

Date: Sunday, September 8, 2019 at 9:50 AM

[GW] comments below

Greg,

I don't know how to reconcile any of this below with the text of your draft (specifically Secs. 1 & 3), which talks about low-bandwidth applications, specifically those over UDP. Are you contending that the sum of these adds up to a significant fraction of traffic on any link?

[GW2] While it’s not difficult to come up with scenarios where the NQB traffic adds up to a significant percentage of link capacity, yet remains NQB, the point I was trying to make was that there isn’t a need to artificially limit NQB traffic to some small fraction of link capacity, and doing so just makes the problem harder.  Using a work-conserving DRR scheduler with 50/50 weighting between the queues is sufficient.  Like Bob Briscoe indicated in one of his recent posts, throw out your preconceived notions of how QoS has traditionally been implemented in networks.  This is different.

That claim is technically false. You cannot achieve 100% utilization on any link (or link slice) at low queueing delay for uncoordinated, non-admissioned controlled traffic. I'll bet you cannot find a carrier anywhere that allocates 50% (or even 20%) of capacity on their metro or long-haul networks to admissioned-controlled & tarriffed EF traffic. So I don't see them doing so for non-admissioned controlled/non-tarriffed traffic.

[GW2] I think you’re missing my point.  The point was that it isn’t necessary that the aggregate of NQB traffic remains below 5% of link capacity.  Also, as the draft discusses, the intended application of the NQB PHB is on access network links, not long-haul or metro links.  Even so, you can’t compare historical configurations of EF prioritization to NQB.   If you need an analogy, imagine a link that supports an FQ-like qdisc, where flows are hashed into two queues, and there is a work-conserving DRR scheduler with equal weighting between the two queues.  While this hypothetical system wouldn’t give a whole lot benefit, I hope you can see that it wouldn’t be objectionable either.   Maybe you get lucky and the queue your flow gets hashed into is less loaded than the other, but maybe it isn’t.

If I am writing a capacity seeking application, and I believe that there is a lightly utilized link slice of 20%-50% of capacity where I can grab more than my fair share, I would be highly tempted to setup two connections, one-NQB marked and one not, and let them each grab as much capacity as possible. Which benefits my app until everyone else does the same, and then we are back to square one.

[GW2] You could be setting up two connections today, even without NQB.  Heck, you could set up 200.  Or you could roll your own congestion controller that cuts its congestion window by 5% in response to a congestion signal (under the assumption that all the other suckers will back off first).  It is possible for the internet to be abused. I still find it useful.
[GW2] As a more specific response to this, what would you use for your congestion controller on the NQB queue?  In the case of DOCSIS, you’ll need to (more-or-less blindly) guarantee that you can maintain sub-millisecond queuing delay while seeking the link capacity in order to succeed. Why not just use TCP Prague or BBRv2 and make your life easier?

[GW2] The whole idea is that estimating NQB bandwidth and tuning isn’t required.

Regards,

// Steve

---

From: Greg White <[email-masked]>
Date: Wed, 11 Sep 2019 17:14:06 +0000

Agreed.  I think the most likely implementation in backbones is to allow NQB marking to survive the interconnection, but treat the traffic identically to default traffic until it reaches the edge (access network).

-Greg

Date: Wednesday, September 11, 2019 at 6:30 AM

Hi Greg,

Regards, Ruediger

Hi Ruediger,

-Greg

Hi Greg,

Regards,

Ruediger

Hi Ruediger,

DSCP
WMM Access Category
001xxx, 010xxx
Background (AC_BK)
000xxx, 011xxx
Best Effort (AC_BE)
10xxxx
Video (AC_VI)
11xxxx
Voice (AC_VO)

-Greg

Hi Greg,

Regards,

Ruediger

Thanks David.

-Greg

---

From: Greg White <[email-masked]>
Date: Thu, 12 Sep 2019 05:21:04 +0000

See [GW2]

[GW2] According to one recent study, roughly 50% of ASes bleach DSCP, so yes a decent percentage of traffic that traverses an interconnect before reaching the WiFi link will end up in AC_BE, but other traffic will not.  None of the traffic generated by local devices on the WiFi network gets bleached.   All OSes that I'm aware of provide open APIs for applications to select their desired DSCP.  In the case of Windows, applications (unless they are privileged) are limited to selecting from a set of 4 DSCPs that map to the 4 WMM Access Categories.  Applications do use these, without validation or policing (at least in residential environments, in enterprise WiFi systems it may be a different story).  In my experience approximately 10% of packets are sent as non-AC_BE (~4% BK, ~1% VI, ~5% VO) in current WiFi networks, though I'm sure there is wide variability depending on the applications in use.  Moreover, I would find it hard to believe that either RFC8325 or the default mappings were defined under the assumption that ALL traffic arriving at a WiFi AP on its WAN port would be guaranteed to be default DSCP.  Given this, I see no justifiable reason for NQB to map to AC_BE in traditional WiFi gear.  The traffic expected to be labeled NQB is sparse unresponsive traffic or smooth low-data rate flows, which is compatible with AC_VO.   We've proposed a DSCP that, conservatively, aligns with the mapping of CS4, AF41, AF42, AF43, CS5, VA and EF to AC_VI in default mapping equipment, and aligns with the mapping of VA, EF and CS6 to AC_VO in RFC8325 equipment.  As I said previously, I don't have a strong view on VI vs VO for RFC8325, but since RFC8325 (unfortunately) maps some capacity-seeking bulk data traffic to AC_VI, I believe that it is more appropriate that NQB sparse traffic be VO in those systems.  The draft has a recommendation that all devices supporting NQB implement a queue protection mechanism for NQB traffic, which for RFC8325 devices supplements all of the detailed recommendations* on protecting QoS in section 8 of RFC8325, which still remain intact.  So, my view is that AC_VI for NQB in RFC8325 equipment is defensible.  

*Note that RFC8325 makes all of these protections RECOMMENDED, not REQUIRED, in alignment with the NQB draft's handling of queue protection.

[GW2]  A point to be repeated here is that this proposed mapping of DSCP to WMM was not driven by the goal of giving priority (either bandwidth priority or latency priority) to NQB traffic, but rather to have it queued separately from QB traffic.  In existing WiFi gear, that isn't possible without using different ACs and we have to work within the limitations of the equipment.   Perhaps the NQB draft should make this more clear.  In other words, the draft should clearly require that WiFi APs that claim compliance with the NQB PHB definition have a separate queue for NQB traffic.  If that separate queue is then scheduled alongside a QB queue, both using AC_BE, that would be compliant.

    > [GW] Perhaps we've left it too ambiguous in the draft, so that needs some review.  It was not the intent that capacity-seeking flows (even L4S) mark their packets as NQB.
    
[GW2] Yes, I have this on my to do list.   I was modeling the NQB draft on some of the PHB RFCs that were very explicit about PHB requirements, but fairly vague on what applications should use it. 

    >  [GW] The target flows would be ones where the sender has a degree of confidence that it will not exceed the available capacity of the path.  
   
[GW2]  There are many applications that can a) be very confident in their belief that they will not exceed the available capacity of the path, and b) that would benefit from not being grouped with capacity-seeking traffic that is causing latency and loss in the network.   Keep in mind that the concept of segregating QB & NQB traffic isn't so that we can guarantee deterministic latency for NQB traffic.  This isn't DETNET.  The goal is to allow NQB applications to have statistically much better latency characteristics than they do today. 

    >  [GW] Other links (including WiFi) could implement it as well.   

[GW2] I'm confident that we come up with language to more clearly describe NQB traffic in a manner that is compatible with the descriptions of other DSCPs that share an AC with it.  I assume this (perhaps in addition to something like I mentioned above about segregation being the only requirement for the PHB) would satisfy you, no? 

    > [GW] For example, WiFi APs or Stations could monitor queue depth or queue latency for the AC_VI (or AC_VO, whichever NQB maps to) queue and re-direct NQB traffic to AC_BE, either in a flow-aware way (probably preferred) or possibly even in a flow-blind way.  In other situations it may not be necessary (e.g. in controlled environments or on links that support FQ), or it may not be feasible.  
    
[GW2]  I hope we're past this in the discussion now, but just in case we're not:   I don't think that NQB materially changes this situation.  Today (effectively) no traffic has its WMM selection policed.  It is totally up to the application to decide if it wants to mark its packets CS7 or not, and no one is mandated by the IETF to stop them.  Don't get me wrong, I am not opposed to WiFi devices implementing queue protection (or the QoS protection features in RFC8325), in fact they SHOULD do so (if they want to comply with the draft anyway).  But as long as the traffic types that are recommended to be marked NQB (i.e. sparse/smooth relatively low data rate flows) are compatible with the types of traffic that are recommended to be marked EF, CS6, etc. then I think we're covered.  

[GW2] I really don’t follow your insistence on the IETF mandating that implementers build in safeguards like this.  Recommendations are great, and I support them. 

[GW2] I know you are a fan of fq_codel.  What do you implement in your router that detects and prevents an application from sharding its traffic across multiple queues?  If you aren't preventing this simple and powerful exploit, then maybe your router should be banned from the internet!  (this is a joke, in case anyone misunderstands)

---

From: <[email-masked]>
Date: Thu, 12 Sep 2019 06:54:57 +0000

Greg,

that’s not what I said. You’ll most likely see remarking to default PHB and forwarding as default PHB, unless there’s an SLA in place, if your desire is to mark NQB by a DSCP 101xxx. To preserve a DSCP clearly indicating NQB at interconnection without an SLA in place, I suggest a DSCP 000xx1.

RFC 8100 contains more information on the background (motivation of the above and how to deal with the issues arising).
https://datatracker.ietf.org/doc/rfc8100/

Regards,

Ruediger

Von: Greg White <[email-masked]>
Gesendet: Mittwoch, 11. September 2019 19:14
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

-Greg

Hi Greg,

Regards, Ruediger

Hi Ruediger,

-Greg

Hi Greg,

Regards,

Ruediger

Hi Ruediger,

DSCP
WMM Access Category
001xxx, 010xxx
Background (AC_BK)
000xxx, 011xxx
Best Effort (AC_BE)
10xxxx
Video (AC_VI)
11xxxx
Voice (AC_VO)

-Greg

Hi Greg,

Regards,

Ruediger

Thanks David.

-Greg

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 12 Sep 2019 10:19:45 +0200

Hi Greg,

> On Sep 12, 2019, at 07:21, Greg White <[email-masked]> wrote:
> 
> See [GW2]
> 
> [GW2] According to one recent study, roughly 50% of ASes bleach DSCP, so yes a decent percentage of traffic that traverses an interconnect before reaching the WiFi link will end up in AC_BE, but other traffic will not.  

	[SM2] Mmmh let's say a typical path traverses at least 2-3 ASs, so the chance marking surviving will be 100*0.5*0.5*0.5  = 12.5 %, meaning that 87.5% of packets will be bleached.

> None of the traffic generated by local devices on the WiFi network gets bleached.  

	[SM2] Sure, but almost no user knows how to actually modify DSCPs for sending applications, and very few application use something other than BE, I know anecdotally because I help support sqm-scripts users that run exactly into these issues, when they actively want to change the DSCPs.

> All OSes that I'm aware of provide open APIs for applications to select their desired DSCP.

	[SM2] But only few applications take advantage of that.

>  In the case of Windows, applications (unless they are privileged) are limited to selecting from a set of 4 DSCPs that map to the 4 WMM Access Categories.  

	[SM2] A priviledged user can override that on the command line, but knowledge about that fact is scarce (in a windows domain, the policy editor allows configuration via GUI ,but typical endusers need to rely on the CLI).

> Applications do use these, without validation or policing (at least in residential environments, in enterprise WiFi systems it may be a different story).

	[SM2] In competently managed enterprise environments I would assume that the admin uses the qos_map feature to tell clients which DSCP to AC mapping to use, but for home networks it is going to be the defaults, as no comerciaklrouter I know allows editing qos_map (nor does OpenWrt).

>  In my experience approximately 10% of packets are sent as non-AC_BE (~4% BK, ~1% VI, ~5% VO) in current WiFi networks, though I'm sure there is wide variability depending on the applications in use.  

	[SM2] So as long as NQB is not going to change that ratio considerably there should be not much changes. Unfortunately the current NQB draft does not contain language to assure that that is the case. Changing these ratios significantly will basically get into untested territory with new unexpected pathologies abound (in my testing the macbook's AC_VO completely dominated the AP's attempts to get packets into AC_VO, potentially indicating different interpretations of the WMM spec). I want to stress, that I base my reservations on real data here, not mere theoretical observations.

> Moreover, I would find it hard to believe that either RFC8325 or the default mappings were defined under the assumption that ALL traffic arriving at a WiFi AP on its WAN port would be guaranteed to be default DSCP.  

	[SM2] I agree, but I also do not see in rfc8325 links to any empiric studies on the effect of the proposed dscp to AC mapping on aggregate throughput, fairness and robustness against starvation. But rfc8325 does seem to recommend to use AC > BE cautiously and should be controlled:

"8.2.  Security Recommendations for WLAN QoS
   The wireless LAN presents a unique DoS attack vector, as endpoint
   devices contend for the shared media on a completely egalitarian
   basis with the network (as represented by the AP).  This means that
   any wireless client could potentially monopolize the air by sending
   packets marked to preferred UP values (i.e., UP values 4-7) in the
   upstream direction.  Similarly, airtime could be monopolized if
   excessive amounts of downstream traffic were marked/mapped to these
   same preferred UP values.  As such, the ability to mark/map to these
   preferred UP values (of UP 4-7) should be controlled.
[...]
Finally, it should be noted that the recommendations put forward in
   this document are not intended to address all attack vectors
   leveraging QoS marking abuse.  Mechanisms that may further help
   mitigate security risks of both wired and wireless networks deploying
   QoS include strong device- and/or user-authentication, access-
   control, rate-limiting, control-plane policing, encryption, and other
   techniques; however, the implementation recommendations for such

The whole section seems relevant in that it explicitly tackles the starvation issues I mentioned and that it recommends to take active measures against getting into pathological conditions.

> Given this, I see no justifiable reason for NQB to map to AC_BE in traditional WiFi gear.  

	[SM2] Well, I fail to see the mitigation techniques that rfc8325 discusses in NQB to merit anything but AC_BE. Given that you explicitly do not want to bandwidth constraint the NQB queue and that you want to lump quite different traffic tuypes into that class. 

> The traffic expected to be labeled NQB is sparse unresponsive traffic or smooth low-data rate flows, which is compatible with AC_VO.  

	[SM2] Great, I bleieve the " low-data rate" aspect has not been sufficiently stressed in the draft yet. That still leaves the issue that NQB type traffic is potentially allowed 100% of the bandwidth, and in that case each individual flow might be justified to AC_VI but the whole aggregate will not be. 

> We've proposed a DSCP that, conservatively, aligns with the mapping of CS4, AF41, AF42, AF43, CS5, VA and EF to AC_VI in default mapping equipment, and aligns with the mapping of VA, EF and CS6 to AC_VO in RFC8325 equipment.  

	[SM2] for end users there is no RFC8325 equipment, these devices so far default to IEEE defaults (or typically linux defaults). I am not saying RFC8325 is bad, but it certainly is not tested well in the real world.

> As I said previously, I don't have a strong view on VI vs VO for RFC8325, but since RFC8325 (unfortunately) maps some capacity-seeking bulk data traffic to AC_VI, I believe that it is more appropriate that NQB sparse traffic be VO in those systems.  

	[SM2] I can understand this from your vantage point, but that is rather selfish and will lead to an arms race to AC_VO quickly.

> The draft has a recommendation that all devices supporting NQB implement a queue protection mechanism for NQB traffic, which for RFC8325 devices supplements all of the detailed recommendations* on protecting QoS in section 8 of RFC8325, which still remain intact.  So, my view is that AC_VI for NQB in RFC8325 equipment is defensible.  

	[SM2] This you got backwards. No wifi device out in the field will do anything of that sort, but most will use the default mappings, so the only conservative approach is to use a dscp that maps to AC_BE and teach new NQG-aware wifi devices to map NQB to something else. You are basically justifying a lot of the properties NQB by declaring it is not a priority-based scheduling system, but on existing wifi, that changes into a semi-strict priority scheme (well it is almost a precedence like scheme with some residual starvation avoidance) which aas we all agree will have pathological issues with out proper access control.

> *Note that RFC8325 makes all of these protections RECOMMENDED, not REQUIRED, in alignment with the NQB draft's handling of queue protection.

	[SM2] In case someone wonders, that is exactly the kind of attitude which makes me completely oppose Bob's quest to avoid a MUST for queue protection... rfc2219  says:

3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

I am not convinced that your rationale above fulfills these requirements yet. But admittedly that is not my call to make, but I feel entitled to voice my concerns and leave the final arbitration to the experts and chairs.

> 
> [GW2]  A point to be repeated here is that this proposed mapping of DSCP to WMM was not driven by the goal of giving priority (either bandwidth priority or latency priority) to NQB traffic, but rather to have it queued separately from QB traffic.

	[SM2] I applaud that goal, but IMHO the real world effect trumps the intention here, and WMM is a priority/precedence scheme, so once wifi becomes the choking point, NQB will look like a priority scheme from the end-user's perspective.

>  In existing WiFi gear, that isn't possible without using different ACs and we have to work within the limitations of the equipment.

	[SM2] That is one option, the other option would be realize the impedance mismatch and opt for the conservative alternative.

>   Perhaps the NQB draft should make this more clear.  In other words, the draft should clearly require that WiFi APs that claim compliance with the NQB PHB definition have a separate queue for NQB traffic.  

	[SM2] +1; IMHO wifi gear that explicitly supports NQB and has the appropriate safety valves in place (anti-starvation/fairness measures) can go all AC_VO on NQB traffic, but that is not the currently rolled-out wifi gear. I am mainly concerned about not making things worse compared to the status quo.

> If that separate queue is then scheduled alongside a QB queue, both using AC_BE, that would be compliant.

	[SM2] ??? If you have you second NQB queue then you can (with some safeties) go wild and use AC_VO as you will still be able to share airtime between QB and NQB equally. But in the current situation you can not do that, so you need to be care-full when using ACs > BE to maintain roughly equal sharing (or at least reasonable starvation avoidance). Again I am voicing an opinion, and the decision is not mine to make.

> [GW2]  There are many applications that can a) be very confident in their belief that they will not exceed the available capacity of the path,

	[SM2] That is obviously wrong as path capacity in general is not fixed but variable; we can haggle about what you mean by "very confident" but that is besides my point, especially in the context of variable rate eifi where the actual rate is set individually for each tx slot.

> and b) that would benefit from not being grouped with capacity-seeking traffic that is causing latency and loss in the network.   Keep in mind that the concept of segregating QB & NQB traffic isn't so that we can guarantee deterministic latency for NQB traffic.  This isn't DETNET.  The goal is to allow NQB applications to have statistically much better latency characteristics than they do today. 

	[SM2] That part I agree with.

> [GW2] I'm confident that we come up with language to more clearly describe NQB traffic in a manner that is compatible with the descriptions of other DSCPs that share an AC with it.  I assume this (…

	[SM2] If the NQB definition is compatible with RFC8325 definitions or goes the extra queue for NQB traffic route, my objections go away (but again, my objections are non-binding anyway).

> [GW2]  I hope we're past this in the discussion now, but just in case we're not:   I don't think that NQB materially changes this situation.  Today (effectively) no traffic has its WMM selection po…

	[SM2] I think that making it even clearer in the draft that the NQB requirement is "sparse/smooth relatively low data rate flows" with some more clear language what "relatively low" actually translates to we have removed one of the obstacles, the other is aggregate traffic volume. I agree that there is nothing stopping people from abusing ACs today, but I observe in my network, that this is not happening quantitatively. If NQB catches on as a method to improve latency for "real-timish" applications this is going to change and since that is predictable we should  tackle this issue today. (IMHO the only two options are NQB is not adopted in real life (unlikely due to DOCSIS)) then nothing really matters, or it will see quantitate use, and the the aggregate traffic volume issue should also be discussed in the draft. Again, nit an IETF member, so all do here is voice my personal opinions and try to get the domain experts to chime in and make the final judgements.

> [GW2] I really don’t follow your insistence on the IETF mandating that implementers build in safeguards like this.  Recommendations are great, and I support them. 

	[SM2] Well reading rfc2119 indicates that even a SHOULD/RECOOMENDED is basically already a mandate, just one that allows an escape route as long that has a strong rationale behind it. See above about your RFC8325 alignment issue (where you argue since rfc8325 only RECOMMENDS protections so we are good without them) and Bob's stance on queue protection in general, why I am wary about being lax in enforcement ;)

> [GW2] I know you are a fan of fq_codel.  What do you implement in your router that detects and prevents an application from sharding its traffic across multiple queues?  If you aren't preventing th…

	[SM2] I guess I need to sharpen my pitch fork and try to get rfc8290 changed ;)

Best Regards
	Sebastian

P.S.: Not that it matters, I am not opposed to the idea behind the NQB PHB in general, and I am also not opposed to a careful increase of wfi ACs > BE, but I would like to see this done in a conservative way that treads lightly and accounts for the fact that there is little operational experience how already deployed wifi gear is going to behave in a high AC_[I/O] environment. This is especially important as  I indicated before, the effect of using the higher ACs will be felt in the whole RF neighbourhood, so side-effects will not be restricted to the user requesting the NQB marked traffic but quite a lot of "innocent" bystanders.

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 12 Sep 2019 10:41:02 +0200

Dear Ruediger,

this time to the list as well ;)

> On Sep 12, 2019, at 08:54, [email-masked] wrote:
> 
> Greg,
>  

[SM] +1; using 000xx1 will also effortlessly solve the wifi WMM issue with APs that do not explicitly support NQB (in essence all existing APs). But I believe Greg is also trying to put in a requirement that ASs should not re-mark NQB to something else (which in principle I support, at least a few of the dscp bits should be reliably e2e).

Best Regards
	Sebastian

---

From: <[email-masked]>
Date: Thu, 12 Sep 2019 09:30:24 +0000

Hi Sebastian,

thanks. My take is as described: if no SLA is required or desired and the DSCP is supposed to survive unchanged end-to-end or end-to-edge, 000xx1 seems a god way to go.

RFC 8100, "Diffserv-Interconnection Classes and Practice" https://datatracker.ietf.org/doc/rfc8100/ was written to describe and specify how to solve Diffserv related issues arising at interconnection.

I'm not sure whether a network provider supporting EF with a priority queue accepts NQB with a DSCP 101xxx at interconnection even with an SLA in place. As a minimum, I'd expect an incoming policer to be required (and the sender should configure NQB at the Gateway's sender interface with a bitrate set to shape this traffic to the policer rate). 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Donnerstag, 12. September 2019 10:41
An: Geib, Rüdiger <[email-masked]>
Cc: <[email-masked]> <[email-masked]>; [email-masked]
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Dear Ruediger,

Best Regards
	Sebastian

> David L. Black, Senior Distinguished Engineer Dell EMC, 176 South St., 
> Hopkinton, MA  01748

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 12 Sep 2019 12:20:28 +0200

Hi Ruediger,

> On Sep 12, 2019, at 11:30, [email-masked] wrote:
> 
> Hi Sebastian,
> 

[TANGENT]
	Somewhat related question, how hard would it be, in your opinion, to convince operators to only bleach those 3 bits of the dscp field that I assume they only look at effectively (modulo EF which unfortunately does not exactly equal CS5)? What I would desire is to split the 6 bit DSCP field into 3 bits for immediate use, and 3 bits for the end points to signal intent. That way intermediate hops can re-mark the first 3 bits as they see fit, while AS/DSDs after that re-marking can still base their own treatment/re-marking/re-mapping on the intent of the endpoints. That way NQB could be completely ignored by intermediate hops and those that should care simply ignore the first bits and just look at the intent bits.
	Sure 8 << 63, but in the end what is the value of 63 code points if they do not carry interpretable information?
[/TANGENT]

Regards
	Sebastian

---

From: <[email-masked]>
Date: Thu, 12 Sep 2019 11:46:51 +0000

Hi Sebastian,

a) I regard your proposal as useful and it was part of the motivation resulting in RFC 8100.
b) but you will have to face stiff opposition, as "IP precedence has been deprecated" (and that's why text relating to such a feature was removed from RFC8100).

If introduced, one needs to be careful interpreting the results of a three bit rewrite. An AFx1 packet will carry DSCP 000010 after remarking to default that way. This effectively blocks some DSCPs, as these might be mis-interpreted (or limits their interpretation to an AF context). 

I'm looking for possibilities keeping the backbone operation simple, while adding value to customer traffic. A few Diffserv PHBs in the backbone, a bigger variety at the edge and all that easy to aggregate, de-aggregate and operate. There are no perfect solutions, but what you propose looks like a pragmatic compromise to me.

Others prefer almost per application DSCPs with bad or no aggregation possibilities coded into the DSCP. You'll find standards and drafts following this philosophy too.

Regards, Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Donnerstag, 12. September 2019 12:20
An: Geib, Rüdiger <[email-masked]>
Cc: <[email-masked]> <[email-masked]>; [email-masked]
Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!

Hi Ruediger,

[TANGENT]

Regards
	Sebastian

>> David L. Black, Senior Distinguished Engineer Dell EMC, 176 South 
>> St., Hopkinton, MA  01748

---

From: Greg White <[email-masked]>
Date: Thu, 12 Sep 2019 20:28:02 +0000

It may be ok for an SLA to be in place in order for the NQB DSCP to pass though unchanged.  The SLA would essentially be that both networks agree that 0x2A means NQB and should be treated accordingly (same priority as default, but queued separately where it is useful to do so).  ISPs that don't have network gear (e.g. DOCSIS or LTE) that supports the NQB PHB  (and don't have customers or peers that wish it to be preserved) don't need to preserve the marking.

I've not read RFC8100. I'll take a look.

-Greg

On 9/12/19, 3:30 AM, "[email-masked]" <[email-masked]> wrote:

    Hi Sebastian,

    
    
    Regards,
    
    Ruediger
    
    -----Ursprüngliche Nachricht-----
    Von: Sebastian Moeller <[email-masked]> 
    Gesendet: Donnerstag, 12. September 2019 10:41
    An: Geib, Rüdiger <[email-masked]>
    Cc: <[email-masked]> <[email-masked]>; [email-masked]
    Betreff: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!
    
    Dear Ruediger,

    
    
    Best Regards
    	Sebastian

---

From: Greg White <[email-masked]>
Date: Thu, 12 Sep 2019 23:34:14 +0000

Sebastian,

Since it seems we’re going in circles (or maybe it's a very slowly converging spiral), why don't I make the edits that I think you want to see, and then we can go from there.

Based on your arguments, I've agreed to add some detail in a section on "Applicability" that indicates more concretely that the intent of the NQB marking is not for capacity-seeking traffic.  That text will supplement the existing language in section 3 of the draft that currently reads:

  There are many applications that send traffic at relatively low data
   rates and/or in a fairly smooth and consistent manner such that they
   are highly unlikely to exceed the available capacity of the network
   path between source and sink.  These applications do not make use of
   network buffers, but nonetheless can be subjected to packet delay and
   delay variation as a result of sharing a network buffer with those
   that do make use of them.  Many of these applications are negatively
   affected by excessive packet delay and delay variation.  Such
   applications are ideal candidates to be queued separately from the
   capacity-seeking applications that are the cause of queue buildup,
   latency and loss.

   These Non-queue-building (NQB) flows are typically UDP flows that
   send traffic at a lower data rate and don't seek the capacity of the
   link (examples: online games, voice chat, DNS lookups).  Here the
   data rate is essentially limited by the Application itself.  In
   contrast, Queue-building (QB) flows include traffic which uses the
   Traditional TCP or QUIC, with BBR or other TCP congestion
   controllers.

If you can point out specifically what in that text led you to believe it was intended for capacity-seeking traffic, I would appreciate it.

-Greg

On 9/12/19, 2:19 AM, "Sebastian Moeller" <[email-masked]> wrote:

    Hi Greg,

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    

    Best Regards
    	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 13 Sep 2019 10:46:16 +0200

Hi Greg,

> On Sep 13, 2019, at 01:34, Greg White <[email-masked]> wrote:
> 
> Sebastian,
> 

	[SM] I am all for that approach, and I want to repeat that I realize that I am not the arbiter in this matter and convincing me (while nice from my perspective) is not important, convincing the chairs that my objections are baseless or handled well seems more important. 
	Since I seem to be the main person in this discussion with the "how does it look from the receiving end/end-user perspective" hat on, I will try to stay maximally critical (but let me repeat that I like the idea in general, modulo the side-effects I predict from a few of the detail-choices in the draft) that might be annoying on an inter-personal social level, but hopefully leads to a better technical soution.

> If you can point out specifically what in that text led you to believe it was intended for capacity-seeking traffic, I would appreciate it.

	[SM] Given how much language lawyering is applicable on when parsing RFCs I would like to see more precision for the following terms:

"relatively low data rates" as an application developer how do I decide what is relatively low? Or put differently relative to what exactly. Also If I have a paced-UDP based video delivery application that does application level switches between different bitrates based on experienced buffering/capacity/delay (like a paced DASH equivalent over UDP, or given that the sending rate for each segment is essentially fixed even standard paced-DASH) does this qualify for NQB or not? I believe the draft should make it crystal clear what can and what can not be considered to be NQB material.

"such that they are highly unlikely to exceed the available capacity of the network path between source and sink" while I understand the thought behind this, it is simply impossible for an application to know that, given that "available capacity" is not fixed (also a sending application will have very little information about the available path capacity when it starts). Maybe I am being to pedantic here, but starting with a wrong premise irks me.

"(examples: online games, voice chat, DNS lookups)" this still has the DNS lookups which collide with rfc8325, I understand why giving DNS lookups a low latency treatment (as DNS latency this directly translates into browsing latency, albeit only for uncached entries), but keeping DNS here has consequences on the wifi section IMHO.

"Queue-building (QB) flows include traffic which uses the Traditional TCP" why the qualifier? As far as I understand TCPPraque would also be classified to the QB queue by virtue of being capacity seeking. In a sense Prague is included in the "other TCP congestion controllers" set, but I would prefer a less ambiguous wording like 

"In contrast, Queue-building (QB) flows include all flows which seek to utilize the full capacity of the link like all TCP flows, as well as many SCTP  and UDP flows, like QUIC"
If you want you can add "all TCPs independent of the used congestion controller". 
If you disagree and believe TCPPrague should be allowed in the NQB queue, then at say so explicitly (but as far as I understand you really want to exclude all TCPs).

Or put differently I would like to see a description of the behavior precise enough that I can easily deduce from a packet capture whether a given flow meets the NQB requirements, in a sense an enumerated list of expected behaviors and of behaviors that disqualify for NQB marking/treatment.
I believe that would be in agreement with the stated goal that "the NQB designation and marking would be intended to convey verifiable traffic behavior, not needs or wants"

	Conceptually my next issue is that I think just trusting the end-points to mark correctly is too optimistic and that any scheduler giving special treatment according to NQB-ness must be required to ascertain that each individual flow is behaving according to the NQB requirements and ideally re-mark non-compliant flows CS0 to protect downstream devices (if NQB maps to anything but AC_BE). This requirement mainly is driven by the side effects NQB will have on wifi, so if the special wifi power of NQB goes away, this issue gets less important to me (also there are wifi scenarios where queue protection as proposed upstream will not be sufficient).

	Final and strongest objection: incompatibility between current wifi gear and NQB. 

I hope we can all agree that :

a) Wifi is a shared system not only between up- and downstream between stations and APS, but also due to the fact that the same frequency bands are often shared between different wifi networks (either directly be using the same center frequency band, or by overlap of the side-bands).

b) no currently deployed wifi gear (NICs and APs) supports special treatment for NQB flows.

c) most deployed gear supports access classes and will in all likelihood use the default DSCP to AC mapping you described in a different post. (as far as I can tell WMM is mandatory for wifi >= 802.11n)

d) The AC priority system is essentially a (weak) precedence system (lower ACs might sneak in a tx_op even with higher AC queues full, but that is gong to be rare and can result in multi second delays for packets in the lower ACs, as observed in my home network). So in traditional thinking using ACs requires some sort of access control/rate limiting to avoid undesired starvation issues.

e) For quite a number of end-users the wifi rate is what limits the internet access and not the access link itself (at least for higher bandwidth internet plans in crowded RF environments like apartment buildings)

f) NQB is not intended to reduce the "performance of QB flows" but rather isolate well behaving sparse and/or paced fixed-rate traffic from the transient queue building effects of non-paced non-rate-limited flows.

g) NQB is intended such that  "that malicious or badly configured nodes can't abuse it."

h) NQB traffic is not rate limited or rate policed. Even if each individual flow is self rate-limiting the aggregate clearly is not intended to be.

i) For QB flows, the QB queue provides better performance (considering latency, loss and throughput) than the NQB queue

j) NQB-marking (hopefully) will one day survive end to end.

Now, from e) it follows that the ISPs upstream NQB aware AQM will not trigger in quite a number of instances, so making sure that NQB does no harm is up to the wifi gear. Due to h) it is clear that NQB traffic will be able to saturate the wifi link. Together with c) and d) this means that the mapping of NQB to AC by the default rules will define the precedence level of the saturating traffic. Due to a) good wifi citizens try to be considerate in not hogging a channel to themselves (and due to CSMA/CA users on the same channel should stochastically share tx_ops "fairly" in each AC). Most wifi traffic currently is using AC_BE, but due to j) and h) that might change in the future.

I now argue that using anything but AC_BE for NQB will violate f) and g). Essentially in this not unlikely scenario NQB will confer a super-power (lower latency and higher bandwidth) without any checks and balances (incompatible with g)). 
To make it explicit, with ACs > BE in play, AC_BE will essentially only get the left-over bandwidth, which runs counter to i) as now QB flows need to mark themselves such that they also get scheduled into the same AC as NQB to get a level playing field.

For me that obviously means that the choice of DSCP for NQB needs to take care that the above scenario can not happen.
So I vote to follow Ruediger's proposal for a 000xx1 codepoint for the intended PHB such that the status quo is not negatively affected. With the added bonus that this now has a chance of actually surviving end to end today.
NQB aware APs/NICs can then implement proper NQB handling with or without selective priority boosting to different ACs while making sure no other flows/APs are starved.

Since I have made this point several times, without convincing you, I would be happy to hear your rationale why this scenario is unlikely enough to still justify aiming for a DSCP that maps to a AC > BE?
Ideally that rationale would be more refined than just noting, that this kind of abuse is possible already today, the aim should be to at least "do no harm", not just "do not make it significantly worse".

Best Regards
	Sebastian

---

From: "Black, David" <[email-masked]>
Date: Sun, 15 Sep 2019 21:38:39 +0000

Resurfacing to top-post and answer Greg's original question:

> On your item 3, could you clarify what you mean?   RFC 8325 does not discuss a “non queue building” service class, and so I don’t see a technical inconsistency between the NQB draft and RFC8325.

The posted version of the NQB draft cites RFC 8325 and then proceeds to ignore it in that Section 8.3 of the NQB draft discusses only what to do with implementations that don't support RFC 8325 (not a good approach).  At a minimum, that needs to be complemented by a discussion of what to do with implementations that do support RFC 8325, and perhaps some discussion of how to cope with the inevitable mix of support/non-support that will occur in practice.  The email discussion that has transpired to date appears to be "digging in a useful place" on how to handle NQB traffic on 802.11 networks, in particular when RFC 8325 is implemented.

Thanks, --David

> From: tsvwg <[email-masked]> On Behalf Of Sebastian Moeller
> Sent: Friday, September 13, 2019 4:46 AM
> To: Greg White
> Cc: [email-masked]
> Subject: Re: [tsvwg] TSVWG: WG adoption of draft-white-tsvwg-nqb!
> 
> 
> [EXTERNAL EMAIL]
> 
> Hi Greg,
> 
> 
> 
> > Since it seems we’re going in circles (or maybe it's a very slowly converging
> spiral), why don't I make the edits that I think you want to see, and then we
> can go from there.
> 
> 	[SM] I am all for that approach, and I want to repeat that I realize that
> I am not the arbiter in this matter and convincing me (while nice from my
> perspective) is not important, convincing the chairs that my objections are
> baseless or handled well seems more important.
> 	Since I seem to be the main person in this discussion with the "how
> does it look from the receiving end/end-user perspective" hat on, I will try to
> stay maximally critical (but let me repeat that I like the idea in general,
> modulo the side-effects I predict from a few of the detail-choices in the
> draft) that might be annoying on an inter-personal social level, but hopefully
> leads to a better technical soution.
> 
> 
> >
> > Based on your arguments, I've agreed to add some detail in a section on
> "Applicability" that indicates more concretely that the intent of the NQB
> marking is not for capacity-seeking traffic.  That text will supplement the
> existing language in section 3 of the draft that currently reads:
> >
> > If you can point out specifically what in that text led you to believe it was
> intended for capacity-seeking traffic, I would appreciate it.
> 
> 	[SM] Given how much language lawyering is applicable on when
> parsing RFCs I would like to see more precision for the following terms:
> 
> "relatively low data rates" as an application developer how do I decide what
> is relatively low? Or put differently relative to what exactly. Also If I have a
> paced-UDP based video delivery application that does application level
> switches between different bitrates based on experienced
> buffering/capacity/delay (like a paced DASH equivalent over UDP, or given
> that the sending rate for each segment is essentially fixed even standard
> paced-DASH) does this qualify for NQB or not? I believe the draft should
> make it crystal clear what can and what can not be considered to be NQB
> material.
> 
> "such that they are highly unlikely to exceed the available capacity of the
> network path between source and sink" while I understand the thought
> behind this, it is simply impossible for an application to know that, given that
> "available capacity" is not fixed (also a sending application will have very little
> information about the available path capacity when it starts). Maybe I am
> being to pedantic here, but starting with a wrong premise irks me.
> 
> "(examples: online games, voice chat, DNS lookups)" this still has the DNS
> lookups which collide with rfc8325, I understand why giving DNS lookups a
> low latency treatment (as DNS latency this directly translates into browsing
> latency, albeit only for uncached entries), but keeping DNS here has
> consequences on the wifi section IMHO.
> 
> "Queue-building (QB) flows include traffic which uses the Traditional TCP"
> why the qualifier? As far as I understand TCPPraque would also be classified
> to the QB queue by virtue of being capacity seeking. In a sense Prague is
> included in the "other TCP congestion controllers" set, but I would prefer a
> less ambiguous wording like
> 
> "In contrast, Queue-building (QB) flows include all flows which seek to utilize
> the full capacity of the link like all TCP flows, as well as many SCTP  and UDP
> flows, like QUIC"
> If you disagree and believe TCPPrague should be allowed in the NQB queue,
> then at say so explicitly (but as far as I understand you really want to exclude
> all TCPs).
> 
> Or put differently I would like to see a description of the behavior precise
> enough that I can easily deduce from a packet capture whether a given flow
> meets the NQB requirements, in a sense an enumerated list of expected
> behaviors and of behaviors that disqualify for NQB marking/treatment.
> I believe that would be in agreement with the stated goal that "the NQB
> behavior, not needs or wants"
> 
> 
> 
> 	Conceptually my next issue is that I think just trusting the end-points
> to mark correctly is too optimistic and that any scheduler giving special
> treatment according to NQB-ness must be required to ascertain that each
> individual flow is behaving according to the NQB requirements and ideally re-
> mark non-compliant flows CS0 to protect downstream devices (if NQB maps
> to anything but AC_BE). This requirement mainly is driven by the side effects
> NQB will have on wifi, so if the special wifi power of NQB goes away, this
> issue gets less important to me (also there are wifi scenarios where queue
> protection as proposed upstream will not be sufficient).
> 
> 
> a) Wifi is a shared system not only between up- and downstream between
> stations and APS, but also due to the fact that the same frequency bands are
> often shared between different wifi networks (either directly be using the
> same center frequency band, or by overlap of the side-bands).
> 
> c) most deployed gear supports access classes and will in all likelihood use the
> default DSCP to AC mapping you described in a different post. (as far as I can
> tell WMM is mandatory for wifi >= 802.11n)
> 
> d) The AC priority system is essentially a (weak) precedence system (lower
> ACs might sneak in a tx_op even with higher AC queues full, but that is gong
> to be rare and can result in multi second delays for packets in the lower ACs,
> as observed in my home network). So in traditional thinking using ACs
> requires some sort of access control/rate limiting to avoid undesired
> starvation issues.
> 
> e) For quite a number of end-users the wifi rate is what limits the internet
> access and not the access link itself (at least for higher bandwidth internet
> plans in crowded RF environments like apartment buildings)
> 
> f) NQB is not intended to reduce the "performance of QB flows" but rather
> isolate well behaving sparse and/or paced fixed-rate traffic from the
> transient queue building effects of non-paced non-rate-limited flows.
> 
> h) NQB traffic is not rate limited or rate policed. Even if each individual flow is
> self rate-limiting the aggregate clearly is not intended to be.
> 
> i) For QB flows, the QB queue provides better performance (considering
> latency, loss and throughput) than the NQB queue
> 
> Now, from e) it follows that the ISPs upstream NQB aware AQM will not
> trigger in quite a number of instances, so making sure that NQB does no
> harm is up to the wifi gear. Due to h) it is clear that NQB traffic will be able to
> saturate the wifi link. Together with c) and d) this means that the mapping of
> NQB to AC by the default rules will define the precedence level of the
> saturating traffic. Due to a) good wifi citizens try to be considerate in not
> hogging a channel to themselves (and due to CSMA/CA users on the same
> channel should stochastically share tx_ops "fairly" in each AC). Most wifi
> traffic currently is using AC_BE, but due to j) and h) that might change in the
> future.
> 
> I now argue that using anything but AC_BE for NQB will violate f) and g).
> Essentially in this not unlikely scenario NQB will confer a super-power (lower
> latency and higher bandwidth) without any checks and balances
> (incompatible with g)).
> To make it explicit, with ACs > BE in play, AC_BE will essentially only get the
> left-over bandwidth, which runs counter to i) as now QB flows need to mark
> themselves such that they also get scheduled into the same AC as NQB to get
> a level playing field.
> 
> 
> For me that obviously means that the choice of DSCP for NQB needs to take
> care that the above scenario can not happen.
> So I vote to follow Ruediger's proposal for a 000xx1 codepoint for the
> intended PHB such that the status quo is not negatively affected. With the
> added bonus that this now has a chance of actually surviving end to end
> today.
> NQB aware APs/NICs can then implement proper NQB handling with or
> without selective priority boosting to different ACs while making sure no
> other flows/APs are starved.
> 
> 
> Since I have made this point several times, without convincing you, I would
> be happy to hear your rationale why this scenario is unlikely enough to still
> justify aiming for a DSCP that maps to a AC > BE?
> Ideally that rationale would be more refined than just noting, that this kind of
> abuse is possible already today, the aim should be to at least "do no harm",
> not just "do not make it significantly worse".
> 
> 
> Best Regards
> 	Sebastian
> 
> 
>

---

From: Greg White <[email-masked]>
Date: Thu, 19 Sep 2019 19:01:55 +0000

Thanks for the clarification.  We can certainly provide some more discussion on NQB in the context of existing WiFi equipment and networks (both RFC8325-compliant and non-compliant), as well a takeaway for me from the discussion is that we would be well served by making further recommendations around possibilities to better align WiFi equipment with the intent of the NQB PHB (e.g. segregation without prioritization & traffic protection).

I'll make a first attempt at this for the WG draft 00.

-Greg

On 9/15/19, 3:40 PM, "Black, David" <[email-masked]> wrote:

    
    
    Thanks, --David