Subject: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-06.txt

From: [email-masked]
Date: Mon, 12 Jul 2021 13:46:27 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Transport Area Working Group WG of the IETF.

        Title           : A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services
        Authors         : Greg White
                          Thomas Fossati
	Filename        : draft-ietf-tsvwg-nqb-06.txt
	Pages           : 18
	Date            : 2021-07-12

Abstract:
   This document specifies properties and characteristics of a Non-
   Queue-Building Per-Hop Behavior (NQB PHB).  The purpose of this NQB
   PHB is to provide a separate queue that enables smooth, low-data-
   rate, application-limited traffic flows, which would ordinarily share
   a queue with bursty and capacity-seeking traffic, to avoid the
   latency, latency variation and loss caused by such traffic.  This PHB
   is implemented without prioritization and without rate policing,
   making it suitable for environments where the use of either these
   features may be restricted.  The NQB PHB has been developed primarily
   for use by access network segments, where queuing delays and queuing
   loss caused by Queue-Building protocols are manifested, but its use
   is not limited to such segments.  In particular, applications to
   cable broadband links, Wi-Fi links, and mobile network radio and core
   segments are discussed.  This document recommends a specific
   Differentiated Services Code Point (DSCP) to identify Non-Queue-
   Building flows.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/

There is also an HTML version available at:
https://www.ietf.org/archive/id/draft-ietf-tsvwg-nqb-06.html

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-nqb-06

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Greg White <[email-masked]>
Date: Mon, 12 Jul 2021 21:30:40 +0000

Hi all,

This update to the NQB draft includes the following changes:

- Recommends DSCPs 5 & 45 based on the analysis presented by Ana Custura at the last IETF
- Clarifies what the PHB provides, in the abstract & intro
- A more concrete description of NQB behavioral characteristics (including a rule-of-thumb value of 1 Mbps maximum burst rate)
- Moved the text on DSCP re-marking pathologies to an Appendix
- Further restructuring of the WiFi section
- Security section mentions default WiFi configuration
- Moved several references to Normative
- Multiple other editorial fixes

Many of these changes were spurred by off list comments received from David Black, Stuart Cheshire and Bob Briscoe (thanks!).

-Greg

On 7/12/21, 2:47 PM, "tsvwg on behalf of [email-masked]" <[email-masked] on behalf of [email-masked]> wrote:

    Abstract:

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 16 Jul 2021 12:26:54 +0100

Greg, list,

Here's my review of the nqb draft-06:

      ==Gap: Partial Deployment==

In §6, it says:

    "In the case of an NQB flow that isn't marked as NQB and ends up in
    the QB queue, it would only impact its own quality of service, and
    so it seems to be of lesser concern."

This made me realize that the draft doesn't say more generally what will 
be the outcome if an NQB flow ends up in a QB queue, just simply because 
NQB is not supported at the bottleneck. It talks about this in specific 
cases (e.g. WiFi), but not in general.

      ==SHOULDs or MUSTs?==

It might help to add some reasoning for why the draft allows 
implementers /not/ to comply with the 'SHOULDs'.
To help in this process, below I've tried to describe the exceptions to 
each SHOULD, so it could be written as "MUST do x..., except if y...".

  *   5. DSCP Marking of NQB Traffic
      o "...SHOULD identify themselves to the network using a Diffserv
        Code Point (DSCP) of 45..."
          + Doesn't say what valid cases would be for not using 45 and
            how that would interwork with other networks
          + e.g. some other DSCP agreed locally with the access network
            would work fine, but some other DSCP to work with an
            unmanaged home WiFi might not work with the rest of the path.
      o "If the application's traffic exceeds more than a few packets
        per RTT, or exceeds approximately 1 Mbps on an instantaneous
        (inter-packet) basis, the application SHOULD NOT mark its
        traffic with the NQB DSCP."
          + If this were 'MUST NOT' it would be too prescriptive as the
            Internet scales. But perhaps it's better to say 'MUST NOT'
            and allow more flexibility in the quantification. For
            instance, instead of 'a few' and 'approximately 1 Mbps',
            explain how these numbers might scale as the Internet scales.
      o "the application SHOULD instead implement a congestion control
        mechanism, for example as described in Section 3.1 of [RFC8085]
        or [I-D.ietf-tsvwg-ecn-l4s-id]."
          + I think it's best to say neither SHOULD nor MUST here, but
            instead refer to drafts that say normative things. Something
            like this: "the application has to instead implement a
            relevant congestion control mechanism, for example...".
            After all, an NQB RFC is not really entitled to say what
            non-NQB traffic should do.
      o "...nodes that do not support the NQB PHB SHOULD treat NQB
        marked traffic the same as traffic marked "Default", and SHOULD
        preserve the NQB marking."
          + Again, this is putting requirements on non-NQB nodes, but in
            this case the requirement is not on the implementer, rather
            it's about the classifier config of the operator. So this
            ought to be phrased as operator guidance (e.g. RECOMMENDED).
          + And it needs to say what will happen in pre-existing
            networks, where the operator is not aware of either of these
            recommendations (see earlier point about partial deployment).
  *   5.1. End-to-end usage and DSCP Re-marking
      o "Absent an explicit agreement to the contrary, networks that
        support the NQB PHB SHOULD preserve a DSCP marking distinction
        between NQB traffic and Default traffic when forwarding via an
        interconnect from or to another network. To facilitate the
        default treatment of NQB traffic in backbones and core networks,
        networks SHOULD remap NQB traffic (DSCP 45) to DSCP 5 prior to
        interconnection, unless agreed otherwise between the
        interconnecting partners."
          + Both these SHOULDs could be MUSTs if the sentence started
            with "To support NQB, ..." because they both allow a locally
            agreed exception.
          + One reason for not complying would be lack of space for the
            marking (e.g. MPLS or Ethernet header space). So it would be
            worth saying that one way to preserve the distinction would
            be encapsulation, with the distinction only preserved in the
            inner.
      o "In order to enable interoperability with WiFi equipment,
        networks SHOULD remap NQB traffic (e.g. DSCP 5) to DSCP 45..."
          + Again, this could be made into a MUST by adding "To support
            NQB, ..." and allowing locally agreed exceptions.
  *   5.2. Aggregation of the NQB PHB with other Diffserv PHBs
      o "the NQB DSCP SHOULD be preserved"
          + As above, explain that you are talking about encapsulation
            to preserve the distinction by preserving the inner
          + (and it would be worth clarifying that the degradations
            discussed in the previous para are only within the
            aggregation network, not after traffic has been
            disaggregated again).
  *   6. Non-Queue-Building PHB Requirements
      o There are lots of SHOULDs here, without any reasoning given for
        why exceptions are allowed, or what sort of exceptions are not
        allowed. They all relate to preserving vs. enforcing incentives.
        So I think it could be said explicitly that where SHOULD rather
        than MUST is used in this section, it is because the details
        depend on the threat level in the deployment environment and
        whether incentives are working sufficiently.
  *   11. Example Use Cases
      o "To support the NQB PHB, the mobile network SHOULD be configured
        to give UEs a dedicated, low-latency, non-GBR, EPS bearer"
          + I assume the SHOULD is because it depends on how things are
            done locally. This could be handled by describing this
            non-normatively, as a way to satisfy the 'MUST' requirement
            in §6 to provide a separate NQB queue.
          + Similarly "SHOULD be routed" and "SHOULD NOT be routed" can
            be non-normative as ways to satisfy earlier requirements in
            current mobile networks.
      o "WiFi equipment SHOULD map the NQB codepoint 45 into a separate
        queue that shares an Access Category"
          + As for mobile, this seems to be a way to satisfy the earlier
            requirements for the /current/ way that WiFi works. These
            are examples, not normative requirements, aren't they?
          + Ditto for "SHOULD deploy a policing function" and "SHOULD
            map the recommended NQB code point 45 to UP_5"
      o However, I understand that you might want to draw implementers'
        attention to what they have to do, by using capitalized
        requirements.

      ==Flow Queuing in scope or not?==

I think the draft ought to specify that its scope is primarily shared 
queues, although there would be some merit in saying how NQB would work 
with an FQ. Per-flow queues provide the necessary isolation, but there's 
no need to make any of the other requirements apply to an FQ {however, 
I'm not sure - see Note 1}. Stated another way, there has been no 
standardization of how FQ systems handle DSCPs, but this draft would be 
the wrong place to introduce that subject for one specific DSCP.

{Note 1}: Except, the isolation provided by an FQ is not absolute. E.g. 
an attacker can spoof flow identifiers on well-known ports in the hope 
it will match a flow ID being used by a genuine NQB flow and create a 
long-standing queue (which might otherwise just exist as a series of 
transient queue instances for sparse packets). I doubt it would be 
possible to provide local per-queue traffic protection against such 
traffic (nothing to distinguish it). But this is where other protection 
techniques come in, like scrubbing farms that can identify such attack 
traffic closer to the spoofed sources.

      ==Partially described scenarios==

I didn't find the following scenarios easy to understand:

  * "Active Queue Management (AQM) mechanisms (such as PIE [RFC8033],
    DOCSIS-PIE [RFC8034], or CoDel [RFC8289]) can improve the quality of
    experience for latency sensitive applications, but there are
    practical limits to the amount of improvement that can be achieved
    without impacting the throughput of capacity-seeking applications,
    particularly when only a few of such flows are present."
      o For all the text after the 'but', I think there's some scenario
        in the authors' heads that isn't fully explained.
  * "Either approach comes with trade-offs: aggregating with
    Default/Elastic traffic could result in a degradation of
    loss/latency/jitter performance for NQB traffic, while aggregating
    with Real-Time risks creating an incentive for mismarking of
    non-compliant traffic as NQB."
      o The scenario isn't clear - is it talking about a bottleneck
        within the aggregated network, or after it?
      o The sentence about r-t incentives needs to explain how it
        reaches that conclusion, rather than expecting the reader to
        recreate the explanation.
  * "In the case of the pipe model, any DSCP manipulation (re-marking)
    of the outer header by intermediate nodes would be discarded at
    tunnel egress, potentially improving the possibility of achieving
    NQB treatment in subsequent nodes."
      o ...Or potentially making it worse? It all depends whether the
        inner marking is meaningful edge-to-edge. I presume the word
        'potentially' alludes to this, but the sentence is rather opaque
        as it stands.
  * "...tunnel protocols that are sensitive to reordering can result in
    undesirable interactions..."
      o "undesirable interactions" is rather opaque. Does it mean within
        the tunnel? or after the egress?
  * "into a separate queue that shares an Access Category with default
    traffic "
      o The word 'shares' threw me at first - I think I've now
        understood; it must just mean 'it is within the same Access
        Category'. I was wondering how it could both be separate and be
        shared.
  * "This document recommends the implementation of a traffic protection
    mechanism to achieve this goal, but recognizes that other options
    may be more desirable in certain situations."
      o I don't think 'other options' have been described in the doc
        (yet). So this seems opaque

      ==Relationship with L4S (§9)==

Ought to say that a non-relationship to L4S is also fine.
And ought to explain that L4S is experimental, while NQB is stds track.

      ==Structure==

There were times when I couldn't really see a logical flow.

Here's the current ToC for reference:
  Table of Contents
1.  Introduction
2.  Requirements Language
3.  Non-Queue-Building Behavior
4.  The NQB PHB and its Relationship to the Diffserv Architecture
5.  DSCP Marking of NQB Traffic
5.1.  End-to-end usage and DSCP Re-marking
5.2.  Aggregation of the NQB PHB with other Diffserv PHBs
6.  Non-Queue-Building PHB Requirements
7.  Impact on Higher Layer Protocols
8.  The NQB PHB and Tunnels
9.  Relationship to L4S
10. Configuration and Management
11. Example Use Cases
etc

§3 serves both as initial tutorial material and as the specification of 
the sender behaviour that is normatively defined as a SHOULD at the 
start of §5. Perhaps the text could be split in two, for these two 
purposes, then most of this section could be moved to a new §5.1, where 
it could be merged with the text in §5 prior to §5.1. Then §5 has a more 
logical structure for all the normative DSCP handling (source nodes, 
interconnect nodes, aggregation nodes)

5.  DSCP Marking of NQB Traffic
5.1.  Non-Queue-Building Source Behavior
5.2   End-to-end usage and DSCP Re-marking
5.3.  Aggregation of the NQB PHB with other Diffserv PHBs

Also, the para just before the current 5.1 ought really to be in the PHB 
section (§6). It follows the incentives logic of the previous paras, but 
I'm not sure whether that's the more important logical flow (?).

Also, I would have thought "8. The NQB PHB and Tunnels" is most closely 
related to "Aggregation of the NQB PHB with other Diffserv PHBs" and 
ought to be perhaps at §5.4. But I'm not sure whether §8 is meant to be 
more about the PHB than the DSCP. The title says PHB, but it seems to be 
more about the DSCP, and therefore rightly belongs in §5.

The single para section "7.  Impact on Higher Layer Protocols" is really 
just a specific point about reordering caused by traffic protection. I 
don't see the need for a separate section.

It also seems odd to have "Relationship to Diffserv" and "Relationship 
to L4S" at opposite ends of the document. I suspect this is a result of 
the original purpose being tutorial, and now it's turned into a spec.

Pulling that all together, might look sthg like this:

1.  Introduction
2.  Requirements Language
3.  Context
3.1.  Non-Queue-Building Behavior
3.2.  Relationship to the Diffserv Architecture
3.3.  Relationship to L4S
4.  DSCP Marking of NQB Traffic
4.1.  Non-Queue-Building Sender Behavior
4.2   End-to-end usage and DSCP Re-marking
4.3.  Aggregation of the NQB PHB with other Diffserv PHBs
4.4.  The NQB DSCP and Tunnels
5.  Non-Queue-Building PHB Requirements
6. Configuration and Management
7. Example Use Cases
etc

HTH
Regards

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Greg White <[email-masked]>
Date: Thu, 29 Jul 2021 02:48:12 +0000

Bob,

Thank you very much for your review.  I am preparing a draft-07 that addresses many of these comments.  Please see some responses below.

-Greg

From: Bob Briscoe <[email-masked]>
Date: Friday, July 16, 2021 at 5:27 AM
To: Greg White <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-06.txt

Greg, list,

In §6, it says:
"In the case of an NQB flow that isn't marked as NQB and ends up in the QB queue, it would only impact its own quality of service, and so it seems to be of lesser concern."
This made me realize that the draft doesn't say more generally what will be the outcome if an NQB flow ends up in a QB queue, just simply because NQB is not supported at the bottleneck. It talks about this in specific cases (e.g. WiFi), but not in general.
[GW] Well there is also some related text in 5.2 and in the paragraph preceding 5.1.  But, I agree, this could be covered more directly.  I will add some text on that topic in 5.2 (Aggregation), which is where it seems most appropriate to me.

==SHOULDs or MUSTs?==

It might help to add some reasoning for why the draft allows implementers /not/ to comply with the 'SHOULDs'.
To help in this process, below I've tried to describe the exceptions to each SHOULD, so it could be written as "MUST do x..., except if y...".
·          5. DSCP Marking of NQB Traffic
o    "...SHOULD identify themselves to the network using a Diffserv Code Point (DSCP) of 45..."
§  Doesn't say what valid cases would be for not using 45 and how that would interwork with other networks
§  e.g. some other DSCP agreed locally with the access network would work fine, but some other DSCP to work with an unmanaged home WiFi might not work with the rest of the path.
[GW] Ok, I added “In networks where another (e.g. a local-use) codepoint is designated for NQB traffic, or where specialized PHBs are available that can meet specific application requirements (e.g. a guaranteed-latency path for voice traffic), it may be preferred to use another DSCP.”
o    "If the application's traffic exceeds more than a few packets per RTT, or exceeds approximately 1 Mbps on an instantaneous (inter-packet) basis, the application SHOULD NOT mark its traffic with the NQB DSCP."
§  If this were 'MUST NOT' it would be too prescriptive as the Internet scales. But perhaps it's better to say 'MUST NOT' and allow more flexibility in the quantification. For instance, instead of 'a few' and 'approximately 1 Mbps', explain how these numbers might scale as the Internet scales.
[GW] I understand your point.  But am going to hold off on making this change.  It isn’t immediately clear to me how to structure this as a MUST NOT.  Maybe we can discuss this in the WG.
o    "the application SHOULD instead implement a congestion control mechanism, for example as described in Section 3.1 of [RFC8085] or [I-D.ietf-tsvwg-ecn-l4s-id]."
§  I think it's best to say neither SHOULD nor MUST here, but instead refer to drafts that say normative things. Something like this: "the application has to instead implement a relevant congestion control mechanism, for example...". After all, an NQB RFC is not really entitled to say what non-NQB traffic should do.
[GW] I agree.
o    "...nodes that do not support the NQB PHB SHOULD treat NQB marked traffic the same as traffic marked "Default", and SHOULD preserve the NQB marking."
§  Again, this is putting requirements on non-NQB nodes, but in this case the requirement is not on the implementer, rather it's about the classifier config of the operator. So this ought to be phrased as operator guidance (e.g. RECOMMENDED).
§  And it needs to say what will happen in pre-existing networks, where the operator is not aware of either of these recommendations (see earlier point about partial deployment).
[GW] I believe that I’ve got this covered now in the Aggregation section.
·          5.1. End-to-end usage and DSCP Re-marking
o    "Absent an explicit agreement to the contrary, networks that support the NQB PHB SHOULD preserve a DSCP marking distinction between NQB traffic and Default traffic when forwarding via an interconnect from or to another network. To facilitate the default treatment of NQB traffic in backbones and core networks, networks SHOULD remap NQB traffic (DSCP 45) to DSCP 5 prior to interconnection, unless agreed otherwise between the interconnecting partners."
§  Both these SHOULDs could be MUSTs if the sentence started with "To support NQB, ..." because they both allow a locally agreed exception.
§  One reason for not complying would be lack of space for the marking (e.g. MPLS or Ethernet header space). So it would be worth saying that one way to preserve the distinction would be encapsulation, with the distinction only preserved in the inner.
o    "In order to enable interoperability with WiFi equipment, networks SHOULD remap NQB traffic (e.g. DSCP 5) to DSCP 45..."
§  Again, this could be made into a MUST by adding "To support NQB, ..." and allowing locally agreed exceptions.
[GW]  For the first one (preserving a marking distinction), I agree with you (and made it a MUST).  For the others that discuss specific DSCPs, I think we need to leave them as SHOULDs because (by definition) a PHB spec only “recommends” DSCPs and does not mandate them.

·          5.2. Aggregation of the NQB PHB with other Diffserv PHBs
o    "the NQB DSCP SHOULD be preserved"
§  As above, explain that you are talking about encapsulation to preserve the distinction by preserving the inner
§  (and it would be worth clarifying that the degradations discussed in the previous para are only within the aggregation network, not after traffic has been disaggregated again).
[GW] I’ll cover the first sub-bullet, but I’m not sure what more I could say on the second sub-bullet. The degradations (loss/latency/jitter), while they would be introduced in an aggregation node, clearly are felt end-to-end.  The statement around preserving NQB DSCP already mentions that it should be preserved “in order to limit the negative impact that such networks would have on end-to-end performance for NQB traffic.”  Maybe I’m misunderstanding what you mean?
·          6. Non-Queue-Building PHB Requirements
o    There are lots of SHOULDs here, without any reasoning given for why exceptions are allowed, or what sort of exceptions are not allowed. They all relate to preserving vs. enforcing incentives. So I think it could be said explicitly that where SHOULD rather than MUST is used in this section, it is because the details depend on the threat level in the deployment environment and whether incentives are working sufficiently.
[GW] I think your comment mainly applies to two of the SHOULDs (equivalent forwarding preference requirement and fair scheduler requirement), so I added it there.  The one that I’m debating is the “… SHOULD NOT be rate limited or rate policed separately from QB traffic …” requirement.  I don’t think this one is entirely related to incentives.  Even in a controlled environment where there is no threat of mismarking, I think this would still be the recommendation.  But, I’m having trouble justifying why it should be a SHOULD as opposed to a MUST.  My original rationale was that there is always a corner case, but I think I would be ok with it either way.

·          11. Example Use Cases
o    "To support the NQB PHB, the mobile network SHOULD be configured to give UEs a dedicated, low-latency, non-GBR, EPS bearer"
§  I assume the SHOULD is because it depends on how things are done locally. This could be handled by describing this non-normatively, as a way to satisfy the 'MUST' requirement in §6 to provide a separate NQB queue.
§  Similarly "SHOULD be routed" and "SHOULD NOT be routed" can be non-normative as ways to satisfy earlier requirements in current mobile networks.
o    "WiFi equipment SHOULD map the NQB codepoint 45 into a separate queue that shares an Access Category"
§  As for mobile, this seems to be a way to satisfy the earlier requirements for the /current/ way that WiFi works. These are examples, not normative requirements, aren't they?
§  Ditto for "SHOULD deploy a policing function" and "SHOULD map the recommended NQB code point 45 to UP_5"
o    However, I understand that you might want to draw implementers' attention to what they have to do, by using capitalized requirements.
[GW] Yes, I like these being called out as requirement statements.
==Flow Queuing in scope or not?==

I think the draft ought to specify that its scope is primarily shared queues, although there would be some merit in saying how NQB would work with an FQ. Per-flow queues provide the necessary isolation, but there's no need to make any of the other requirements apply to an FQ {however, I'm not sure - see Note 1}. Stated another way, there has been no standardization of how FQ systems handle DSCPs, but this draft would be the wrong place to introduce that subject for one specific DSCP.

{Note 1}: Except, the isolation provided by an FQ is not absolute. E.g. an attacker can spoof flow identifiers on well-known ports in the hope it will match a flow ID being used by a genuine NQB flow and create a long-standing queue (which might otherwise just exist as a series of transient queue instances for sparse packets). I doubt it would be possible to provide local per-queue traffic protection against such traffic (nothing to distinguish it). But this is where other protection techniques come in, like scrubbing farms that can identify such attack traffic closer to the spoofed sources.
[GW] Good suggestion.  I added a sentence in the Introduction.
==Partially described scenarios==
I didn't find the following scenarios easy to understand:
·         "Active Queue Management (AQM) mechanisms (such as PIE [RFC8033], DOCSIS-PIE [RFC8034], or CoDel [RFC8289]) can improve the quality of experience for latency sensitive applications, but there are practical limits to the amount of improvement that can be achieved without impacting the throughput of capacity-seeking applications, particularly when only a few of such flows are present."
o    For all the text after the 'but', I think there's some scenario in the authors' heads that isn't fully explained.
·         "Either approach comes with trade-offs: aggregating with Default/Elastic traffic could result in a degradation of loss/latency/jitter performance for NQB traffic, while aggregating with Real-Time risks creating an incentive for mismarking of non-compliant traffic as NQB."
o    The scenario isn't clear - is it talking about a bottleneck within the aggregated network, or after it?
o    The sentence about r-t incentives needs to explain how it reaches that conclusion, rather than expecting the reader to recreate the explanation.
[GW] Ok, I added some more detail for those.

·         "In the case of the pipe model, any DSCP manipulation (re-marking) of the outer header by intermediate nodes would be discarded at tunnel egress, potentially improving the possibility of achieving NQB treatment in subsequent nodes."
o    ...Or potentially making it worse? It all depends whether the inner marking is meaningful edge-to-edge. I presume the word 'potentially' alludes to this, but the sentence is rather opaque as it stands.
·         "...tunnel protocols that are sensitive to reordering can result in undesirable interactions..."
o    "undesirable interactions" is rather opaque. Does it mean within the tunnel? or after the egress?
[GW] The language comes from RFC2983, and the sentence begins with “As is discussed in RFC2983, …”  I think it is ok to assume a confused reader will know where to look to figure this one out.
·         "into a separate queue that shares an Access Category with default traffic "
o    The word 'shares' threw me at first - I think I've now understood; it must just mean 'it is within the same Access Category'. I was wondering how it could both be separate and be shared.
[GW] Ok, fixed.
·         "This document recommends the implementation of a traffic protection mechanism to achieve this goal, but recognizes that other options may be more desirable in certain situations."
o    I don't think 'other options' have been described in the doc (yet). So this seems opaque
[GW] Two other options are provided in the PHB Requirements section.

[GW] Done.
==Structure==

§3 serves both as initial tutorial material and as the specification of the sender behaviour that is normatively defined as a SHOULD at the start of §5. Perhaps the text could be split in two, for these two purposes, then most of this section could be moved to a new §5.1, where it could be merged with the text in §5 prior to §5.1. Then §5 has a more logical structure for all the normative DSCP handling (source nodes, interconnect nodes, aggregation nodes)

Also, the para just before the current 5.1 ought really to be in the PHB section (§6). It follows the incentives logic of the previous paras, but I'm not sure whether that's the more important logical flow (?).

Also, I would have thought "8. The NQB PHB and Tunnels" is most closely related to "Aggregation of the NQB PHB with other Diffserv PHBs" and ought to be perhaps at §5.4. But I'm not sure whether §8 is meant to be more about the PHB than the DSCP. The title says PHB, but it seems to be more about the DSCP, and therefore rightly belongs in §5.

The single para section "7.  Impact on Higher Layer Protocols" is really just a specific point about reordering caused by traffic protection. I don't see the need for a separate section.

It also seems odd to have "Relationship to Diffserv" and "Relationship to L4S" at opposite ends of the document. I suspect this is a result of the original purpose being tutorial, and now it's turned into a spec.

1.  Introduction
2.  Requirements Language
3.  Context

[GW] I largely agree, and thanks for taking the time to think through and suggest this.  I’ve kept the Impact on Higher Layer Protocols as a separate section (I think it’s worth highlighting, if for no other reason than it is a mandatory section per RFC2745 Guidelines), but otherwise reorganized the material along these lines.

HTH
Regards

Bob

Hi all,

-Greg

On 7/12/21, 2:47 PM, "tsvwg on behalf of [email-masked]<mailto:[email-masked]>" <[email-masked] on behalf of [email-masked]><mailto:[email-masked]@ietf.org> wrote:

            Authors         : Greg White

                              Thomas Fossati

           Pages           : 18

           Date            : 2021-07-12

    Abstract:

       Building flows.

--

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 29 Jul 2021 15:06:21 +0100

Greg,

I've had a look at -07.
Thank you. See [BB] inline...

On 29/07/2021 03:48, Greg White wrote:
>
> Bob,
>
> Thank you very much for your review.  I am preparing a draft-07 that 
> addresses many of these comments. Please see some responses below.
>
> -Greg
>
> *From: *Bob Briscoe <[email-masked]>
> *Date: *Friday, July 16, 2021 at 5:27 AM
> *To: *Greg White <[email-masked]>, "[email-masked]" 
> <[email-masked]>
> *Subject: *Re: [tsvwg] I-D Action: draft-ietf-tsvwg-nqb-06.txt
>
> Greg, list,
>
>     "In the case of an NQB flow that isn't marked as NQB and ends up
>     in the QB queue, it would only impact its own quality of service,
>     and so it seems to be of lesser concern."
>
> This made me realize that the draft doesn't say more generally what 
> will be the outcome if an NQB flow ends up in a QB queue, just simply 
> because NQB is not supported at the bottleneck. It talks about this in 
> specific cases (e.g. WiFi), but not in general.
>
> [GW] Well there is also some related text in 5.2 and in the paragraph 
> preceding 5.1.  But, I agree, this could be covered more directly.  I 
> will add some text on that topic in 5.2 (Aggregation), which is where 
> it seems most appropriate to me.
>

[BB] Yes, the beginning of §4.2 covers this from the network point of 
view, thank you. I suggest
s/This preservation of the NQB marking/
  /This preservation of the NQB marking distinction/

I was also thinking from the sender's point of view that, if it marks 
its traffic NQB, even if the bottleneck doesn't support NQB, it at least 
won't get worse delay than if it had marked its traffic as Default (BE). 
I guess it would be futile to require non-NQB nodes not to selectively 
block or discard NQB traffic. But the sender does need some reassurance 
that the service will never be worse than Default, so that it doesn't 
have to think twice before marking as NQB.

> To help in this process, below I've tried to describe the exceptions 
> to each SHOULD, so it could be written as "MUST do x..., except if y...".
>
> o"...SHOULD identify themselves to the network using a Diffserv Code 
> Point (DSCP) of 45..."
>
> §Doesn't say what valid cases would be for not using 45 and how that 
> would interwork with other networks
>
> §e.g. some other DSCP agreed locally with the access network would 
> work fine, but some other DSCP to work with an unmanaged home WiFi 
> might not work with the rest of the path.
>
> [GW] Ok, I added “In networks where another (e.g. a local-use) 
> codepoint is designated for NQB traffic, or where specialized PHBs are 
> available that can meet specific application requirements (e.g. a 
> guaranteed-latency path for voice traffic), it may be preferred to use 
> another DSCP.”
>

[BB] "May be preferred" is sort-of worse. I was trying to replace 
vagueness with more certainty.

I was thinking more of something like. 'The "SHOULD" is intended not to 
preclude use of a different DSCP by arrangement with the local operator.'

> o"If the application's traffic exceeds more than a few packets per 
> RTT, or exceeds approximately 1 Mbps on an instantaneous 
> (inter-packet) basis, the application SHOULD NOT mark its traffic with 
> the NQB DSCP."
>
> §If this were 'MUST NOT' it would be too prescriptive as the Internet 
> scales. But perhaps it's better to say 'MUST NOT' and allow more 
> flexibility in the quantification. For instance, instead of 'a few' 
> and 'approximately 1 Mbps', explain how these numbers might scale as 
> the Internet scales.
>
> [GW] I understand your point.  But am going to hold off on making this 
> change.  It isn’t immediately clear to me how to structure this as a 
> MUST NOT.  Maybe we can discuss this in the WG.
>

[BB] I agree that it would probably be impossible to come up with a MUST 
NOT here. But with a SHOULD NOT, I still think the quantified 
requirement will become outdated unless it is somehow explained how it 
will scale in the future. Yes, for WG discussion.

> o"the application SHOULD instead implement a congestion control 
> mechanism, for example as described in Section 3.1 of [RFC8085] or 
> [I-D.ietf-tsvwg-ecn-l4s-id]."
>
> §I think it's best to say neither SHOULD nor MUST here, but instead 
> refer to drafts that say normative things. Something like this: "the 
> application has to instead implement a relevant congestion control 
> mechanism, for example...". After all, an NQB RFC is not really 
> entitled to say what non-NQB traffic should do.
>
> [GW] I agree.
>
> o"...nodes that do not support the NQB PHB SHOULD treat NQB marked 
> traffic the same as traffic marked "Default", and SHOULD preserve the 
> NQB marking."
>
> §Again, this is putting requirements on non-NQB nodes, but in this 
> case the requirement is not on the implementer, rather it's about the 
> classifier config of the operator. So this ought to be phrased as 
> operator guidance (e.g. RECOMMENDED).
>
> §And it needs to say what will happen in pre-existing networks, where 
> the operator is not aware of either of these recommendations (see 
> earlier point about partial deployment).
>

[BB] Yup

> o"Absent an explicit agreement to the contrary, networks that support 
> the NQB PHB SHOULD preserve a DSCP marking distinction between NQB 
> traffic and Default traffic when forwarding via an interconnect from 
> or to another network. To facilitate the default treatment of NQB 
> traffic in backbones and core networks, networks SHOULD remap NQB 
> traffic (DSCP 45) to DSCP 5 prior to interconnection, unless agreed 
> otherwise between the interconnecting partners."
>
> §Both these SHOULDs could be MUSTs if the sentence started with "To 
> support NQB, ..." because they both allow a locally agreed exception.
>
> §One reason for not complying would be lack of space for the marking 
> (e.g. MPLS or Ethernet header space). So it would be worth saying that 
> one way to preserve the distinction would be encapsulation, with the 
> distinction only preserved in the inner.
>
> o"In order to enable interoperability with WiFi equipment, networks 
> SHOULD remap NQB traffic (e.g. DSCP 5) to DSCP 45..."
>
> §Again, this could be made into a MUST by adding "To support NQB, ..." 
> and allowing locally agreed exceptions.
>
> [GW] For the first one (preserving a marking distinction), I agree 
> with you (and made it a MUST).  For the others that discuss specific 
> DSCPs, I think we need to leave them as SHOULDs because (by 
> definition) a PHB spec only “recommends” DSCPs and does not mandate them.
>

[BB] Then another way to tighten this up would be to try to describe 
valid exceptions.

> o"the NQB DSCP SHOULD be preserved"
>
> §As above, explain that you are talking about encapsulation to 
> preserve the distinction by preserving the inner
>
> §(and it would be worth clarifying that the degradations discussed in 
> the previous para are only within the aggregation network, not after 
> traffic has been disaggregated again).
>
> [GW] I’ll cover the first sub-bullet, but I’m not sure what more I 
> could say on the second sub-bullet. The degradations 
> (loss/latency/jitter), while they would be introduced in an 
> aggregation node, clearly are felt end-to-end.  The statement around 
> preserving NQB DSCP already mentions that it should be preserved “in 
> order to limit the negative impact that such networks would have on 
> end-to-end performance for NQB traffic.”  Maybe I’m misunderstanding 
> what you mean?
>

[BB] First, I've just realized that perhaps this should say "the NQB 
DSCP /distinction/ SHOULD be preserved".

Then, yes, I didn't mean performance degradations after the tunnel 
egress. I meant that it might be useful to explain how the DSCP 
distinction can be preserved both across the tunnel and after it. Not 
essential, because such tutorial is really the job of a Diffserv 
aggregation RFC. I meant that encapsulation can be used to aggregate 
DSCPs together (e.g. where MPLS or Ethernet provide fewer Traffic Class 
codepoints), but still preserve DSCP distinctions within the 
encapsulation (inner headers) so that, beyond the scope of the 
encapsulation the distinct DSCPs can be revealed again.

> oThere are lots of SHOULDs here, without any reasoning given for why 
> exceptions are allowed, or what sort of exceptions are not allowed. 
> They all relate to preserving vs. enforcing incentives. So I think it 
> could be said explicitly that where SHOULD rather than MUST is used in 
> this section, it is because the details depend on the threat level in 
> the deployment environment and whether incentives are working 
> sufficiently.
>
> [GW] I think your comment mainly applies to two of the SHOULDs 
> (equivalent forwarding preference requirement and fair scheduler 
> requirement), so I added it there.  The one that I’m debating is the 
> “… SHOULD NOT be rate limited or rate policed separately from QB 
> traffic …” requirement.  I don’t think this one is entirely related to 
> incentives.  Even in a controlled environment where there is no threat 
> of mismarking, I think this would still be the recommendation.  But, 
> I’m having trouble justifying why it should be a SHOULD as opposed to 
> a MUST.  My original rationale was that there is always a corner case, 
> but I think I would be ok with it either way.
>

[BB] Interesting that you've brought that up. I was looking at that and 
wondering whether it was intended to mean "...need not be rate limited 
or policed separately..." There is an obvious downside of rate policing 
two traffic classes separately when it's unnecessary: it unnecessarily 
limits users. But if the operator wants to limit its users 
unnecessarily, surely we don't need to stop it? Is there an interop 
concern here?

> · 11. Example Use Cases
>
> o"To support the NQB PHB, the mobile network SHOULD be configured to 
> give UEs a dedicated, low-latency, non-GBR, EPS bearer"
>
> §I assume the SHOULD is because it depends on how things are done 
> locally. This could be handled by describing this non-normatively, as 
> a way to satisfy the 'MUST' requirement in §6 to provide a separate 
> NQB queue.
>
> §Similarly "SHOULD be routed" and "SHOULD NOT be routed" can be 
> non-normative as ways to satisfy earlier requirements in current 
> mobile networks.
>
> o"WiFi equipment SHOULD map the NQB codepoint 45 into a separate queue 
> that shares an Access Category"
>
> §As for mobile, this seems to be a way to satisfy the earlier 
> requirements for the /current/ way that WiFi works. These are 
> examples, not normative requirements, aren't they?
>
> §Ditto for "SHOULD deploy a policing function" and "SHOULD map the 
> recommended NQB code point 45 to UP_5"
>
> oHowever, I understand that you might want to draw implementers' 
> attention to what they have to do, by using capitalized requirements.
>
> queues, although there would be some merit in saying how NQB would 
> work with an FQ. Per-flow queues provide the necessary isolation, but 
> there's no need to make any of the other requirements apply to an FQ 
> {however, I'm not sure - see Note 1}. Stated another way, there has 
> been no standardization of how FQ systems handle DSCPs, but this draft 
> would be the wrong place to introduce that subject for one specific DSCP.
>
> {Note 1}: Except, the isolation provided by an FQ is not absolute. 
> E.g. an attacker can spoof flow identifiers on well-known ports in the 
> hope it will match a flow ID being used by a genuine NQB flow and 
> create a long-standing queue (which might otherwise just exist as a 
> series of transient queue instances for sparse packets). I doubt it 
> would be possible to provide local per-queue traffic protection 
> against such traffic (nothing to distinguish it). But this is where 
> other protection techniques come in, like scrubbing farms that can 
> identify such attack traffic closer to the spoofed sources.
>
> ·"Active Queue Management (AQM) mechanisms (such as PIE [RFC8033], 
> experience for latency sensitive applications, but there are practical 
> limits to the amount of improvement that can be achieved without 
> impacting the throughput of capacity-seeking applications, 
> oFor all the text after the 'but', I think there's some scenario in 
> the authors' heads that isn't fully explained.
>
> ·"Either approach comes with trade-offs: aggregating with 
> oThe scenario isn't clear - is it talking about a bottleneck within 
> the aggregated network, or after it?
>
> oThe sentence about r-t incentives needs to explain how it reaches 
> that conclusion, rather than expecting the reader to recreate the 
> explanation.
>

[BB] I understand now. Much better.

> ·"In the case of the pipe model, any DSCP manipulation (re-marking) of 
> the outer header by intermediate nodes would be discarded at tunnel 
> egress, potentially improving the possibility of achieving NQB 
> treatment in subsequent nodes."
>
> o...Or potentially making it worse? It all depends whether the inner 
> marking is meaningful edge-to-edge. I presume the word 'potentially' 
> alludes to this, but the sentence is rather opaque as it stands.
>
> ·"...tunnel protocols that are sensitive to reordering can result in 
> o"undesirable interactions" is rather opaque. Does it mean within the 
> tunnel? or after the egress?
>
> [GW] The language comes from RFC2983, and the sentence begins with “As 
> is discussed in RFC2983, …”  I think it is ok to assume a confused 
> reader will know where to look to figure this one out.
>

[BB] It wasn't the pipe model I didn't understand. The pipe model 
certainly preserves the inner, but that's only useful if the mapping of 
the inner DSCP to a meaning  is shared between the operators of the two 
ends of the pipe, which are not necessarily neighbours. Or are you 
meaning that they can refer to RFC2983 for that as well?

> ·"into a separate queue that shares an Access Category with default 
> traffic "
>
> oThe word 'shares' threw me at first - I think I've now understood; it 
> must just mean 'it is within the same Access Category'. I was 
> wondering how it could both be separate and be shared.
>
> [GW] Ok, fixed.
>
> ·"This document recommends the implementation of a traffic protection 
> mechanism to achieve this goal, but recognizes that other options may 
> be more desirable in certain situations."
>
> oI don't think 'other options' have been described in the doc (yet). 
> So this seems opaque
>

[BB] Found them. Thanks.

> And ought to explain that L4S is experimental, while NQB is stds track.

[BB] Thx.
This section cites l4s-arch and aqm-dualq-coupled. Altho both those 
mention NQB, ecn-l4s-id is the most relevant L4S draft. It says how to 
interwork with NQB traffic (specifically §5.4.1). I suggest you cite it 
instead of, or in addition to l4s-arch.

> §3 serves both as initial tutorial material and as the specification 
> of the sender behaviour that is normatively defined as a SHOULD at the 
> purposes, then most of this section could be moved to a new §5.1, 
> where it could be merged with the text in §5 prior to §5.1. Then §5 
> has a more logical structure for all the normative DSCP handling 
> (source nodes, interconnect nodes, aggregation nodes)
>
> Also, the para just before the current 5.1 ought really to be in the 
> PHB section (§6). It follows the incentives logic of the previous 
> paras, but I'm not sure whether that's the more important logical flow 
> (?).
>
> Also, I would have thought "8. The NQB PHB and Tunnels" is most 
> closely related to "Aggregation of the NQB PHB with other Diffserv 
> PHBs" and ought to be perhaps at §5.4. But I'm not sure whether §8 is 
> meant to be more about the PHB than the DSCP. The title says PHB, but 
> it seems to be more about the DSCP, and therefore rightly belongs in §5.
>
> The single para section "7.  Impact on Higher Layer Protocols" is 
> really just a specific point about reordering caused by traffic 
> protection. I don't see the need for a separate section.
>
> to L4S" at opposite ends of the document. I suspect this is a result 
> of the original purpose being tutorial, and now it's turned into a spec.
>
> [GW] I largely agree, and thanks for taking the time to think through 
> and suggest this.  I’ve kept the Impact on Higher Layer Protocols as a 
> separate section (I think it’s worth highlighting, if for no other 
> reason than it is a mandatory section per RFC2745 Guidelines), but 
> otherwise reorganized the material along these lines.
>

[BB] OK, glad that worked out. And I didn't realize that was a mandatory 
section - that explains it.

Bob

>     On 7/12/21, 2:47 PM, "tsvwg on behalf [email-masked]  <mailto:[email-masked]>"<[email-masked] on behalf of [email-masked]>  <mailto:[email-masked]@ietf.org>  wrote:
>
>          https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/  <https://datatracker.ietf.org/doc/draft-ietf-tsvwg-nqb/>
>
>          https://www.ietf.org/archive/id/draft-ietf-tsvwg-nqb-06.html  <https://www.ietf.org/archive/id/draft-ietf-tsvwg-nqb-06.html>
>
>          https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-nqb-06  <https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-nqb-06>
>
>          ftp://ftp.ietf.org/internet-drafts/  <ftp://ftp.ietf.org/internet-drafts/>
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 29 Jul 2021 21:39:19 +0100

Greg,

Following up my own reply - see [BB2]

On 29/07/2021 15:06, Bob Briscoe wrote:
>> (inter-packet) basis, the application SHOULD NOT mark its traffic 
>> with the NQB DSCP."
>>
>> [GW] I understand your point.  But am going to hold off on making 
>> this change.  It isn’t immediately clear to me how to structure this 
>> as a MUST NOT.  Maybe we can discuss this in the WG.
>>
>
> [BB] I agree that it would probably be impossible to come up with a 
> MUST NOT here. But with a SHOULD NOT, I still think the quantified 

[BB2] I was thinking that none of the proposed metrics change much over 
the years as link capacity scales. So are none of these in the right units?
* "a few packets per RTT"
* 1Mb/s instantaneous data rate
* one 1250B packet every 10ms

Straw man:
I think the real requirement here is that there's a lot of space between 
the packets, relative to the time the packets spend serializing. That's 
what ensures they add very little to queuing.

You have to know the link rate to know the serialization time. But, in 
general an application or transport doesn't know the bottleneck rate of 
its path, and certainly not the available rate. So, I think the numbers 
in the draft are perhaps unconsciously related to what the authors 
consider to be the average bottleneck link rate at the time of writing. 
But this will not be a useful guide in 2031.

Current global average access bandwidth according to speedtest.net's 
global index https://www.speedtest.net/global-index are:
     fixed    down 107Mb/s up 58Mb/s
     mobile down 55Mb/s up 13Mb/s
[To anyone tempted to dispute that: Please let's not debate how 
selective/biased/correct those numbers are. I'm just looking for a 
ball-park]

So 1Mb/s consumes about 2-10% of each RTT for the lowest of these 
averages (mobile upstream). Yes, there is wide variation around those 
averages. But that's why the figure is in the single digit percentages.

So perhaps "an instantaneous packet rate that is no more than a single 
digit percentage of the global average access link capacity at the 
time," Then give an example for the time of writing.

Equivalently, "the packet serialization times consume no more than a 
single digit percentage of each RTT". But I think that's unnecessarily 
obscure.

Cheers

Bob