Subject: [tsvwg] NQB PHB description and assumptions

From: <[email-masked]>
Date: Fri, 6 Sep 2019 07:12:16 +0000

Hi Bob,

the following excerpt of your reply to David and Sebastian contains text for which I couldn’t identify a clear reference in the NQB draft. You ask the authors of that draft to add a description of the scheduling. I agree with you and like to expand that to a description of the PHB so that a reader has an idea about how to implement or configure a NQB PHB beside the default PHB.

What the NQB draft doesn’t mention, but you do:

  *   You state NQB doesn’t require priority queuing (the NQB draft suggests to operate EF and NQB via the same queue with DSCPs 101110 for EF and 101010 for NQB, which to me means, it’s traffic destined to a priority queue).
  *   You state that NQB is enabled largely by high bandwidth Internet accesses. The NQB draft doesn’t.
  *   You are quite specific with one possible requirement, which in general would be that a maximum length PDU should be serialized by a link suitable for NQB within less than 1 ms. The NQB draft doesn’t say so.
  *   Further, your example suggests a 3ms@accessbandwidth buffer depth for the NQB Queue. The draft is not specific here, and it likely doesn’t have to be that detailed. In my eyes, the draft NQB PHB description is vague. I’m clueless how to implement or configure an NQB queue, if I read the draft.

I’d like to add another point: if, in the example below, the NQB traffic stays below 50% link utilization also in the case of access link congestion, it shouldn’t see a queue.

Regards,

Ruediger
===Misunderstanding?==
[snip]

Please confirm that you (both) understand that, with NQB, the low latency is not provided by scheduling in the network. It is primarily a property of NQB traffic sources, and the only necessary property of the network's per-hop behaviour for NQB is to isolate NQB from QB traffic (i.e. in a separate queue for the NQB class). There is no priority scheduling requirement.

[A comment I would make to the authors about the draft: it needs to give some example approaches to scheduling between NQB and QB, or at least talk about this scheduling.]
==Incentive Alignment?==
Background: NQB has become possible largely because Internet access link rates have typically become fast enough that the serialization delay of a packet can be sub-millisecond, and therefore a queue of a few packets introduces delay that is small relative to other non-optional sources of delay like propagation. In these cases we no longer need priority scheduling for low delay.

Config:

  *   Scheduler:
     *   WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least 60Mb/s for NQB flows.
     *   Scheduler quantum: 1500B.
  *   Buffering:
     *   The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
     *   The QB buffer is deeper (say 200ms) with an AQM target delay of say 10ms.

---

From: Greg White <[email-masked]>
Date: Fri, 6 Sep 2019 19:56:31 +0000

Hi Ruediger,

These are good suggestions for improvement of the draft.

Thanks,
Greg

From: tsvwg <[email-masked]> on behalf of "[email-masked]" <[email-masked]>
Date: Friday, September 6, 2019 at 1:13 AM
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: [tsvwg] NQB PHB description and assumptions

Hi Bob,

-          You state NQB doesn’t require priority queuing (the NQB draft suggests to operate EF and NQB via the same queue with DSCPs 101110 for EF and 101010 for NQB, which to me means, it’s traffic destined to a priority queue).

-          You state that NQB is enabled largely by high bandwidth Internet accesses. The NQB draft doesn’t.

-          You are quite specific with one possible requirement, which in general would be that a maximum length PDU should be serialized by a link suitable for NQB within less than 1 ms. The NQB draft doesn’t say so.

-          Further, your example suggests a 3ms@accessbandwidth buffer depth for the NQB Queue. The draft is not specific here, and it likely doesn’t have to be that detailed. In my eyes, the draft NQB PHB description is vague. I’m clueless how to implement or configure an NQB queue, if I read the draft.

Regards,

Ruediger
===Misunderstanding?==
[snip]

Config:
·         Scheduler:
o    WRR with weight 0.5 for NQB on a 120Mb/s link. That gives at least 60Mb/s for NQB flows.
o    Scheduler quantum: 1500B.
·         Buffering:
o    The NQB buffer is fairly shallow (30 packets or 3ms at 120Mb/s).
o    The QB buffer is deeper (say 200ms) with an AQM target delay of say 10ms.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 9 Sep 2019 14:39:41 +0100

Ruediger,

Altho Greg's taken your suggestions on board, there's some need for me 
to explain those that weren't in the draft... inline...

On 06/09/2019 08:12, [email-masked] wrote:
>
> Hi Bob,
>
> the following excerpt of your reply to David and Sebastian contains 
> text for which I couldn’t identify a clear reference in the NQB draft. 
> You ask the authors of that draft to add a description of the 
> scheduling. I agree with you and like to expand that to a description 
> of the PHB so that a reader has an idea about how to implement or 
> configure a NQB PHB beside the default PHB.
>
>   * You state NQB doesn’t require priority queuing (the NQB draft
>     suggests to operate EF and NQB via the same queue with DSCPs
>     101110 for EF and 101010 for NQB, which to me means, it’s traffic
>     destined to a priority queue).
>
You're assuming that NQB is classified into a queue designed the way EF 
has been specified in the RFC. In the solution for low latency DOCSIS, a 
queue is designed for sources that behave in an NQB way. And, altho the 
queue does not comply with the RFC defining EF, packets tagged EF are 
also classified into it, because their sources satisfy the NQB behaviour 
and they can take advantage of the low delay.

>     accesses. The NQB draft doesn’t.
I didn't intend hi b/w to be a requirement for NQB.

That was merely me saying that, if you have hi bandwidth, NQB could be 
designed like I proposed in my straw man. There might be other NQB 
designs for low b/w (obviously more difficult tho).

I.e. the more b/w you have, the less you're tied to 'old-school' QoS 
designs, where low delay depends on bandwidth priority.

>   * You are quite specific with one possible requirement, which in
>     general would be that a maximum length PDU should be serialized by
>     a link suitable for NQB within less than 1 ms. The NQB draft
>     doesn’t say so.
>
I've checked and I can't find where you think I said that. If I implied 
it, I didn't mean to. The closest was when I said this...

On 05/09/2019 19:23, Bob Briscoe wrote:
>> _Background_: NQB has become possible largely because Internet access 
>> link rates have typically become fast enough that the serialization 
>> delay of a packet can be sub-millisecond, and therefore a queue of a 
>> few packets introduces delay that is small relative to other 
>> non-optional sources of delay like propagation. In these cases we no 
>> longer need priority scheduling for low delay._
>> _

I only said sub-millisecond 'cos it's about an order or magnitude lower 
than typical propagation delays over the public Internet, and therefore 
tends towards an insignificant contribution to overall delay.

In the context of a data centre with 1ms RTT diameter, that sentence 
would translate to 'link rates ... fast enough that serialization delay 
.. can be sub-100us'.

>   * Further, your example suggests a 3ms@accessbandwidth buffer depth
>     for the NQB Queue. The draft is not specific here, and it likely
>     doesn’t have to be that detailed.
>
I gave a strawman example so I could argue with David & Sebastian in a 
concrete way. No implication that the draft should be as concrete, or 
even that it should use the example.

>   * In my eyes, the draft NQB PHB description is vague. I’m clueless
>     how to implement or configure an NQB queue, if I read the draft.
>
> I’d like to add another point: if, in the example below, the NQB 
> traffic stays below 50% link utilization also in the case of access 
> link congestion, it shouldn’t see a queue.
>
Er no... Consider bursts of ten back-to-back 1500B packets sent on 
average every 30 ms from a 1Gb/s NIC into a 100Mb/s link. That's an 
average of only 4% utilization (4Mb/s), but at the end of each burst the 
queue is 9 packets.

Even if each NQB flow perfectly paces its own packets, the size of queue 
will depend on the number of uncoordinated flows. That's because the 
number of flows determines how many packets could arrive at about the 
same time and cause a queue. My calculation gave the worst case queue 
where a packet from all n flows happens to arrive at the same time. 
Statistically unlikely, but I explained I was calculating the tail of 
the delay distribution.

Bob

> Regards,
>
> Ruediger
>
>
>       ===Misunderstanding?==
>
> [snip]
>
> Please confirm that you (both) understand that, with NQB, the low 
> latency is not provided by scheduling in the network. It is primarily 
> a property of NQB traffic sources, and the only necessary property of 
> the network's per-hop behaviour for NQB is to isolate NQB from QB 
> traffic (i.e. in a separate queue for the NQB class). There is no 
> priority scheduling requirement.
>
> [A comment I would make to the authors about the draft: it needs to 
> give some example approaches to scheduling between NQB and QB, or at 
> least talk about this scheduling.]
>
>
>       ==Incentive Alignment?==
>
> _Background_: NQB has become possible largely because Internet access 
> link rates have typically become fast enough that the serialization 
> delay of a packet can be sub-millisecond, and therefore a queue of a 
> few packets introduces delay that is small relative to other 
> non-optional sources of delay like propagation. In these cases we no 
> longer need priority scheduling for low delay._
>
> Config_:
>
>   * Scheduler:

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: <[email-masked]>
Date: Tue, 10 Sep 2019 10:00:50 +0000

Hi Bob,

a PHB usually isn’t defined to the detail of implementation. AF and EF allow for a coarse understanding how to implement and configure the required functions. For me, NQB isn’t at that level yet.

I remove some of the prior text and reply [RG], marking your statements [BB]

#######################

  *   You state that NQB is enabled largely by high bandwidth Internet accesses. The NQB draft doesn’t.
     [BB] I didn't intend hi b/w to be a requirement for NQB.

     [BB] That was merely me saying that, if you have hi bandwidth, NQB could be designed like I proposed in my straw man. There might be other NQB designs for low b/w (obviously more difficult tho).

      [BB] I.e. the more b/w you have, the less you're tied to 'old-school' QoS designs, where low delay depends on bandwidth priority.

[RG] I’d appreciate, if the draft clarifies this point. Is NQB recommended for high bandwidth links? If yes, what’s meant by “high speed”? When is “old-school” QoS recommended and how will NQB work there? Some rough guidance should be given.

#################################

   [BB] I only said sub-millisecond 'cos it's about an order or magnitude lower than typical propagation delays over the public Internet, and therefore tends towards an insignificant contribution to overall delay.

    [BB] In the context of a data centre with 1ms RTT diameter, that sentence would translate to 'link rates ... fast enough that serialization delay .. can be sub-100us'.

[RG] I’d appreciate text telling a reader, what NQB is about and how it works. In that sense, picking up and mixing your statements, is it something like that:

The bitrate configured for NQB traffic should ensure that a queue of a few MTU sized packets introduces a delay causing only an insignificant contribution to the overall end-to-end delay.

######################################

  *   Further, your example suggests a 3ms@accessbandwidth buffer depth for the NQB Queue. ..[snip]

I’d like to add another point: if, in the example below, the NQB traffic stays below 50% link utilization also in the case of access link congestion, it shouldn’t see a queue.
    [BB] Er no... Consider bursts of ten back-to-back 1500B packets sent on average every 30 ms from a 1Gb/s NIC into a 100Mb/s link. That's an average of only 4% utilization (4Mb/s), but at the end of each burst the queue is 9 packets.

[RG] Your explanation is correct. Unfortunately, it doesn’t help me. Do you want to say, the example flow is NQB compliant? Then this shouldn’t be an issue (it adds of 2,4 ms delay in the worst case)? Or this is an issue? But what happens then and what’s the impact on the NQB traffic? None? I’m lost. Your example proposed a configuration and I take the available NQB text giving some examples for NQB traffic by the letter. What’s wrong here?

      [BB] Even if each NQB flow perfectly paces its own packets, the size of queue will depend on the number of uncoordinated flows. That's because the number of flows determines how many packets could arrive at about the same time and cause a queue. My calculation gave the worst case queue where a packet from all n flows happens to arrive at the same time. Statistically unlikely, but I explained I was calculating the tail of the delay distribution.

[RG] Yes. That’s why I’d expect the draft to be more specific on the traffic properties expected by NQB compliant traffic. Current text is

        There are many applications that send traffic at relatively low data
        rates and/or in a fairly smooth and consistent manner such that they
        are highly unlikely to exceed the available capacity of the network
        path between source and sink.
        These Non-queue-building (NQB) flows ..typically ….
        send traffic at a lower data rate and don't seek the capacity of the
         link (examples: online games, voice chat, DNS lookups).  Here the
        data rate is essentially limited by the Application itself.

[RG] What’s a “lower data rate”? What are the multiplexing conditions to avoid a queue build up at a link operating NQB (to quote you:… because the number of flows determines how many packets could arrive at about the same time and cause a queue)? How does an application developer and a service provider decide, which traffic is NQB? Which congestion indication is required by NQB and is an AQM expected? If yes, what is the AQM supposed to do an where’s that specified?

[RG] If the sole signaling of congestion is to forward flows identified as causing congestion by a QB PHB, the draft should state it that way. If the bottleneck is congested and the NQB flows consume more than (say) 80% of the bandwidth set aside for the NQB PHB, all flows could be perfectly well behaving and still a queue starts to build up. How does NQB work then and how to avoid these situations or return to desired NQB operational conditions for all NQB marked flows (please note, I don’t assume misuse and per definition admission control is absent)?

Regards,

Ruediger

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 10 Sep 2019 15:29:29 +0100

Ruediger,

I'm not a co-author. But I did work alongside Greg while he and Thomas 
were developing the idea. So I hope my interventions are helping, but 
they might be adding confusion.

I will start out by saying that people who think they understand QoS 
will not understand NQB unless they wipe most of their 'old-school QoS' 
preconceptions.

Responses inline tagged [BB2]...

On 10/09/2019 11:00, [email-masked] wrote:
>
> Hi Bob,
>
> a PHB usually isn’t defined to the detail of implementation. AF and EF 
> allow for a coarse understanding how to implement and configure the 
> required functions. For me, NQB isn’t at that level yet.
>
[BB2] NQB is primarily a property of a single traffic flow.

I believe the description of an NQB PHB would be pretty much null, which 
is perhaps why you think it isn't sufficient yet. As I said, the primary 
property of the PHB is merely isolation, i.e. a separate queue for NQB 
traffic. The sentence in the draft describing the 'useful property' of 
an NQB PHB (that you're worse off mismarking) is really the only 
additional specification it needs.

I asked the authors for examples of how an NQB queue might be scheduled, 
but they would be examples, not definitions of the PHB.

>       * You state that NQB is enabled largely by high bandwidth
>         Internet accesses. The NQB draft doesn’t.
>
>      [BB] That was merely me saying that, if you have hi bandwidth, 
> NQB could be designed like I proposed in my straw man. There might be 
> other NQB designs for low b/w (obviously more difficult tho).
>
>       [BB] I.e. the more b/w you have, the less you're tied to 
> 'old-school' QoS designs, where low delay depends on bandwidth priority.
>
> [RG] I’d appreciate, if the draft clarifies this point. Is NQB 
> recommended for high bandwidth links? If yes, what’s meant by “high 
> speed”? When is “old-school” QoS recommended and how will NQB work 
> there? Some rough guidance should be given.
>
>         _Background_: NQB has become possible largely because Internet
>         access link rates have typically become fast enough that the
>         serialization delay of a packet can be sub-millisecond, and
>         therefore a queue of a few packets introduces delay that is
>         small relative to other non-optional sources of delay like
>         propagation. In these cases we no longer need priority
>         scheduling for low delay.
>
>
>    [BB] I only said sub-millisecond 'cos it's about an order or 
> magnitude lower than typical propagation delays over the public 
> Internet, and therefore tends towards an insignificant contribution to 
> overall delay.
>
>     [BB] In the context of a data centre with 1ms RTT diameter, that 
> sentence would translate to 'link rates ... fast enough that 
> serialization delay .. can be sub-100us'.
>
> [RG] I’d appreciate text telling a reader, what NQB is about and how 
> it works. In that sense, picking up and mixing your statements, is it 
> something like that:
>
> The bitrate configured for NQB traffic should ensure that a queue of a 
> few MTU sized packets introduces a delay causing only an insignificant 
> contribution to the overall end-to-end delay.
>
[BB2] Again, I used link rate as a condition for the straw man NQB 
implementation I described. I do not believe link rate is (or should be) 
a condition in any general definition of NQB.
>     traffic stays below 50% link utilization also in the case of
>     access link congestion, it shouldn’t see a queue.
>
>     [BB] Er no... Consider bursts of ten back-to-back 1500B packets 
> sent on average every 30 ms from a 1Gb/s NIC into a 100Mb/s link. 
> That's an average of only 4% utilization (4Mb/s), but at the end of 
> each burst the queue is 9 packets.
>
> [RG] Your explanation is correct. Unfortunately, it doesn’t help me. 
> Do you want to say, the example flow is NQB compliant? Then this 
> shouldn’t be an issue (it adds of 2,4 ms delay in the worst case)? Or 
> this is an issue? But what happens then and what’s the impact on the 
> NQB traffic? None? I’m lost. Your example proposed a configuration and 
> I take the available NQB text giving some examples for NQB traffic by 
> the letter. What’s wrong here?
>
>       [BB] Even if each NQB flow perfectly paces its own packets, the 
> size of queue will depend on the number of uncoordinated flows. That's 
> because the number of flows determines how many packets could arrive 
> at about the same time and cause a queue. My calculation gave the 
> worst case queue where a packet from all n flows happens to arrive at 
> the same time. Statistically unlikely, but I explained I was 
> calculating the tail of the delay distribution.
>
> [RG] Yes. That’s why I’d expect the draft to be more specific on the 
> traffic properties expected by NQB compliant traffic. Current text is
>
> [RG] What’s a “lower data rate”? What are the multiplexing conditions 
> to avoid a queue build up at a link operating NQB (to quote you:… 
> because the number of flows determines how many packets could arrive 
> at about the same time and cause a queue)? How does an application 
> developer and a service provider decide, which traffic is NQB? Which 
> congestion indication is required by NQB and is an AQM expected? If 
> yes, what is the AQM supposed to do an where’s that specified?
>
[BB2] Wipe your preconceptions. I don't think NQB is as precise as you 
want it to be. My calculations were precise for the traffic scenario I 
assumed, but the service cannot be defined precisely, because the 
behaviour aggregate could contain an arbitrary number of flows (even 
though there will be a likely number).

It's more like: "NQB is targeted at access links, where flow 
multiplexing will probably be low enough most of the time that it will 
be worthwhile isolating the aggregate of all NQB flows from any QB 
flows, such that queuing delay for all of the NQB flows together, 
without any additional controls."

The main point is that, even if "old-school" QoS folks are tempted to 
over-engineer this, the isolation of the NQB behaviour aggregate gives 
most of the benefit. So let's try to keep it to isolation alone.

> [RG] If the sole signaling of congestion is to forward flows 
> identified as causing congestion by a QB PHB, the draft should state 
> it that way. If the bottleneck is congested and the NQB flows consume 
> more than (say) 80% of the bandwidth set aside for the NQB PHB, all 
> flows could be perfectly well behaving and still a queue starts to 
> build up. How does NQB work then and how to avoid these situations or 
> return to desired NQB operational conditions for all NQB marked flows 
> (please note, I don’t assume misuse and per definition admission 
> control is absent)?
>
[BB2] As above, yes the queue does build up as the amount of NQB traffic 
increases. But if one assumes all that NQB traffic is necessary, just 
isolating all the NQB from the QB still gives all the NQB traffic the 
lowest delay possible, given the link resource available.

The DOCSIS queue protection function keeps queuing delay below a 
threshold, by reclassifying packets from the least NQB flows to protect 
the most NQB flows. But you might not want that.

Cheers

Bob

---

From: <[email-masked]>
Date: Wed, 11 Sep 2019 09:29:41 +0000

Bob, Greg

if I got Bob right, the main feature of the NQB queue is a measurement functionality demoting “the least NQB like flow” to another queue (or drop it?) , once a queue build up is detected in the NQB queue.

I think this measurement function to determine “the least NQB like flow” requires some text in the draft. I’m interested in engineering my network. To do so, I need to understand, what goes on in the routers. I’m least interested in last level escalation due to a black box behavior which is under my engineering responsibility.

Other NQB features seem to be a bandwidth-share and a queue depth (be that statically in bytes or in ms with some dynamic behavior like PIE or Codel).

Further, the authors should add a description of the sender behavior (the draft states, they should not be queue building – I’d rather be interested in learning, what a typical traffic behavior of an NQB sender is, than reading, what they should not be doing).

If additional AQM features are recommended, please say so. Demotion to another queue (or drop) may seem a little hard without prior warning. If “the least NQB like flow” can be classified and demoted, further options likely exist.

If I missed an important feature, please let me know.

Regards, Ruediger

Von: Bob Briscoe <[email-masked]>
Gesendet: Dienstag, 10. September 2019 16:29
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]
Betreff: Re: AW: [tsvwg] NQB PHB description and assumptions

Ruediger,

I'm not a co-author. But I did work alongside Greg while he and Thomas were developing the idea. So I hope my interventions are helping, but they might be adding confusion.

I will start out by saying that people who think they understand QoS will not understand NQB unless they wipe most of their 'old-school QoS' preconceptions.

On 10/09/2019 11:00, [email-masked]<mailto:[email-masked]> wrote:
Hi Bob,

I believe the description of an NQB PHB would be pretty much null, which is perhaps why you think it isn't sufficient yet. As I said, the primary property of the PHB is merely isolation, i.e. a separate queue for NQB traffic. The sentence in the draft describing the 'useful property' of an NQB PHB (that you're worse off mismarking) is really the only additional specification it needs.

I asked the authors for examples of how an NQB queue might be scheduled, but they would be examples, not definitions of the PHB.

#######################

The bitrate configured for NQB traffic should ensure that a queue of a few MTU sized packets introduces a delay causing only an insignificant contribution to the overall end-to-end delay.
[BB2] Again, I used link rate as a condition for the straw man NQB implementation I described. I do not believe link rate is (or should be) a condition in any general definition of NQB.

[RG] What’s a “lower data rate”? What are the multiplexing conditions to avoid a queue build up at a link operating NQB (to quote you:… because the number of flows determines how many packets could arrive at about the same time and cause a queue)? How does an application developer and a service provider decide, which traffic is NQB? Which congestion indication is required by NQB and is an AQM expected? If yes, what is the AQM supposed to do an where’s that specified?
[BB2] Wipe your preconceptions. I don't think NQB is as precise as you want it to be. My calculations were precise for the traffic scenario I assumed, but the service cannot be defined precisely, because the behaviour aggregate could contain an arbitrary number of flows (even though there will be a likely number).

It's more like: "NQB is targeted at access links, where flow multiplexing will probably be low enough most of the time that it will be worthwhile isolating the aggregate of all NQB flows from any QB flows, such that queuing delay for all of the NQB flows together, without any additional controls."

The main point is that, even if "old-school" QoS folks are tempted to over-engineer this, the isolation of the NQB behaviour aggregate gives most of the benefit. So let's try to keep it to isolation alone.

[RG] If the sole signaling of congestion is to forward flows identified as causing congestion by a QB PHB, the draft should state it that way. If the bottleneck is congested and the NQB flows consume more than (say) 80% of the bandwidth set aside for the NQB PHB, all flows could be perfectly well behaving and still a queue starts to build up. How does NQB work then and how to avoid these situations or return to desired NQB operational conditions for all NQB marked flows (please note, I don’t assume misuse and per definition admission control is absent)?
[BB2] As above, yes the queue does build up as the amount of NQB traffic increases. But if one assumes all that NQB traffic is necessary, just isolating all the NQB from the QB still gives all the NQB traffic the lowest delay possible, given the link resource available.

The DOCSIS queue protection function keeps queuing delay below a threshold, by reclassifying packets from the least NQB flows to protect the most NQB flows. But you might not want that.

Cheers

Bob

Regards,

Ruediger

--