Subject: [tsvwg] https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03#page-10

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 Mar 2019 09:25:49 +0100

So the L4S architecture description has the following rationale, why L4S opted for the dual-queue approach instead of mandatinf flow queueing:

https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03#page-10:

 Per-flow queuing:  Similarly per-flow queuing is not incompatible
      with the L4S approach.  However, one queue for every flow can be
      thought of as overkill compared to the minimum of two queues for
      all traffic needed for the L4S approach.  The overkill of per-flow
      queuing has side-effects:

     A.  fq makes high performance networking equipment costly
          (processing and memory) - in contrast dual queue code can be
          very simple;

      B.  fq requires packet inspection into the end-to-end transport
          layer, which doesn't sit well alongside encryption for privacy
          - in contrast the use of ECN as the classifier for L4S
          requires no deeper inspection than the IP layer;

      C.  fq isolates the queuing of each flow from the others but not
          from itself so, unlike L4S, it does not support applications
          that need both capacity-seeking behaviour and very low
          latency.

          It might seem that self-inflicted queuing delay should not
          count, because if the delay wasn't in the network it would
          just shift to the sender.  However, modern adaptive
          applications, e.g.  HTTP/2 [RFC7540] or the interactive media
          applications described in Section 6, can keep low latency
          objects at the front of their local send queue by shuffling
          priorities of other objects dependent on the progress of other
          transfers.  They cannot shuffle packets once they have
          released them into the network.

      D.  fq prevents any one flow from consuming more than 1/N of the
          capacity at any instant, where N is the number of flows.  This
          is fine if all flows are elastic, but it does not sit well
          with a variable bit rate real-time multimedia flow, which
          requires wriggle room to sometimes take more and other times
          less than a 1/N share.

          It might seem that an fq scheduler offers the benefit that it
          prevents individual flows from hogging all the bandwidth.
          However, L4S has been deliberately designed so that policing
          of individual flows can be added as a policy choice, rather
          than requiring one specific policy choice as the mechanism
          itself.  A scheduler (like fq) has to decide packet-by-packet
          which flow to schedule without knowing application intent.
          Whereas a separate policing function can be configured less
          strictly, so that senders can still control the instantaneous
          rate of each flow dependent on the needs of each application
          (e.g. variable rate video), giving more wriggle-room before a
          flow is deemed non-compliant.  Also policing of queuing and of
          flow-rates can be applied independently.

And then cablelabs, the organization closest to deploying L4S, in Data-Over-Cable Service Interface Specifications DOCSIS® 3.1, MAC and Upper Layer Protocols Interface SpecificationCM-SP-MULPIv3.1-I17-190121, Annex P Queue Protection Algorithm (Normative) says:

"This annex defines the Queue Protection algorithm that is required to be supported by the CM in the upstream (see Section 7.7.6.1). It is also the Queue Protection algorithm that CMTS Queue Protection algorithms are required to support (see Section 7.7.6.2). In either direction, this algorithm is intended to be applied solely to a Low Latency Service Flow. It detects queue- building Microflows and redirects some or all of their packets to the Classic Service Flow in order to protect the Latency Service Flow from excessive queuing. A Microflow is defined in Section P.3, but typically it is an end-to- end transport layer data flow."

To me this looks like the introduction of flow-queuing through a backdoor, instead of doing it upfront this tries to measure whether flows behave well enough to keep enjoying the L4S special treatment and demotes non-compliant flows to the TCP-friendly queue. I would be intersted to learn how this mandatory requirement can be implemented without at least having similar cost like flow queueung (at least I fear it will drag the identifies issues A. and B. from above into the system running the L4S-compliant AQM). I would love to learn where my interpretation is wrong.

Many Thanks in advance

Regards
	Sebastian Moeller

---

From: G Fairhurst <[email-masked]>
Date: Thu, 28 Mar 2019 09:52:52 +0100

> To me this looks like the introduction of flow-queuing through a backdoor, instead of doing it upfront this tries to measure whether flows behave well enough to keep enjoying the L4S special treatm…
I'm not sure I agree this is necessarily a change of direction - I 
personally see queue protection as about a control plane optimisation to 
identify and mitigate traffic from non-conformant flows. Per-flow 
accounting is not the same as per-flow queueing. (From a different 
perspective: a circuit-breaker that monitors an envelope is not the same 
as a transport congestion controller or the same as a network traffic 
shaper/scheduler).

My expectation would be that someone implementing L4S doesn't need to do 
this? ... unless someone wants to offer this sort of protection. In the 
same way that not all current Ethernet switches protect from overload, 
but it is important that those people who care about these things can 
acquire this when they needs it.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 Mar 2019 09:59:48 +0100

Dear Gorry,

> On Mar 28, 2019, at 09:52, G Fairhurst <[email-masked]> wrote:
> 
> I'm not sure I agree this is necessarily a change of direction - I personally see queue protection as about a control plane optimisation to identify and mitigate traffic from non-conformant flows.

	Sure, but it is telling that the one organization that is in the process of rolling-out L4S makes this a _mandatory_ requirement, no?

> Per-flow accounting is not the same as per-flow queueing. (From a different perspective: a circuit-breaker that monitors an envelope is not the same as a transport congestion controller or the same as a network traffic shaper/scheduler).

	Sure, but if "Per-flow accounting" has similar computational cost " per-flow queueing" the rationale for not directly mandating  per-flow queueing seems somewhat diminished.

> 
> My expectation would be that someone implementing L4S doesn't need to do this?

	I agree, it is not in the L4S RFCs, so it is not a MUST from the RFCs perspective.

> ... unless someone wants to offer this sort of protection. In the same way that not all current Ethernet switches protect from overload, but it is important that those people who care about these things can acquire this when they needs it.

Fair enough, point taken.

Thanks
	Sebastian

---

From: G Fairhurst <[email-masked]>
Date: Thu, 28 Mar 2019 10:08:29 +0100

On 28/03/2019, 09:59, Sebastian Moeller wrote:
> Dear Gorry,
>
>> On Mar 28, 2019, at 09:52, G Fairhurst<[email-masked]>  wrote:
>>
I'll let those who worked on this spec comment upon this.
> 	Sure, but if "Per-flow accounting" has similar computational cost " per-flow queueing" the rationale for not directly mandating  per-flow queueing seems somewhat diminished.
Unpicking this and asking for help:

Is it really is the same "cost"?

And... do people see it as something that all routers along the path 
should do, or something that an edge router may do?
Gorry

---

From: <[email-masked]>
Date: Thu, 28 Mar 2019 11:10:58 +0000

Hi Gorry,

If I get you right, you are asking whether all routers along an end to end path should generally deploy per flow queuing or whether that's more suitable for edge routers. 

I don't recommend per IP flow queuing in MPLS backbone networks. I also don't recommend per top label or per label stack queuing in MPLS backbones.

Some routers support Virtual Output Queuing, but this more refers to an incoming to outgoing interface multipoint to point relation. 

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: tsvwg <[email-masked]> Im Auftrag von G Fairhurst
Gesendet: Donnerstag, 28. März 2019 10:08
An: Sebastian Moeller <[email-masked]>
Cc: tsvwg IETF list <[email-masked]>
Betreff: Re: [tsvwg] https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03#page-10

And... do people see it as something that all routers along the path should do, or something that an edge router may do?
Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 28 Mar 2019 13:10:17 +0100

Hi Ruediger,

> On Mar 28, 2019, at 12:10, <[email-masked]> <[email-masked]> wrote:
> 
> Hi Gorry,
> 

Do you recommend to deploy AQM in those routers at all? In other words would you bother to look into a MPLS encapsulated IP header to get access to the ECN bits in your MPLS cloud at all, and how about at the edges of the MPLS network?
I naively thought that such an AQM only is needed the edges (access links, and transit/peering points, basically the only places where congestion is likely enough to justify the cost of doing AQM in the first place).

Best Regards
	Sebastian

---

From: <[email-masked]>
Date: Thu, 28 Mar 2019 13:28:12 +0000

Hi Sebastian,

it is hard to engineer hardware around cross layer functionalities. So reading and setting IP ECN bits behind an MPLS label stack is no good idea, I think. Then all in all, there are 8 codepoints to address DiffServ and ECN in MPLS. I think, spending a single ECN codepoint may be feasible (there's an RFC specifying MPLS/IP ECN interoperartion). Getting 4 ECN codepoints for a single PHB in MPLS will not work.

I personally like the idea of signaling congestion by packet marks rather than by drops. To get MPLS support, one must be sure it is honored E2E. 

AQM can be deployed to manage queue drops. It's deployed in the sections of the backbone that I am aware of (flow there is referring to a Diffserv aggregate, not to anything address-related). Looking at what's offered by our vendors, I think WRED dropping (or the like) is a commodity in the backbone.

ECN should first be deployed, where congestion appears most frequent. That's the Internet access, I think.

Regards,

Ruediger

-----Ursprüngliche Nachricht-----
Von: Sebastian Moeller <[email-masked]> 
Gesendet: Donnerstag, 28. März 2019 13:10
An: Geib, Rüdiger <[email-masked]>
Cc: [email-masked]; [email-masked]
Betreff: Re: [tsvwg] https://tools.ietf.org/html/draft-ietf-tsvwg-l4s-arch-03#page-10

Hi Ruediger,

Best Regards
	Sebastian