Subject: Re: [tsvwg] [Ecn-sane] per-flow scheduling

From: Sebastian Moeller <[email-masked]>
Date: Wed, 17 Jul 2019 23:33:17 +0200

Dear Bob, dear IETF team,

> On Jun 19, 2019, at 16:12, Bob Briscoe <[email-masked]> wrote:
> 
> Jake, all,
> 
> You may not be aware of my long history of concern about how per-flow scheduling within endpoints and networks will limit the Internet in future. I find per-flow scheduling a violation of the e2e principle in such a profound way - the dynamic choice of the spacing between packets - that most people don't even associate it with the e2e principle.

	This does not rhyme well with the L4S stated advantage of allowing packet reordering (due to mandating RACK for all L4S tcp endpoints). Because surely changing the order of packets messes up the "the dynamic choice of the spacing between packets" in a significant way. IMHO it is either L4S is great because it will give intermediate hops more leeway to re-order packets, or "a sender's packet spacing" is sacred, please make up your mind which it is.

> 
> I detected that you were talking about FQ in a way that might have assumed my concern with it was just about implementation complexity. If you (or anyone watching) is not aware of the architectural concerns with per-flow scheduling, I can enumerate them.

	Please do not hesitate to do so after your deserved holiday, and please state a superior alternative.

Best Regards
	Sebastian

> 
> I originally started working on what became L4S to prove that it was possible to separate out reducing queuing delay from throughput scheduling. When Koen and I started working together on this, we discovered we had identical concerns on this.
> 
> 
> 
> Bob
> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> 
> _______________________________________________
> Ecn-sane mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/ecn-sane

---

From: "David P. Reed" <[email-masked]>
Date: Wed, 17 Jul 2019 18:18:37 -0400 (EDT)

I do want to toss in my personal observations about the "end-to-end argument" related to per-flow-scheduling. (Such arguments are, of course, a class of arguments to which my name is attached. Not that I am a judge/jury of such questions...)

A core principle of the Internet design is to move function out of the network, including routers and middleboxes, if those functions

a) can be properly accomplished by the endpoints, and 
b) are not relevant to all uses of the Internet transport fabric being used by the ends.

The rationale here has always seemed obvious to me. Like Bob Briscoe suggests, we were very wary of throwing features into the network that would preclude unanticipated future interoperability needs, new applications, and new technology in the infrastructure of the Internet as a whole. 

So what are we talking about here (ignoring the fine points of SCE, some of which I think are debatable - especially the focus on TCP alone, since much traffic will likely move away from TCP in the near future.

A second technical requirement (necessary invariant) of the Internet's transport is that the entire Internet depends on rigorously stopping queueing delay from building up anywhere except at the endpoints, where the ends can manage it.This is absolutely critical, though it is peculiar in that many engineers, especially those who work at the IP layer and below, have a mental model of routing as essentially being about building up queueing delay (in order to manage priority in some trivial way by building up the queue on purpose, apparently).

This second technical requirement cannot be resolved merely by the endpoints.
The reason is that the endpoints cannot know accurately what host-host paths share common queues.

This lack of a way to "cooperate" among independent users of a queue cannot be solved by a purely end-to-end solution. (well, I suppose some genius might invent a way, but I have not seen one in my 36 years closely watching the Internet in operation since it went live in 1983.)

So, what the end-to-end argument would tend to do here, in my opinion, is to provide the most minimal mechanism in the devices that are capable of building up a queue in order to allow all the ends sharing that queue to do their job - which is to stop filling up the queue!

Only the endpoints can prevent filling up queues. And depending on the protocol, they may need to make very different, yet compatible choices.

This is a question of design at the architectural level. And the future matters.

So there is an end-to-end argument to be made here, but it is a subtle one.

The basic mechanism for controlling queue depth has been, and remains, quite simple: dropping packets. This has two impacts: 1) immediately reducing queueing delay, and 2) signalling to endpoints that are paying attention that they have contributed to an overfull queue.

The optimum queueing delay in a steady state would always be one packet or less. Kleinrock has shown this in the last few years. Of course there aren't steady states. But we don't want a mechanism that can't converge to that steady state *quickly*, for all queues in the network.

Another issue is that endpoints are not aware of the fact that packets can take multiple paths to any destination. In the future, alternate path choices can be made by routers (when we get smarter routing algorithms based on traffic engineering).

So again, some minimal kind of information must be exposed to endpoints that will continue to communicate. Again, the routers must be able to help a wide variety of endpoints with different use cases to decide how to move queue buildup out of the network itself.

Now the decision made by the endpoints must be made in the context of information about fairness. Maybe this is what is not obvious.

The most obvious notion of fairness is equal shares among source host, dest host pairs. There are drawbacks to that, but the benefit of it is that it affects the IP layer alone, and deals with lots of boundary cases like the case where a single host opens a zillion TCP connections or uses lots of UDP source ports or destinations to somehow "cheat" by appearing to have "lots of flows".

Another way to deal with dividing up flows is to ignore higher level protocol information entirely, and put the flow idenfitication in the IP layer. A 32-bit or 64-bit random number could be added as an "option" to IP to somehow extend the flow space.

But that is not the most important thing today.

I write this to say:
1) some kind of per-flow queueing, during the transient state where a queue is overloaded before packets are dropped would provide much needed information to the ends of every flow sharing a common queue.
2) per-flow queueing, minimized to a very low level, using IP envelope address information (plus maybe UDP and TCP addresses for those protocols in an extended address-based flow definition) is totally compatible with end-to-end arguments, but ONLY if the decisions made are certain to drive queueing delay out of the router to the endpoints.

On Wednesday, July 17, 2019 5:33pm, "Sebastian Moeller" <[email-masked]> said:

>> You may not be aware of my long history of concern about how per-flow scheduling
>> within endpoints and networks will limit the Internet in future. I find per-flow
>> scheduling a violation of the e2e principle in such a profound way - the dynamic
>> choice of the spacing between packets - that most people don't even associate it
>> with the e2e principle.
> 
> 	This does not rhyme well with the L4S stated advantage of allowing packet
> reordering (due to mandating RACK for all L4S tcp endpoints). Because surely
> changing the order of packets messes up the "the dynamic choice of the spacing
> between packets" in a significant way. IMHO it is either L4S is great because it
> will give intermediate hops more leeway to re-order packets, or "a sender's
> packet spacing" is sacred, please make up your mind which it is.
> 
>>
>> I detected that you were talking about FQ in a way that might have assumed my
>> concern with it was just about implementation complexity. If you (or anyone
>> watching) is not aware of the architectural concerns with per-flow scheduling, I
>> can enumerate them.
> 
>> I originally started working on what became L4S to prove that it was possible to
>> separate out reducing queuing delay from throughput scheduling. When Koen and I
>> started working together on this, we discovered we had identical concerns on
>> this.
>>
>>
>>
>> Bob
>>
>>
>> --

---

From: "David P. Reed" <[email-masked]>
Date: Wed, 17 Jul 2019 18:34:15 -0400 (EDT)

A follow up point that I think needs to be made is one more end-to-end argument:

It is NOT the job of the IP transport layer to provide free storage for low priority packets. The end-to-end argument here says: the ends can and must hold packets until they are either delivered or not relevant (in RTP, they become irrelevant when they get older than their desired delivery time, if you want an example of the latter), SO, the network should not provide the function of storage beyond the minimum needed to deal with transients.

That means, unfortunately, that the dream of some kind of "background" path that stores "low priority" packets in the network fails the end-to-end argument test.

If you think about this, it even applies to some imaginary interplanetary IP layer network. Queueing delay is not a feature of any end-to-end requirement.

What may be desired at the router/link level in an interplanetary IP layer is holding packets because a link is actually down, or using link-level error correction coding or retransmission to bring the error rate down to an acceptable level before declaring it down. But that's quite different - it's the link level protocol, which aims to deliver minimum queueing delay under tough conditions, without buffering more than needed for that (the number of bits that fit in the light-speed transmission at the transmission rate.

So, the main reason I'm saying this is because again, there are those who want to implement the TCP function of reliable delivery of each packet in the links. That's a very bad idea.

On Wednesday, July 17, 2019 6:18pm, "David P. Reed" <[email-masked]> said:

> I do want to toss in my personal observations about the "end-to-end argument"
> related to per-flow-scheduling. (Such arguments are, of course, a class of
> arguments to which my name is attached. Not that I am a judge/jury of such
> questions...)
> 
> A core principle of the Internet design is to move function out of the network,
> including routers and middleboxes, if those functions
> 
> The rationale here has always seemed obvious to me. Like Bob Briscoe suggests, we
> were very wary of throwing features into the network that would preclude
> unanticipated future interoperability needs, new applications, and new technology
> in the infrastructure of the Internet as a whole.
> 
> So what are we talking about here (ignoring the fine points of SCE, some of which
> I think are debatable - especially the focus on TCP alone, since much traffic will
> likely move away from TCP in the near future.
> 
> A second technical requirement (necessary invariant) of the Internet's transport
> is that the entire Internet depends on rigorously stopping queueing delay from
> building up anywhere except at the endpoints, where the ends can manage it.This is
> absolutely critical, though it is peculiar in that many engineers, especially
> those who work at the IP layer and below, have a mental model of routing as
> essentially being about building up queueing delay (in order to manage priority in
> some trivial way by building up the queue on purpose, apparently).
> 
> This lack of a way to "cooperate" among independent users of a queue cannot be
> solved by a purely end-to-end solution. (well, I suppose some genius might invent
> a way, but I have not seen one in my 36 years closely watching the Internet in
> operation since it went live in 1983.)
> 
> So, what the end-to-end argument would tend to do here, in my opinion, is to
> provide the most minimal mechanism in the devices that are capable of building up
> a queue in order to allow all the ends sharing that queue to do their job - which
> is to stop filling up the queue!
> 
> Only the endpoints can prevent filling up queues. And depending on the protocol,
> they may need to make very different, yet compatible choices.
> 
> The basic mechanism for controlling queue depth has been, and remains, quite
> simple: dropping packets. This has two impacts: 1) immediately reducing queueing
> delay, and 2) signalling to endpoints that are paying attention that they have
> contributed to an overfull queue.
> 
> The optimum queueing delay in a steady state would always be one packet or less.
> Kleinrock has shown this in the last few years. Of course there aren't steady
> states. But we don't want a mechanism that can't converge to that steady state
> *quickly*, for all queues in the network.
> 
> Another issue is that endpoints are not aware of the fact that packets can take
> multiple paths to any destination. In the future, alternate path choices can be
> made by routers (when we get smarter routing algorithms based on traffic
> engineering).
> 
> So again, some minimal kind of information must be exposed to endpoints that will
> continue to communicate. Again, the routers must be able to help a wide variety of
> endpoints with different use cases to decide how to move queue buildup out of the
> network itself.
> 
> Now the decision made by the endpoints must be made in the context of information
> about fairness. Maybe this is what is not obvious.
> 
> The most obvious notion of fairness is equal shares among source host, dest host
> pairs. There are drawbacks to that, but the benefit of it is that it affects the
> IP layer alone, and deals with lots of boundary cases like the case where a single
> host opens a zillion TCP connections or uses lots of UDP source ports or
> destinations to somehow "cheat" by appearing to have "lots of flows".
> 
> Another way to deal with dividing up flows is to ignore higher level protocol
> information entirely, and put the flow idenfitication in the IP layer. A 32-bit or
> 64-bit random number could be added as an "option" to IP to somehow extend the
> flow space.
> 
> 1) some kind of per-flow queueing, during the transient state where a queue is
> overloaded before packets are dropped would provide much needed information to the
> ends of every flow sharing a common queue.
> 2) per-flow queueing, minimized to a very low level, using IP envelope address
> information (plus maybe UDP and TCP addresses for those protocols in an extended
> address-based flow definition) is totally compatible with end-to-end arguments,
> but ONLY if the decisions made are certain to drive queueing delay out of the
> router to the endpoints.
> 
> 
> 
>

---

From: Dave Taht <[email-masked]>
Date: Wed, 17 Jul 2019 16:23:26 -0700

> That means, unfortunately, that the dream of some kind of "background" path that stores "low priority" packets in the network fails the end-to-end argument test.

I do not mind reserving a tiny portion of the network for "background"
traffic. This
is different (I think?) than storing low priority packets in the
network. A background
traffic "queue" of 1 packet would be fine....

> What may be desired at the router/link level in an interplanetary IP layer is holding packets because a link is actually down, or using link-level error correction coding or retransmission to bring…

As I outlined in my mit wifi talk - 1 layer of retry of at the wifi
mac layer made it
work, in 1998, and that seemed a very acceptable compromise at the
time. Present day
retries at the layer, not congestion controlled, is totally out of hand.

In thinking about starlink's mac, and mobility, I gradulally came to
the conclusion that
1 retry from satellites 550km up (3.6ms rtt) was needed, as much as I
disliked the idea.

I still dislike retries at layer 2, even for nearby sats. really
complicates things. so for all I know I'll be advocating ripping 'em
out in starlink, if they are indeed, in there, next week.

> So, the main reason I'm saying this is because again, there are those who want to implement the TCP function of reliable delivery of each packet in the links. That's a very bad idea.

It was tried in the arpanet, and didn't work well there. There's a
good story about many
of the flaws of the Arpanet's design, including that problem, in the
latter half of Kleinrock's second book on queue theory, at least the
first edition...

Wifi (and 345g) re-introduced the same problem with retransmits and
block acks at layer 2.

and after dissecting my ecn battlemesh data and observing what the
retries at the mac layer STILL do on wifi with the current default
wifi codel target (20ms AFTER two txops are in the hardware) currently
achieve (50ms, which is 10x worse than what we could do and still
better performance under load than any other shipping physical layer
we have with fifos)... and after thinking hard about nagle's thought
that "every application has a right to one packet in the network", and
this very long thread reworking the end to end argument in a similar,
but not quite identical direction, I'm coming to a couple conclusions
I'd possibly not quite expressed well before.

1) transports should treat an RFC3168 CE coupled with loss (drop and
mark) as an even stronger signal of congestion than either, and that
this bit of the codel algorithm,
when ecn is in use, is wrong, and has always been wrong:

https://github.com/dtaht/fq_codel_fast/blob/master/codel_impl.h#L178

(we added this arbitrarily to codel in the 5th day of development in
2012. Using FQ masked it's effects on light traffic)

What it should do instead is peek the queue and drop until it hits a
markable packet, at the very least.

Pie has an arbitrary drop at 10% figure, which does lighten the load
some... cake used to have drop and mark also until a year or two
back...

2) At low rates and high contention, we really need pacing and fractional cwnd.

(while I would very much like to see a dynamic reduction of MSS tried,
that too has a bottom limit)

even then, drop as per bullet 1.

3) In the end, I could see a world with SCE marks, and CE being
obsoleted in favor of drop, or CE only being exerted on really light
loads similar to (or less than!) what the arbitrary 10% figure for pie
uses

4) in all cases, I vastly prefer somehow ultimately shifting greedy
transports to RTT rather than drop or CE as their primary congestion
control indicator. FQ makes that feasible today. With enough FQ
deployed for enough congestive scenarios and hardware, and RTT
becoming the core indicator for more transports, single queued designs
become possible in the distant future.

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Dave Taht <[email-masked]>
Date: Wed, 17 Jul 2019 17:20:59 -0700

> markable packet, at the very least.

I didn't say this well. It should drop otherwise markable packets until it
exits the loop, and then mark the one it delivers from that flow, if it delivers
one from that flow. That gets rid of all the extra mass ecn creates...

but I should go code it up again and see what happens on wifi. Worst case I
prove yet again, that reasoning about the behavior of queues if futile.

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 18 Jul 2019 07:31:02 +0300

> On 18 Jul, 2019, at 1:18 am, David P. Reed <[email-masked]> wrote:
> 

As a point of order, SCE is not specific to TCP.  TCP is merely the most convenient congestion-aware protocol to experiment with, and therefore the one we have adapted first.  Other protocols which already are (or aspire to be) TCP friendly, especially QUIC, should also be straightforward to adapt to SCE.

I should also note that TCP is the de-facto gold standard, by which all other congestion control is measured, for better or worse.  SCE is included in this, insofar as competing reasonably with standard TCP flows under all reasonable network conditions is necessary to introduce a new congestion control paradigm.  This, I think, is also part of the end-to-end principle.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 18 Jul 2019 08:24:18 +0300

> 2) per-flow queueing, minimized to a very low level, using IP envelope address information (plus maybe UDP and TCP addresses for those protocols in an extended address-based flow definition) is tot…

These are points that I can agree with quite easily, and which are reflected in my work to date.  Although I don't usually quote Kleinrock by name, the principle of always permitting one MTU per flow in the queue is fairly obvious.

One of the things that per-flow queuing provides to endpoints is differential treatment in AQM.  This is even true of LFQ, although the mechanism may not be obvious to all since there is only one set of AQM state; at minimum, AQM signals are suppressed for sparse flows and thus only provided to saturating flows.  SCE marking would be based on individual packets' sojourn times, which are logically independent from their physical position in the queue.  Careful implementation of a Codel-type AQM would also suppress CE marks (or drops) from well-behaved flows whose sojourn times are below the target, even if unresponsive flows are also present in the same queue, without losing accumulated state about the latter; this is I think already a property of COBALT (as implemented in Cake).  Other AQMs which convert a sojourn time more-or-less directly into a marking rate would also be a good fit.

I would only quibble that providing per-L4-flow fairness *within* a per-host or per-subscriber fairness structure is also feasible, in at least some contexts; Cake implements that, for example.  I hope to be able to amplify the LFQ draft to show how to provide that in a more lightweight manner than Cake manages, on my way to Montreal.

I may also publish CNQ (Cheap Nasty Queuing) as a straw-man draft at the same time, depending on my mood.  It should be good for light relief if nothing else.  It's even lighter-weight than LFQ - but, unlike LFQ, achieves this at the expense of performance.  It maintains only enough state to prioritise sparse flows, with a rather strict definition of "sparse".

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 18 Jul 2019 08:30:28 +0300

> On 18 Jul, 2019, at 3:20 am, Dave Taht <[email-masked]> wrote:
> 

You know, I think I finally understand what you're talking about here.  You want to treat cases where the marking rate exceeds the flow's packet delivery rate as an overload condition justifying a shift to packet drops.

This actually seems like a sane idea.  There is I think one caveat: selecting the Codel 'interval' parameter will now have an increased penalty for getting it wrong, especially on the too-small side.

 - Jonathan Morton

---

From: "David P. Reed" <[email-masked]>
Date: Thu, 18 Jul 2019 11:02:12 -0400 (EDT)

Dave -
The context of my remarks was about the end-to-end arguments for placing function in the Internet.

To that end, that "you do not mind putting storage for low priority packets in the routers" doesn't matter, for two important reasons:

1) the idea that one should "throw in a feature" because people "don't mind" is exactly what leads to feature creep of the worst kind - features that serve absolutely no real purpose. That's what we rigorously objected to in the late 1970's. No, we would NOT throw in features as they were "requested" because we didn't mind.

2) you have made no argument that the function cannot be done properly at the ends, and no argument that putting it in the network is necessary for the ends to achieve storage.

On Wednesday, July 17, 2019 7:23pm, "Dave Taht" <[email-masked]> said:

>> It is NOT the job of the IP transport layer to provide free storage for low
>> priority packets. The end-to-end argument here says: the ends can and must hold
>> packets until they are either delivered or not relevant (in RTP, they become
>> irrelevant when they get older than their desired delivery time, if you want an
>> example of the latter), SO, the network should not provide the function of
>> storage beyond the minimum needed to deal with transients.
>>
>> That means, unfortunately, that the dream of some kind of "background" path that
>> stores "low priority" packets in the network fails the end-to-end argument test.
> 
>> If you think about this, it even applies to some imaginary interplanetary IP
>> layer network. Queueing delay is not a feature of any end-to-end requirement.
>>
>> What may be desired at the router/link level in an interplanetary IP layer is
>> holding packets because a link is actually down, or using link-level error
>> correction coding or retransmission to bring the error rate down to an acceptable
>> level before declaring it down. But that's quite different - it's the link level
>> protocol, which aims to deliver minimum queueing delay under tough conditions,
>> without buffering more than needed for that (the number of bits that fit in the
>> light-speed transmission at the transmission rate.
> 
>> So, the main reason I'm saying this is because again, there are those who want to
>> implement the TCP function of reliable delivery of each packet in the links.
>> That's a very bad idea.
> 
>> > b) are not relevant to all uses of the Internet transport fabric being used by
>> the
>> > ends.
>> >
>> > The reason is that the endpoints cannot know accurately what host-host paths
>> share
>> > common queues.
>> >
>> >>      Please do not hesitate to do so after your deserved holiday, and please
>> state a
>> >> superior alternative.
>> >>
>> >> Best Regards
>> >>      Sebastian
>> >>
>> >>
>> >>>

---

From: "David P. Reed" <[email-masked]>
Date: Thu, 18 Jul 2019 11:52:22 -0400 (EDT)

On Thursday, July 18, 2019 12:31am, "Jonathan Morton" <[email-masked]> said:

>> I think are debatable - especially the focus on TCP alone, since much traffic
>> will likely move away from TCP in the near future.
> 
> As a point of order, SCE is not specific to TCP.  TCP is merely the most
> convenient congestion-aware protocol to experiment with, and therefore the one we
> have adapted first.  Other protocols which already are (or aspire to be) TCP
> friendly, especially QUIC, should also be straightforward to adapt to SCE.
>
I agree that it is important to show that SCE in the IP layer can be interpreted in each congestion management system, and TCP is a major one. Ideally there would be a general theory that is protocol and use case agnostic, so that the functions in IP are helpful both in particular protocols and also in the very important case of interacti0ns between different coexisting protocols. I believe that SCE can be structured so that it serves that purpose as more and more protocols that are based on UDP generate more and more traffic - 
when we designed UDP we decided NOT to try to put congestion management in UDP deliberately, for two reasons:
1) we didn't yet have a good congestion management approach for TCP, and
2) major use cases that argued for UDP (packet speech, in particular, but lots of computer-computer interactions on LANs, such as Remote Procedure Calls, etc.) were known to require different approaches to congestion management beyond the basic packet-drop (such as rate management via compression variation).
UDP was part of the design choice to allow end-to-end agreement about congestion management implementation.
We now have a very complex protocol due to the WWW, that imperfectly works on TCP. Thus, a new UDP based protocol framework is proposed, and will be quite heavily used in access networks that need congestion management, both at the server end and the client end.
And we have heavy use of media streaming (though it matches TCP adequately well, being file-transfer-like due to buffering at the receiving end). 

Google and others are working hard to transition entirely away from HTTP/TCP to HTTP/QUIC/UDP. This transition will be concurrent, if not prior, to SCE integration into IP. I would hope that QUIC could use SCE to great advantage, especially in helping the co-existence of two competing uses for the same bottleneck paths without queueing delay.

That's the case that matters to me, along with RTP and other uses. From browser-level monitoring, we already see many landing web pages open up HTTP requests to 100's of different server hosts concurrently. Yes, that is hundreds for one, count them, one click.

This is not a bad thing. The designers of the Internet should not be allowed to say that it is wrong. Because it isn't wrong - it's exactly what the Internet is supposed to be able to do! However, the browser or its host must have the information to avoid queue overflow in this new protocol. That means a useful means like SCE

It also, I believe, means that arbitration based on "flows" matter. So per-flow interactions matter. I don't know, but I believe that when lots of browsers end up sharing a bottleneck link/queue, per-flow scheduling may help a reasonable amount, primarily by preventing starvation of resources. (In scheduling of parallel computing, we call that "prevention of livelock". And when you have a hundred processors on a computer - which is what my day job supports, you get livelock ALL the time if you don't guarantee that all contenders on a resource get a chance.)  What does NOT matter is some complex (intserv/diffserv) differentiation at the router level, or at least not much.

> I should also note that TCP is the de-facto gold standard, by which all other
> congestion control is measured, for better or worse.  SCE is included in this,
> insofar as competing reasonably with standard TCP flows under all reasonable
> network conditions is necessary to introduce a new congestion control paradigm. 
> This, I think, is also part of the end-to-end principle.
> 
>  - Jonathan Morton

---

From: Dave Taht <[email-masked]>
Date: Thu, 18 Jul 2019 09:06:22 -0700

> 1) the idea that one should "throw in a feature" because people "don't mind" is exactly what leads to feature creep of the worst kind - features that serve absolutely no real purpose. That's what w…

I dig it. :) If only the 5G folk had had your approach.....

> 2) you have made no argument that the function cannot be done properly at the ends, and no argument that putting it in the network is necessary for the ends to achieve storage.

You are correct.

> >> will

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740

---

From: Dave Taht <[email-masked]>
Date: Thu, 18 Jul 2019 11:12:06 -0700

"David P. Reed" <[email-masked]> writes:

> I agree that it is important to show that SCE in the IP layer can be
> interpreted in each congestion management system, and TCP is a major
> one. Ideally there would be a general theory that is protocol and use
> case agnostic, so that the functions in IP are helpful both in
> particular protocols and also in the very important case of
> interacti0ns between different coexisting protocols. I believe that
> SCE can be structured so that it serves that purpose as more and more
> protocols that are based on UDP generate more and more traffic -
> 2) major use cases that argued for UDP (packet speech, in particular,
> but lots of computer-computer interactions on LANs, such as Remote
> Procedure Calls, etc.) were known to require different approaches to
> congestion management beyond the basic packet-drop (such as rate
> management via compression variation).
> We now have a very complex protocol due to the WWW, that imperfectly
> works on TCP. Thus, a new UDP based protocol framework is proposed,
> and will be quite heavily used in access networks that need congestion
> management, both at the server end and the client end.
> And we have heavy use of media streaming (though it matches TCP
> adequately well, being file-transfer-like due to buffering at the
> receiving end).
>
> Google and others are working hard to transition entirely away from
> HTTP/TCP to HTTP/QUIC/UDP. This transition will be concurrent, if not
> prior, to SCE integration into IP. I would hope that QUIC could use
> SCE to great advantage, especially in helping the co-existence of two
> competing uses for the same bottleneck paths without queueing delay.
>
> That's the case that matters to me, along with RTP and other
> uses.

These are my own primary foci as well. I think SCE would be great on
videoconferencing apps. I'd *really* like to be working on QUIC for
a variety of reasons and in a variety of situations and daemons and
libraries, but lacking funding for *anything* at present best I can
do is urge the work continue.

I sat down to try and add SCE to gcc ("google congestion control")
a few weeks back and got befuddled by just how hard it is to build
a browser from scratch nowadays.

> From browser-level monitoring, we already see many landing web
> pages open up HTTP requests to 100's of different server hosts
> concurrently. Yes, that is hundreds for one, count them, one click.

I kind of regard that number as high, but what counts for me is that
these are *persistent* mobile capable connections that can be
"nailed up" just how tcp used to be. nailing up tcp connections to
anywhere has become increasingly difficult.

prior work on mosh and wireguard is also pointing the way towards
less tcp in the internet.

I was trying to find the epic reddit discussion of this piece of
avery's regarding ipv6 adoption, can't find it, pointing here:

https://apenwarr.ca/log/20170810

>
> This is not a bad thing. The designers of the Internet should not be
> allowed to say that it is wrong. Because it isn't wrong - it's exactly
> what the Internet is supposed to be able to do! However, the browser
> or its host must have the information to avoid queue overflow in this
> new protocol. That means a useful means like SCE
>
> It also, I believe, means that arbitration based on "flows" matter. So
> per-flow interactions matter. I don't know, but I believe that when
> lots of browsers end up sharing a bottleneck link/queue, per-flow
> scheduling may help a reasonable amount, primarily by preventing
> starvation of resources. (In scheduling of parallel computing, we call
> that "prevention of livelock". And when you have a hundred processors
> on a computer - which is what my day job supports, you get livelock
> ALL the time if you don't guarantee that all contenders on a resource
> get a chance.)  What does NOT matter is some complex
> (intserv/diffserv) differentiation at the router level, or at least
> not much.

yep.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 22 Jul 2019 09:44:00 -0400

Folks,

As promised, I've pulled together and uploaded the main architectural 
arguments about per-flow scheduling that cause concern:

Per-Flow Scheduling and the End-to-End Argument 
<http://bobbriscoe.net/projects/latency/per-flow_tr.pdf>

It runs to 6 pages of reading. But I tried to make the time readers will 
have to spend worth it.

I have to get on with other stuff for a while. I've caught up with 
/reading/ this thread (after returning from my break), but refrained 
from responding to some individual points, Will try to do that soon.

Finally, I want to emphasize that the purpose is for advocates of 
per-flow scheduling to understand that there is a coherent world view 
that agrees with the arguments in this paper. If you have a different 
set of assumptions and perspectives that leads you to advocate per-flow 
scheduling and disagree with some of these arguments, that's to be 
expected.

The purpose is to explain why some people don't want FQ, and therefore 
why it's important to leave the choice open between FQ and DualQ. That 
has always been the intent of L4S, which supports both.

Bob

On 18/07/2019 06:24, Jonathan Morton wrote:
> Quoting selectively:
>
>> On 18 Jul, 2019, at 1:18 am, David P. Reed<[email-masked]>  wrote:
>>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 23 Jul 2019 01:00:54 -0400

> On 22 Jul, 2019, at 9:44 am, Bob Briscoe <[email-masked]> wrote:
> 
> As promised, I've pulled together and uploaded the main architectural arguments about per-flow scheduling that cause concern:
> It runs to 6 pages of reading. But I tried to make the time readers will have to spend worth it.

Thanks for posting this.  Upon reading, there is much to disagree with, but at least some points of view can be better understood in their context.

However, there is one thing I must take issue with up front - your repeated assertion that SCE cannot work without FQ.  As some of the plots we showed you this weekend (and maybe even earlier) demonstrate, we *do* have SCE working in a single-queue environment, even with non-SCE traffic mixed in; this is a development since your initial view of SCE several months ago.

SCE will also work just fine (as it always did) with plain RFC-3168 compliant single-queue AQMs, as might reasonably be deployed to improve the short-term overload performance of core or access networks, or as a low-cost bufferbloat mitigation mechanism in switching, head-end or CPE devices.  In that case SCE's behaviour is RFC-3168 compliant and TCP-friendly, and specifically does not require special treatment in the network.

I would therefore appreciate a revision of your paper which removes that false assertion from the several places it is made.

> Finally, I want to emphasize that the purpose is for advocates of per-flow scheduling to understand that there is a coherent world view that agrees with the arguments in this paper. If you have a different set of assumptions and perspectives that leads you to advocate per-flow scheduling and disagree with some of these arguments, that's to be expected. 
> 
> The purpose is to explain why some people don't want FQ, and therefore why it's important to leave the choice open between FQ and DualQ. That has always been the intent of L4S, which supports both. 

A central theme of your paper is a distinction of needs between times of "plenty" and "famine" in terms of network capacity.  However, the dividing line between these states is not defined.  This is a crucial deficiency when the argument made from the start is that FQ can be helpful during "famine" but is detrimental during "plenty".

One reasonable definition of such a dividing line is that "plenty" exists whenever the link is not fully utilised.  But in that case, any modern FQ algorithm will deliver packets in order of arrival, allowing all flows to use as much capacity as they desire.  Only when the packet arrival rate begins to exceed that of draining, when a "famine" state could be said to exist, does FQ apply any management at all.  Even then, if capacity is left over after metering out each flow's fair share, traffic is free to make use of it.  Thus under this definition, FQ is strictly harmless.

Reading between the lines, I suspect that what you mean is that if, after subtracting the capacity used by inelastic flows (such as the VR streams mentioned in one scenario), there is "sufficient" capacity left for elastic flows to experience good performance, then you would say there is still a state of "plenty".  But this is a highly subjective definition and liable to change over time; would 1.5Mbps leftover capacity, equivalent to a T1 line of old, be acceptable today?  I think most modern users would classify it as "famine", but it was considered quite a luxury 20 years ago.

Regardless, my assertion is not that FQ is required for ultra-low latency, but that flows requiring ultra-low latency must be isolated from general traffic.  FQ is a convenient, generic way to do that, and DualQ is another way; other ways exist, such as through Diffserv.  If both traffic types are fed through a single queue, they will also share fates with respect to latency performance, such that every little misbehaviour of the general traffic will be reflected in the latency seen by the more sensitive flows.

It is true that SCE doesn't inherently carry a label distinguishing its traffic from the general set, and thus DualQ cannot be directly applied to it.  But there is a straightforward way to perform this labelling if required, right next door in the Diffserv field.  The recently proposed NQB DSCP would likely be suitable.  I don't think that the majority of potential SCE users would need or even want this distinction (the primary benefit of SCE being better link utilisation by eliminating the traditional deep sawtooth), but the mechanism exists, orthogonally to SCE itself.

I have also drawn up, as a straw-man proposal, CNQ - Cheap Nasty Queuing:

	https://tools.ietf.org/html/draft-morton-tsvwg-cheap-nasty-queueing-00

This is a single-queue AQM, plus a side channel for prioritising sparse flows, although the definition of "sparse" is much stricter than for a true FQ implementation (even LFQ).  In essence, CNQ treats a flow as sparse if its inter-packet gap is greater than the sojourn time of the main queue, and does not attempt to enforce throughput fairness.  This is probably adequate to assist some common latency-sensitive protocols, such as ARP, SSH, NTP and DNS, as well as the initial handshake of longer-lived bulk flows.  You will also notice that there is support for SCE in the application of AQM, though the AQM algorithm itself is only generically specified.

In common with a plain single-queue AQM, CNQ will converge to approximate fairness between TCP-friendly flows, while keeping typical latencies under reasonable control.  Aggressive or meek flows will also behave as expected for a single queue, up to a limit where an extremely meek flow might fall into the sparse queue and thus limit its ability to give way.  This limit will relatively depend on the latency maintained in the main queue, and will probably be several times less than the fair share.

I hope this serves to illustrate that I'm not against single-queue AQMs in an appropriate context, but that their performance limitations need to be borne in mind.  In particular, I consider a single-queue AQM (or CNQ) to be a marked improvement over a dumb FIFO at any bottleneck.

 - Jonathan Morton

---

From: Luca Muscariello <[email-masked]>
Date: Tue, 23 Jul 2019 13:35:49 +0200

Starting a new thread as it looks like this new proposal can be
integrated in a modular way with several other loosely coupled
components by just using two hardware queues, which are
already available in current HW.

Modifications would just be in the ingress  enqueue time “self-classifier”.

It also looks like possible to append different optional packet droppers
to the backlogged queue such as AFD or others to provide
some level of flow protection to backlogged flows as well.

On Tue, Jul 23, 2019 at 7:01 AM Jonathan Morton <[email-masked]>
wrote:

>
> It is true that SCE doesn't inherently carry a label distinguishing its
> traffic from the general set, and thus DualQ cannot be directly applied to
> it.  But there is a straightforward way to perform this labelling if
> required, right next door in the Diffserv field.  The recently proposed NQB
> DSCP would likely be suitable.  I don't think that the majority of
> potential SCE users would need or even want this distinction (the primary
> benefit of SCE being better link utilisation by eliminating the traditional
> deep sawtooth), but the mechanism exists, orthogonally to SCE itself.
>
> This is a single-queue AQM, plus a side channel for prioritising sparse
> flows, although the definition of "sparse" is much stricter than for a true
> FQ implementation (even LFQ).  In essence, CNQ treats a flow as sparse if
> its inter-packet gap is greater than the sojourn time of the main queue,
> and does not attempt to enforce throughput fairness.  This is probably
> adequate to assist some common latency-sensitive protocols, such as ARP,
> SSH, NTP and DNS, as well as the initial handshake of longer-lived bulk
> flows.  You will also notice that there is support for SCE in the
> application of AQM, though the AQM algorithm itself is only generically
> specified.
>
> In common with a plain single-queue AQM, CNQ will converge to approximate
> fairness between TCP-friendly flows, while keeping typical latencies under
> reasonable control.  Aggressive or meek flows will also behave as expected
> for a single queue, up to a limit where an extremely meek flow might fall
> into the sparse queue and thus limit its ability to give way.  This limit
> will relatively depend on the latency maintained in the main queue, and
> will probably be several times less than the fair share.
>
> I hope this serves to illustrate that I'm not against single-queue AQMs in
> an appropriate context, but that their performance limitations need to be
> borne in mind.  In particular, I consider a single-queue AQM (or CNQ) to be
> a marked improvement over a dumb FIFO at any bottleneck.
>
>  - Jonathan Morton
>

---

From: Kyle Rose <[email-masked]>
Date: Tue, 23 Jul 2019 11:12:46 -0400

On Mon, Jul 22, 2019 at 9:44 AM Bob Briscoe <[email-masked]> wrote:

> Per-Flow Scheduling and the End-to-End Argum ent

Before reading the other responses (poisoning my own thinking), I wanted to
offer my own reaction. In the discussion of figure 1, you seem to imply
that there's some obvious choice of bin packing for the flows involved, but
that can't be right. What if the dark green flow has deadlines? Why should
that be the one that gets only leftover bandwidth? I'll return to this
point in a bit.

The tl;dr summary of the paper seems to be that the L4S approach leaves the
allocation of limited bandwidth up to the endpoints, while FQ arbitrarily
enforces equality in the presence of limited bandwidth; but in reality the
bottleneck device needs to make *some* choice when there's a shortage and
flows don't respond. That requires some choice of policy.

In FQ, the chosen policy is to make sure every flow has the ability to get
low latency for itself, but in the absence of some other kind of trusted
signaling allocates an equal proportion of the available bandwidth to each
flow. ISTM this is the best you can do in an adversarial environment,
because anything else can be gamed to get a more than equal share (and
depending on how "flow" is defined, even this can be gamed by opening up
more flows; but this is not a problem unique to FQ).

In L4S, the policy is to assume one queue is well-behaved and one not, and
to use the ECT(1) codepoint as a classifier to get into one or the other.
But policy choice doesn't end there: in an uncooperative or adversarial
environment, you can easily get into a situation in which the bottleneck
has to apply policy to several unresponsive flows in the supposedly
well-behaved queue. Note that this doesn't even have to involve bad actors
misclassifying on purpose: it could be two uncooperative 200 Mb VR flows
competing for 300 Mb of bandwidth. In this case, L4S falls back to classic,
which with DualQ means every flow, not just the uncooperative ones,
suffers. As a user, I don't want my small, responsive flows to suffer when
uncooperative actors decide to exceed the BBW.

Getting back to figure 1, how do you choose the right allocation? With the
proposed use of ECT(1) as classifier, you have exactly one bit available to
decide which queue, and therefore which policy, applies to a flow. Should
all the classic flows get assigned whatever is left after the L4S flows are
allocated bandwidth? That hardly seems fair to classic flows. But let's say
this policy is implemented. It then escapes me how this is any different
from the trust problems facing end-to-end DSCP/QoS: why wouldn't everyone
just classify their classic flows as L4S, forcing everything to be treated
as classic and getting access to a (greater) share of the overall BBW? Then
we're left both with a spent ECT(1) codepoint and a need for FQ or some
other queuing policy to arbitrate between flows, without any bits with
which to implement the high-fidelity congestion signal required to achieve
low latency without getting squeezed out.

The bottom line is that I see no way to escape the necessity of something
FQ-like at bottlenecks outside of the sender's trust domain. If FQ can't be
done in backbone-grade hardware, then the only real answer is pipes in the
core big enough to force the bottleneck to live somewhere closer to the
edge, where FQ does scale.

Note that, in a perfect world, FQ wouldn't trigger at all because there
would always be enough bandwidth for everything users wanted to do, but in
the real world it seems like the best you can possibly do in the absence of
trusted information about how to prioritize traffic. IMO, best to think of
FQ as a last-ditch measure indicating to the operator that they're gonna
need a bigger pipe than as a steady-state bandwidth allocator.

Kyle

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 23 Jul 2019 16:14:22 -0400

Jonathan

> I would therefore appreciate a revision of your paper which removes that false assertion from the several places it is made.
[BB] I was careful not to say "does not work", because I know your 
definition of "works" is not the same as the community working on 
ultra-low latency. I said it needs FQ "in order to provide benefit". 
I'll explain why SCE doesn't provide benefit in a response to your 
posting on LFQ, to keep both this thread and that one on topic.

> Reading between the lines, I suspect that what you mean is that if, after subtracting the capacity used by inelastic flows (such as the VR streams mentioned in one scenario), there is "sufficient"…
[BB] The objective measure of famine/plenty that all congestion controls 
use is a combination of the loss level, ECN level, queue delay, etc. In 
the paper, I referred to BBRv2's distinction between famine and plenty 
(which is at 1% loss) and in a footnote I expressed a preference for a 
more gradual transition.

Many real-time congestion controls (e.g. the algos used in production 
video chat and conferencing products etc) include such a gradual 
transition from fast responsiveness at high congestion levels (loss in 
single digit percentage area) to being virtually unresponsive at much 
lower levels of congestion.

There's lots of diversity, and this ecosystem is healthy and nowhere 
near threatening Internet collapse. So I think introducing a "standard" 
transition would just create a chilling effect on innovation.

The use of the two words famine and plenty wasn't intended to imply only 
two states. It's a continuum (like the spectrum between famine and plenty).

> It is true that SCE doesn't inherently carry a label distinguishing its traffic from the general set, and thus DualQ cannot be directly applied to it.  But there is a straightforward way to perform…
To enable SCE and RFC3168 in two queues rather than per-flow queues, if 
you required SCE packets to be identified by a DSCP, if the DSCP got 
wiped (which it often does), your SCE traffic would mix with 3168 
traffic and starve itself.

> I hope this serves to illustrate that I'm not against single-queue AQMs in an appropriate context, but that their performance limitations need to be borne in mind.  In particular, I consider a sing…
OK. Can you say whether you've tested this exhausitvely? Need to know 
before we all spend time reading it in too much depth.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 23 Jul 2019 18:24:55 -0400

> [BB] The objective measure of famine/plenty that all congestion controls use is a combination of the loss level, ECN level, queue delay, etc. In the paper, I referred to BBRv2's distinction between famine and plenty (which is at 1% loss) and in a footnote I expressed a preference for a more gradual transition.

Coincidentally, 1% loss corresponds to about 1.5Mbps goodput at an Internet-scale 80ms RTT, assuming a Reno transport.  Obviously at different RTTs it corresponds to different goodputs, and you might argue that shorter RTTs are also common.  But now we have a common reference point.

Oh, and under similar conditions, 1% marking corresponds to about 30Mbps with L4S (ie. cwnd=200).  That's a 20:1 ratio versus Reno, which you might want to think about carefully when it comes to fair competition.

> The use of the two words famine and plenty wasn't intended to imply only two states. It's a continuum (like the spectrum between famine and plenty).

Okay.

I still happen to disagree with the argument, but single-queue AQMs are still a valid improvement over single dumb FIFOs.  They improve reliability by reducing losses and timeouts, and help to reduce lag in online games.  That's the practical problem facing most Internet users today, and that's where my solutions are focused.

>> Regardless, my assertion is not that FQ is required for ultra-low latency, but that flows requiring ultra-low latency must be isolated from general traffic…
>> 

> To enable SCE and RFC3168 in two queues rather than per-flow queues, if you required SCE packets to be identified by a DSCP, if the DSCP got wiped (which it often does), your SCE traffic would mix with 3168 traffic and starve itself.

Under certain simplifying assumptions, yes.  But those assumptions would include that the 3168 queue was also providing SCE marking in the FQ style, which might not be appropriate for what is effectively a single queue carrying mixed traffic.  It would be as if your DualQ was providing L4S-style signalling in its Classic queue, which I'm sure you would not advocate.

As a de-facto representative of the cable industry, I hope you are aware of the irony that it is chiefly cable ISPs who are bleaching Diffserv information out of consumer traffic?

The starvation problem can be eliminated entirely by providing SCE marking only on the queue intended for SCE traffic (so only CE marks on the 3168 queue).  Mis-marked traffic would then revert to purely 3168 compliant behaviour, which SCE does naturally when given only CE marks.  This option is an important advantage of having a clear distinction between the two signals; there is no ambiguity at the receiver about what type of signal it's receiving and thus which response is demanded.

As a reminder, we *also* have a solution specifically for single-queue AQMs implementing SCE.  It's not a knobs-free solution as the FQ version is, but it exists and it seems to work.  I expect we will need to explore its dynamic characteristics more thoroughly in the near future.

> OK. Can you say whether you've tested this exhausitvely? Need to know before we all spend time reading it in too much depth.

To quote Knuth: "I have only proved the above code correct, not tried it."  But we may get time to quickly implement CNQ and/or LFQ, in between preparations for Friday.  (By which I mean implementing it in Linux.)  This stuff is not central to our work on SCE, since we already have Cake for running experiments.

I think Pete Heist wants to try putting CNQ in his offline simulator as well, which already has LFQ.  So that should provide an early sanity check.

 - Jonathan Morton

---

From: "Holland, Jake" <[email-masked]>
Date: Thu, 25 Jul 2019 19:25:56 +0000

Hi Kyle,

I almost agree, except that the concern is not about classic flows.

I agree (with caveats) with what Bob and Greg have said before: ordinary classic flows don’t have an incentive to mis-mark if they’ll be responding normally to CE, because a classic flow will back off too aggressively and starve itself if it’s getting CE marks from the LL queue.

That said, I had a message where I tried to express something similar to the concerns I think you just raised, with regard to a different category of flow:
https://mailarchive.ietf.org/arch/msg/tsvwg/bUu7pLmQo6BhR1mE2suJPPluW3Q

So I agree with the concerns you’ve raised here, and I want to +1 that aspect of it while also correcting that I don’t think these apply for ordinary classic flows, but rather for flows that use application-level quality metrics to change bit-rates instead responding at the transport level.

For those flows (which seems to include some of today’s video conferencing traffic), I expect they really would see an advantage by mis-marking themselves, and will require policing that imposes a policy decision.  Given that, I agree that I don’t see a simple alternative to FQ for flows originating outside the policer’s trust domain when the network is fully utilized.

I hope that makes at least a little sense.

Best regards,
Jake

From: Kyle Rose <[email-masked]>
Date: 2019-07-23 at 11:13
To: Bob Briscoe <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, tsvwg IETF list <[email-masked]>, "David P. Reed" <[email-masked]>
Subject: Re: [tsvwg] [Ecn-sane] per-flow scheduling

On Mon, Jul 22, 2019 at 9:44 AM Bob Briscoe <[email-masked]<mailto:[email-masked]>> wrote:
Folks,

Per-Flow Scheduling and the End-to-End Argum ent<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_projects_latency_per-2Dflow-5Ftr.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=PI1HWa27sXLOTKR6A5e3p0PaPt7vS4SMNHQKYIzfXxM&s=ACtkb-7e-7Ifb6QsnMjd4WSYrCfUyWGIbBuNkDZ8V3E&e=>

Before reading the other responses (poisoning my own thinking), I wanted to offer my own reaction. In the discussion of figure 1, you seem to imply that there's some obvious choice of bin packing for the flows involved, but that can't be right. What if the dark green flow has deadlines? Why should that be the one that gets only leftover bandwidth? I'll return to this point in a bit.

The tl;dr summary of the paper seems to be that the L4S approach leaves the allocation of limited bandwidth up to the endpoints, while FQ arbitrarily enforces equality in the presence of limited bandwidth; but in reality the bottleneck device needs to make *some* choice when there's a shortage and flows don't respond. That requires some choice of policy.

In FQ, the chosen policy is to make sure every flow has the ability to get low latency for itself, but in the absence of some other kind of trusted signaling allocates an equal proportion of the available bandwidth to each flow. ISTM this is the best you can do in an adversarial environment, because anything else can be gamed to get a more than equal share (and depending on how "flow" is defined, even this can be gamed by opening up more flows; but this is not a problem unique to FQ).

In L4S, the policy is to assume one queue is well-behaved and one not, and to use the ECT(1) codepoint as a classifier to get into one or the other. But policy choice doesn't end there: in an uncooperative or adversarial environment, you can easily get into a situation in which the bottleneck has to apply policy to several unresponsive flows in the supposedly well-behaved queue. Note that this doesn't even have to involve bad actors misclassifying on purpose: it could be two uncooperative 200 Mb VR flows competing for 300 Mb of bandwidth. In this case, L4S falls back to classic, which with DualQ means every flow, not just the uncooperative ones, suffers. As a user, I don't want my small, responsive flows to suffer when uncooperative actors decide to exceed the BBW.

Getting back to figure 1, how do you choose the right allocation? With the proposed use of ECT(1) as classifier, you have exactly one bit available to decide which queue, and therefore which policy, applies to a flow. Should all the classic flows get assigned whatever is left after the L4S flows are allocated bandwidth? That hardly seems fair to classic flows. But let's say this policy is implemented. It then escapes me how this is any different from the trust problems facing end-to-end DSCP/QoS: why wouldn't everyone just classify their classic flows as L4S, forcing everything to be treated as classic and getting access to a (greater) share of the overall BBW? Then we're left both with a spent ECT(1) codepoint and a need for FQ or some other queuing policy to arbitrate between flows, without any bits with which to implement the high-fidelity congestion signal required to achieve low latency without getting squeezed out.

The bottom line is that I see no way to escape the necessity of something FQ-like at bottlenecks outside of the sender's trust domain. If FQ can't be done in backbone-grade hardware, then the only real answer is pipes in the core big enough to force the bottleneck to live somewhere closer to the edge, where FQ does scale.

Note that, in a perfect world, FQ wouldn't trigger at all because there would always be enough bandwidth for everything users wanted to do, but in the real world it seems like the best you can possibly do in the absence of trusted information about how to prioritize traffic. IMO, best to think of FQ as a last-ditch measure indicating to the operator that they're gonna need a bigger pipe than as a steady-state bandwidth allocator.

Kyle

---

From: Kyle Rose <[email-masked]>
Date: Sat, 27 Jul 2019 11:35:28 -0400

Right, I understand that under 3168 behavior the sender would react
differently to ECE markings than L4S flows would, but I guess I don't
understand why a sender willing to misclassify traffic with ECT(1) wouldn't
also choose to react non-normatively to ECE markings. On the rest, I think
we agree.

Kyle

On Thu, Jul 25, 2019 at 3:26 PM Holland, Jake <[email-masked]> wrote:

> I agree (with caveats) with what Bob and Greg have said before: ordinary
> classic flows don’t have an incentive to mis-mark if they’ll be responding
> normally to CE, because a classic flow will back off too aggressively and
> starve itself if it’s getting CE marks from the LL queue.
>
>
>
> That said, I had a message where I tried to express something similar to
> the concerns I think you just raised, with regard to a different category
> of flow:
>
> So I agree with the concerns you’ve raised here, and I want to +1 that
> aspect of it while also correcting that I don’t think these apply for
> ordinary classic flows, but rather for flows that use application-level
> quality metrics to change bit-rates instead responding at the transport
> level.
>
>
>
> For those flows (which seems to include some of today’s video conferencing
> traffic), I expect they really would see an advantage by mis-marking
> themselves, and will require policing that imposes a policy decision.
> Given that, I agree that I don’t see a simple alternative to FQ for flows
> originating outside the policer’s trust domain when the network is fully
> utilized.
>
>
>
> *From: *Kyle Rose <[email-masked]>
> *Date: *2019-07-23 at 11:13
> *To: *Bob Briscoe <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>,
> tsvwg IETF list <[email-masked]>, "David P. Reed" <[email-masked]>
> *Subject: *Re: [tsvwg] [Ecn-sane] per-flow scheduling
>
>
>
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_projects_latency_per-2Dflow-5Ftr.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=PI1HWa27sXLOTKR6A5e3p0PaPt7vS4SMNHQKYIzfXxM&s=ACtkb-7e-7Ifb6QsnMjd4WSYrCfUyWGIbBuNkDZ8V3E&e=>
>
>
> Before reading the other responses (poisoning my own thinking), I wanted
> to offer my own reaction. In the discussion of figure 1, you seem to imply
> The tl;dr summary of the paper seems to be that the L4S approach leaves
> the allocation of limited bandwidth up to the endpoints, while FQ
> arbitrarily enforces equality in the presence of limited bandwidth; but in
> reality the bottleneck device needs to make *some* choice when there's a
> shortage and flows don't respond. That requires some choice of policy.
>
>
>

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 27 Jul 2019 22:42:22 +0300

RFC compliant traffic finding its way into the L4S queue is one thing - it
obviously results in a performance degradation to one or the other,
depending on other factors - but I'm more concerned about L4S traffic
entering the classic queue, which would have a severe impact on general
traffic throughout.  By default this always occurs when a single queue AQM
that is not L4S aware is encountered.