Subject: [tsvwg] per-flow scheduling

From: Bob Briscoe <[email-masked]>
Date: Wed, 19 Jun 2019 15:12:53 +0100

Jake, all,

You may not be aware of my long history of concern about how per-flow 
scheduling within endpoints and networks will limit the Internet in 
future. I find per-flow scheduling a violation of the e2e principle in 
such a profound way - the dynamic choice of the spacing between packets 
- that most people don't even associate it with the e2e principle.

I detected that you were talking about FQ in a way that might have 
assumed my concern with it was just about implementation complexity. If 
you (or anyone watching) is not aware of the architectural concerns with 
per-flow scheduling, I can enumerate them.

I originally started working on what became L4S to prove that it was 
possible to separate out reducing queuing delay from throughput 
scheduling. When Koen and I started working together on this, we 
discovered we had identical concerns on this.

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Kyle Rose <[email-masked]>
Date: Wed, 19 Jun 2019 10:20:29 -0400

> per-flow scheduling, I can enumerate them.

I would certainly be interested in reading more about these concerns. Even
a reference that I can read out-of-band would be fine. Thanks, Bob.

Kyle

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 21 Jun 2019 08:59:14 +0200

> On Jun 19, 2019, at 16:12, Bob Briscoe <[email-masked]> wrote:
> 
> Jake, all,
> 
> You may not be aware of my long history of concern about how per-flow scheduling within endpoints and networks will limit the Internet in future. I find per-flow scheduling a violation of the e2e principle in such a profound way - the dynamic choice of the spacing between packets - that most people don't even associate it with the e2e principle.

Maybe because it is not a violation of the e2e principle at all? My point is that with shared resources between the endpoints, the endpoints simply should have no expectancy that their choice of spacing between packets will be conserved. For the simple reason that it seems generally impossible to guarantee that inter-packet spacing is conserved (think "cross-traffic" at the bottleneck hop along the path and general bunching up of packets in the queue of a fast to slow transition*). I also would claim that the way L4S works (if it works) is to synchronize all active flows at the bottleneck which in tirn means each sender has only a very small timewindow in which to transmit a packet for it to hits its "slot" in the bottleneck L4S scheduler, otherwise, L4S's low queueing delay guarantees will not work. In other words the senders have basically no say in the "spacing between packets", I fail to see how L4S improves upon FQ in that regard.

 IMHO having per-flow fairness as the defaults seems quite reasonable, endpoints can still throttle flows to their liking. Now per-flow fairness still can be "abused", so by itself it might not be sufficient, but neither is L4S as it has at best stochastic guarantees, as a single queue AQM (let's ignore the RFC3168 part of the AQM) there is the probability to send a throtteling signal to a low bandwidth flow (fair enough, it is only a mild throtteling signal, but still).
But enough about my opinion, what is the ideal fairness measure in your mind, and what is realistically achievable over the internet?

Best Regards
	Sebastian

> 
> I detected that you were talking about FQ in a way that might have assumed my concern with it was just about implementation complexity. If you (or anyone watching) is not aware of the architectural concerns with per-flow scheduling, I can enumerate them.
> 
> I originally started working on what became L4S to prove that it was possible to separate out reducing queuing delay from throughput scheduling. When Koen and I started working together on this, we discovered we had identical concerns on this.
> 
> 
> 
> Bob
> 
> 
> -- 
> _______________________________________________
> Ecn-sane mailing list
> [email-masked]
> https://lists.bufferbloat.net/listinfo/ecn-sane

---

From: Luca Muscariello <[email-masked]>
Date: Fri, 21 Jun 2019 11:33:00 +0200

+ David Reed, as I'm not sure he's on the ecn-sane list.

To me, it seems like a very religious position against per-flow queueing.
BTW, I fail to see how this would violate (in a "profound" way ) the e2e
principle.

When I read it (the e2e principle)

Saltzer, J. H., D. P. Reed, and D. D. Clark (1981) "End-to-End Arguments in
System Design".
In: Proceedings of the Second International Conference on Distributed
Computing Systems. Paris, France.
April 8–10, 1981. IEEE Computer Society, pp. 509-512.
(available on line for free).

It seems very much like the application of the Occam's razor to function
placement in communication networks back in the 80s.
I see no conflict between what is written in that paper and per-flow
queueing today, even after almost 40 years.

If that was the case, then all service differentiation techniques would
violate the e2e principle in a "profound" way too,
and dualQ too. A policer? A shaper? A priority queue?

Luca

On Fri, Jun 21, 2019 at 9:00 AM Sebastian Moeller <[email-masked]> wrote:

> scheduling within endpoints and networks will limit the Internet in future.
> I find per-flow scheduling a violation of the e2e principle in such a
> profound way - the dynamic choice of the spacing between packets - that
> most people don't even associate it with the e2e principle.
>
> Maybe because it is not a violation of the e2e principle at all? My point
> is that with shared resources between the endpoints, the endpoints simply
> should have no expectancy that their choice of spacing between packets will
> be conserved. For the simple reason that it seems generally impossible to
> guarantee that inter-packet spacing is conserved (think "cross-traffic" at
> the bottleneck hop along the path and general bunching up of packets in the
> queue of a fast to slow transition*). I also would claim that the way L4S
> works (if it works) is to synchronize all active flows at the bottleneck
> which in tirn means each sender has only a very small timewindow in which
> to transmit a packet for it to hits its "slot" in the bottleneck L4S
> scheduler, otherwise, L4S's low queueing delay guarantees will not work. In
> other words the senders have basically no say in the "spacing between
> packets", I fail to see how L4S improves upon FQ in that regard.
>
>
>  IMHO having per-flow fairness as the defaults seems quite reasonable,
> endpoints can still throttle flows to their liking. Now per-flow fairness
> still can be "abused", so by itself it might not be sufficient, but neither
> is L4S as it has at best stochastic guarantees, as a single queue AQM
> (let's ignore the RFC3168 part of the AQM) there is the probability to send
> a throtteling signal to a low bandwidth flow (fair enough, it is only a
> mild throtteling signal, but still).
> But enough about my opinion, what is the ideal fairness measure in your
> mind, and what is realistically achievable over the internet?
>
>
> Best Regards
>         Sebastian
>
>
>
>
> >
> assumed my concern with it was just about implementation complexity. If you
> (or anyone watching) is not aware of the architectural concerns with
> possible to separate out reducing queuing delay from throughput scheduling.
> When Koen and I started working together on this, we discovered we had
> identical concerns on this.
> >
> >
> >
> > Bob
> >
> >
> > --

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 22 Jun 2019 08:37:59 +1200

Below...
On 21-Jun-19 21:33, Luca Muscariello wrote:
> BTW, I fail to see how this would violate (in a "profound" way ) the e2e principle.
> 
> Saltzer, J. H., D. P. Reed, and D. D. Clark (1981) "End-to-End Arguments in System Design". 
> In: Proceedings of the Second International Conference on Distributed Computing Systems. Paris, France. 
> It seems very much like the application of the Occam's razor to function placement in communication networks back in the 80s.
> I see no conflict between what is written in that paper and per-flow queueing today, even after almost 40 years.
> 
> If that was the case, then all service differentiation techniques would violate the e2e principle in a "profound" way too,

Quoting RFC2638 (the "two-bit" RFC):

>>>    Both these
>>>    proposals seek to define a single common mechanism that is used by
>>>    interior network routers, pushing most of the complexity and state of
>>>    differentiated services to the network edges.

I can't help thinking that if DDC had felt this was against the E2E principle,
he would have kicked up a fuss when it was written.

Bob's right, however, that there might be a tussle here. If end-points are
attempting to pace their packets to suit their own needs, and the network is
policing packets to support both service differentiation and fairness,
these may well be competing rather than collaborating behaviours. And there
probably isn't anything we can do about it by twiddling with algorithms.

    Brian

> 
> 
> 
> 
> 
> 
>  
> 
> On Fri, Jun 21, 2019 at 9:00 AM Sebastian Moeller <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> 
>     > On Jun 19, 2019, at 16:12, Bob Briscoe <[email-masked] <mailto:[email-masked]>> wrote:
>     >
>     > Jake, all,
>     >
>     > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 22 Jun 2019 23:47:57 +0300

> On 22 Jun, 2019, at 10:50 pm, David P. Reed <[email-masked]> wrote:
> 
> Pragmatic networks (those that operate in the real world) do not choose to operate with shared links in a saturated state. That's known in the phone business as the Mother's Day problem. You want to have enough capacity for the rare near-overload to never result in congestion.

This is most likely true for core networks.  However, I know of several real-world networks and individual links which, in practice, are regularly in a saturated and/or congested state.

Indeed, the average Internet consumer's ADSL or VDSL last-mile link becomes saturated for a noticeable interval, every time his operating system or game vendor releases an update.  In my case, I share a 3G/4G tower's airtime with whatever variable number of subscribers to the same network happen to be in the area on any given day; today, during midsummer weekend, that number is considerably inflated compared to normal, and my available link bandwidth is substantially impacted as a result, indicating congestion.

I did not see anything in your argument specifically about per-flow scheduling for the simple purpose of fairly sharing capacity between flows and/or between subscribers, and minimising the impact of elephants on mice.  Empirical evidence suggests that it makes the network run more smoothly.  Does anyone have a concrete refutation?

 - Jonathan Morton

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 23 Jun 2019 09:10:24 +1200

Just three or four small comments:

On 23-Jun-19 07:50, David P. Reed wrote:
> Two points:
> 
>  
> 
> - Jerry Saltzer and I were the primary authors of the End-to-end argument paper, and the motivation was based *my* work on the original TCP and IP protocols. Dave Clark got involved significantly later than all those decisions, which were basically complete when he got involved. (Jerry was my thesis supervisor, I was his student, and I operated largely independently, taking input from various others at MIT). I mention this because Dave understands the end-to-end arguments, but he understands (as we all did) that it was a design *principle* and not a perfectly strict rule. That said, it's a rule that has a strong foundational argument from modularity and evolvability in a context where the system has to work on a wide range of infrastructures (not all knowable in advance) and support a wide range of usage/application-areas (not all knowable in advance). Treating the paper as if it were "DDC" declaring a law is just wrong. He wasn't Moses and it is not written on tablets. Dave
> did have some "power" in his role of trying to achieve interoperability across diverse implementations. But his focus was primarily on interoperability, not other things. So ideas in the IP protocol like "TOS" which were largely placeholders for not-completely-worked-out concepts deferred to the future were left till later.

Yes, well understood, but he was in fact the link between the e2e paper and the differentiated services work. Although not a nominal author of the "two-bit" RFC, he was heavily involved in it, which is why I mentioned him. And he was very active in the IETF diffserv WG. 
> - It is clear (at least to me) that from the point of view of the source of an IP datagram, the "handling" of that datagram within the network of networks can vary, and so that is why there is a TOS field - to specify an interoperable, meaningfully described per-packet indicator of differential handling. In regards to the end-to-end argument, that handling choice is a network function, *to the extent that it can completely be implemented in the network itself*.
> 
> Congestion management, however, is not achievable entirely and only within the network. That's completely obvious: congestion happens when the source-destination flows exceed the capacity of the network of networks to satisfy all demands.
> 
> The network can only implement *certain* general kinds of mechanisms that may be used by the endpoints to resolve congestion:
> 
> 1) admission controls. These are implemented at the interface between the source entity and the network of networks. They tend to be impractical in the Internet context, because there is, by a fundamental and irreversible design choice made by Cerf and Kahn (and the rest of us), no central controller of the entire network of networks. This is to make evolvability and scalability work. 5G (not an Internet system) implies a central controller, as does SNA, LTE, and many other networks. The Internet is an overlay on top of such networks.
> 
> 2) signalling congestion to the endpoints, which will respond by slowing their transmission rate (or explicitly re-routing transmission, or compressing their content) through the network to match capacity. This response is done *above* the IP layer, and has proven very practical. The function in the network is reduced to "congestion signalling", in a universally understandable meaningful mechanism: packet drops, ECN, packet-pair separation in arrival time, ...  This limited function is essential within the network, because it is the state of the path(s) that is needed to implement the full function at the end points. So congestion signalling, like ECN, is implemented according to the end-to-end argument by carefully defining the network function to be the minimum necessary mechanism so that endpoints can control their rates.
> 
> 3) automatic selection of routes for flows. It's perfectly fine to select different routes based on information in the IP header (the part that is intended to be read and understood by the network of networks). Now this is currently *rarely* done, due to the complexity of tracking more detailed routing information at the router level. But we had expected that eventually the Internet would be so well connected that there would be diverse routes with diverse capabilities. For example, the "Interplanetary Internet" works with datagrams, that can be implemented with IP, but not using TCP, which requires very low end-to-end latency. Thus, one would expect that TCP would not want any packets transferred over a path via Mars, or for that matter a geosynchronous satellite, even if the throughput would be higher.
> 
> So one can imagine that eventually a "TOS" might say - send this packet preferably along a path that has at most 200 ms. RTT, *even if that leads to congestion signalling*, while another TOS might say "send this path over the most "capacious" set of paths, ignoring RTT entirely. (these are just for illustration, but obviously something like this woujld work).
> 
> Note that TOS is really aimed at *route selection* preferences, and not queueing management of individual routers.

That may well have been the original intention, but it was hardly mentioned at all in the diffserv WG (which I co-chaired), and "QOS-based routing" was in very bad odour at that time.
  
> 
> Queueing management to share a single queue on a path for multiple priorities of traffic is not very compatible with "end-to-end arguments". There are any number of reasons why this doesn't work well. I can go into them. Mainly these reasons are why "diffserv" has never been adopted - 

Oh, but it has, in lots of local deployments of voice over IP for example. It's what I've taken to calling a limited domain protocol. What has not happened is Internet-wide deployment, because...

> it's NOT interoperable because the diversity of traffic between endpoints is hard to specify in a way that translates into the network mechanisms. Of course any queue can be managed in some algorithmic way with parameters, but the endpoints that want to specify an end-to-end goal don't have a way to understand the impact of those parameters on a specific queue that is currently congested.

Yes. And thanks for your insights.

   Brian

> 
>  
> 
> Instead, the history of the Internet (and for that matter *all* networks, even Bell's voice systems) has focused on minimizing queueing delay to near zero throughout the network by whatever means it has at the endpoints or in the design. This is why we have AIMD's MD as a response to detection of congestion.
> 
>  
> 
> Pragmatic networks (those that operate in the real world) do not choose to operate with shared links in a saturated state. That's known in the phone business as the Mother's Day problem. You want to have enough capacity for the rare near-overload to never result in congestion.  Which means that the normal state of the network is very lightly loaded indeed, in order to minimize RTT. Consequently, focusing on somehow trying to optimize the utilization of the network to 100% is just a purely academic exercise. Since "priority" at the packet level within a queue only improves that case, it's just a focus of (bad) Ph.D. theses. (Good Ph.D. theses focus on actual real problems like getting the queues down to 1 packet or less by signalling the endpoints with information that allows them to do their job).
> 
>  
> 
> So, in considering what goes in the IP layer, both its header and the mechanics of the network of networks, it is those things that actually have implementable meaning in the network of networks when processing the IP datagram. The rest is "content" because the network of networks doesn't need to see it.
> 
>  
> 
> Thus, don't put anything in the IP header that belongs in the "content" part, just being a signal between end points. Some information used in the network of networks is also logically carried between endpoints.
> 
>  
> 
>  
> 
> On Friday, June 21, 2019 4:37pm, "Brian E Carpenter" <[email-masked]> said:
> 
>> Below...
>> > > You may not be aware of my long history of concern about how
>> per-flow scheduling within endpoints and networks will limit the Internet in
>> future. I find per-flow scheduling a violation of the e2e principle in such a
>> profound way - the dynamic choice of the spacing between packets - that most
>> people don't even associate it with the e2e principle.
>> >
>> is that with shared resources between the endpoints, the endpoints simply should
>> have no expectancy that their choice of spacing between packets will be conserved.
>> For the simple reason that it seems generally impossible to guarantee that
>> inter-packet spacing is conserved (think "cross-traffic" at the bottleneck hop
>> along the path and general bunching up of packets in the queue of a fast to slow
>> transition*). I also would claim that the way L4S works (if it works) is to
>> synchronize all active flows at the bottleneck which in tirn means each sender has
>> only a very small timewindow in which to transmit a packet for it to hits its
>> "slot" in the bottleneck L4S scheduler, otherwise, L4S's low queueing delay
>> guarantees will not work. In other words the senders have basically no say in the
>> "spacing between packets", I fail to see how L4S improves upon FQ in that regard.
>> >
>> >
>> >  IMHO having per-flow fairness as the defaults seems quite
>> reasonable, endpoints can still throttle flows to their liking. Now per-flow
>> fairness still can be "abused", so by itself it might not be sufficient, but
>> neither is L4S as it has at best stochastic guarantees, as a single queue AQM
>> (let's ignore the RFC3168 part of the AQM) there is the probability to send a
>> throtteling signal to a low bandwidth flow (fair enough, it is only a mild
>> throtteling signal, but still).
>> assumed my concern with it was just about implementation complexity. If you (or
>> anyone watching) is not aware of the architectural concerns with per-flow
>> scheduling, I can enumerate them.
>> > >
>> possible to separate out reducing queuing delay from throughput scheduling. When
>> Koen and I started working together on this, we discovered we had identical
>> concerns on this.
>> > >
>> > >
>> > >
>> > > Bob
>> > >
>> > >
>> > > --
>> <mailto:[email-masked]>
>> <mailto:[email-masked]>

---

From: Luca Muscariello <[email-masked]>
Date: Sun, 23 Jun 2019 00:03:35 +0200

On Sat 22 Jun 2019 at 22:48, Jonathan Morton <[email-masked]> wrote:

> > Pragmatic networks (those that operate in the real world) do not choose
> to operate with shared links in a saturated state. That's known in the
> phone business as the Mother's Day problem. You want to have enough
> capacity for the rare near-overload to never result in congestion.
>
> This is most likely true for core networks.  However, I know of several
> real-world networks and individual links which, in practice, are regularly
> in a saturated and/or congested state.
>
> Indeed, the average Internet consumer's ADSL or VDSL last-mile link
> becomes saturated for a noticeable interval, every time his operating
> system or game vendor releases an update.  In my case, I share a 3G/4G
> tower's airtime with whatever variable number of subscribers to the same
> network happen to be in the area on any given day; today, during midsummer
> weekend, that number is considerably inflated compared to normal, and my
> available link bandwidth is substantially impacted as a result, indicating
> congestion.
>
> I did not see anything in your argument specifically about per-flow
> scheduling for the simple purpose of fairly sharing capacity between flows
> and/or between subscribers, and minimising the impact of elephants on
> mice.  Empirical evidence suggests that it makes the network run more
> smoothly.  Does anyone have a concrete refutation?
>
>  - Jonathan Morton

I don’t think you would be able to find a refutation.
Going back for a second to what David and also Brian have said about
diffserv, QoS have proved to be an intractable  problem and I won’t blame
those who have tried to propose solutions that currently work under very
special  circumstances.  Things have not changed to make that problem
simpler, quite the opposite, mostly because the mix of applications is way
more diverse today with less predictable patters.

If I apply the same mindset used in  David’s paper, i.e. the Occam’s razor,
to get design principles to obtain a solution that is simple
and tractable,  flow-queuing in your DSL link looks like a perfectly
acceptable solution.
And I say that w/o religious positions.

The fact that flow-isolation generates incentives in sources to well behave
is good evidence to me.
Also the fact that even in situations that may look like the law of the
jungle, flow-isolation brings me performance that is predictable.
That brings more evidence that the solution is a good one.
In this respect Fq_codel (RFC 8290) looks like a simple useful tool.

---

From: Luca Muscariello <[email-masked]>
Date: Sun, 23 Jun 2019 00:30:18 +0200

Thanks for the insights.

On Sun 23 Jun 2019 at 00:25, David P. Reed <[email-masked]> wrote:

> Given the complexity of my broader comments, let me be clear that I have
> no problem with the broad concept of diffserv being compatible with the
> end-to-end arguments. I was trying to lay out what I think is a useful way
> to think about these kinds of issues within the Internet context.
>
>
>
> Similarly, per-flow scheduling as an end-to-end concept (different flows
> defined by address pairs being jointly managed as entities) makes great
> sense, but it's really important to be clear that queue prioritization
> within a single queue at entry to a bottleneck link is a special case
> mechanism, and not a general end-to-end concept at the IP datagram level,
> given the generality of IP as a network packet transport protocol. It's
> really tied closely to routing, which isn't specified in any way by IP,
> other than "best efforts", a term that has become much more well defined
> over the years (including the notions of dropping rather than storing
> packets, the idea that successive IP datagrams should traverse roughly the
> same path in order to have stable congestion detection, ...).
>
>
>
> Per-flow scheduling seems to work quite well in the cases where it
> applies, transparently below the IP datagram layer (that is, underneath the
> hourglass neck). IP effectively defines "flows", and it is reasonable to me
> that "best efforts" as a concept could include some notion of network-wide
> fairness among flows. Link-level "fairness" isn't a necessary precondition
> to network level fairness.
>
>
>
> On Saturday, June 22, 2019 5:10pm, "Brian E Carpenter" <
> [email-masked]> said:
>
> > > - Jerry Saltzer and I were the primary authors of the End-to-end
> argument
> > paper, and the motivation was based *my* work on the original TCP and IP
> > protocols. Dave Clark got involved significantly later than all those
> decisions,
> > which were basically complete when he got involved. (Jerry was my thesis
> > supervisor, I was his student, and I operated largely independently,
> taking input
> > from various others at MIT). I mention this because Dave understands the
> > end-to-end arguments, but he understands (as we all did) that it was a
> design
> > *principle* and not a perfectly strict rule. That said, it's a rule that
> has a
> > strong foundational argument from modularity and evolvability in a
> context where
> > the system has to work on a wide range of infrastructures (not all
> knowable in
> > advance) and support a wide range of usage/application-areas (not all
> knowable in
> > advance). Treating the paper as if it were "DDC" declaring a law is just
> wrong. He
> > wasn't Moses and it is not written on tablets. Dave
> > > did have some "power" in his role of trying to achieve interoperability
> > across diverse implementations. But his focus was primarily on
> interoperability,
> > not other things. So ideas in the IP protocol like "TOS" which were
> largely
> > placeholders for not-completely-worked-out concepts deferred to the
> future were
> > left till later.
> >
> > Yes, well understood, but he was in fact the link between the e2e paper
> and the
> > differentiated services work. Although not a nominal author of the
> "two-bit" RFC,
> > he was heavily involved in it, which is why I mentioned him. And he was
> very
> > active in the IETF diffserv WG.
> > > - It is clear (at least to me) that from the point of view of the
> source of
> > an IP datagram, the "handling" of that datagram within the network of
> networks can
> > vary, and so that is why there is a TOS field - to specify an
> interoperable,
> > meaningfully described per-packet indicator of differential handling. In
> regards
> > to the end-to-end argument, that handling choice is a network function,
> *to the
> > extent that it can completely be implemented in the network itself*.
> > >
> > > Congestion management, however, is not achievable entirely and only
> within
> > the network. That's completely obvious: congestion happens when the
> > source-destination flows exceed the capacity of the network of networks
> to satisfy
> > all demands.
> > >
> > > The network can only implement *certain* general kinds of mechanisms
> that may
> > be used by the endpoints to resolve congestion:
> > >
> > > 1) admission controls. These are implemented at the interface between
> the
> > source entity and the network of networks. They tend to be impractical
> in the
> > Internet context, because there is, by a fundamental and irreversible
> design
> > choice made by Cerf and Kahn (and the rest of us), no central controller
> of the
> > entire network of networks. This is to make evolvability and scalability
> work. 5G
> > (not an Internet system) implies a central controller, as does SNA, LTE,
> and many
> > other networks. The Internet is an overlay on top of such networks.
> > >
> > > 2) signalling congestion to the endpoints, which will respond by
> slowing
> > their transmission rate (or explicitly re-routing transmission, or
> compressing
> > their content) through the network to match capacity. This response is
> done
> > *above* the IP layer, and has proven very practical. The function in the
> network
> > is reduced to "congestion signalling", in a universally understandable
> meaningful
> > mechanism: packet drops, ECN, packet-pair separation in arrival time,
> ...
> > This limited function is essential within the network, because it is the
> state of
> > the path(s) that is needed to implement the full function at the end
> points. So
> > congestion signalling, like ECN, is implemented according to the
> end-to-end
> > argument by carefully defining the network function to be the minimum
> necessary
> > mechanism so that endpoints can control their rates.
> > >
> > > 3) automatic selection of routes for flows. It's perfectly fine to
> select
> > different routes based on information in the IP header (the part that is
> intended
> > to be read and understood by the network of networks). Now this is
> currently
> > *rarely* done, due to the complexity of tracking more detailed routing
> information
> > at the router level. But we had expected that eventually the Internet
> would be so
> > well connected that there would be diverse routes with diverse
> capabilities. For
> > example, the "Interplanetary Internet" works with datagrams, that can be
> > implemented with IP, but not using TCP, which requires very low
> end-to-end
> > latency. Thus, one would expect that TCP would not want any packets
> transferred
> > over a path via Mars, or for that matter a geosynchronous satellite,
> even if the
> > throughput would be higher.
> > >
> > > So one can imagine that eventually a "TOS" might say - send this packet
> > preferably along a path that has at most 200 ms. RTT, *even if that
> leads to
> > congestion signalling*, while another TOS might say "send this path over
> the most
> > "capacious" set of paths, ignoring RTT entirely. (these are just for
> illustration,
> > but obviously something like this woujld work).
> > >
> > > Note that TOS is really aimed at *route selection* preferences, and not
> > queueing management of individual routers.
> >
> > That may well have been the original intention, but it was hardly
> mentioned at all
> > in the diffserv WG (which I co-chaired), and "QOS-based routing" was in
> very bad
> > odour at that time.
> >
> > >
> > > Queueing management to share a single queue on a path for multiple
> priorities
> > of traffic is not very compatible with "end-to-end arguments". There are
> any
> > number of reasons why this doesn't work well. I can go into them. Mainly
> these
> > reasons are why "diffserv" has never been adopted -
> >
> > Oh, but it has, in lots of local deployments of voice over IP for
> example. It's
> > what I've taken to calling a limited domain protocol. What has not
> happened is
> > Internet-wide deployment, because...
> >
> > > it's NOT interoperable because the diversity of traffic between
> endpoints is
> > hard to specify in a way that translates into the network mechanisms. Of
> course
> > any queue can be managed in some algorithmic way with parameters, but the
> > endpoints that want to specify an end-to-end goal don't have a way to
> understand
> > the impact of those parameters on a specific queue that is currently
> congested.
> >
> > > Instead, the history of the Internet (and for that matter *all*
> networks,
> > even Bell's voice systems) has focused on minimizing queueing delay to
> near zero
> > throughout the network by whatever means it has at the endpoints or in
> the design.
> > This is why we have AIMD's MD as a response to detection of congestion.
> > >
> > >
> > >
> > > Pragmatic networks (those that operate in the real world) do not
> choose to
> > operate with shared links in a saturated state. That's known in the
> phone business
> > as the Mother's Day problem. You want to have enough capacity for the
> rare
> > near-overload to never result in congestion.  Which means that the normal
> > state of the network is very lightly loaded indeed, in order to minimize
> RTT.
> > Consequently, focusing on somehow trying to optimize the utilization of
> the
> > network to 100% is just a purely academic exercise. Since "priority" at
> the packet
> > level within a queue only improves that case, it's just a focus of (bad)
> Ph.D.
> > theses. (Good Ph.D. theses focus on actual real problems like getting
> the queues
> > down to 1 packet or less by signalling the endpoints with information
> that allows
> > them to do their job).
> > >
> > >
> > >
> > > So, in considering what goes in the IP layer, both its header and the
> > mechanics of the network of networks, it is those things that actually
> have
> > implementable meaning in the network of networks when processing the IP
> datagram.
> > The rest is "content" because the network of networks doesn't need to
> see it.
> > >
> > >
> > >
> > > Thus, don't put anything in the IP header that belongs in the
> "content" part,
> > just being a signal between end points. Some information used in the
> network of
> > networks is also logically carried between endpoints.
> > >
> > >
> > >
> > >
> > >
> > >> >>> proposals seek to define a single common mechanism that is
> > used
> > >> by
> > >> >>> interior network routers, pushing most of the complexity and
> > state
> > >> of
> > >> attempting to pace their packets to suit their own needs, and the
> network
> > is
> > >> > On Fri, Jun 21, 2019 at 9:00 AM Sebastian Moeller
> > <[email-masked]
> > >> <mailto:[email-masked]>> wrote:
> > >> >
> > >> >
> > >> >
> > >> per-flow scheduling within endpoints and networks will limit the
> Internet
> > in
> > such a
> > most
> > should
> > >> have no expectancy that their choice of spacing between packets will
> be
> > conserved.
> > >> along the path and general bunching up of packets in the queue of a
> fast
> > to slow
> > >> transition*). I also would claim that the way L4S works (if it works)
> is
> > to
> > >> synchronize all active flows at the bottleneck which in tirn means
> each
> > sender has
> > >> only a very small timewindow in which to transmit a packet for it to
> hits
> > its
> > >> "spacing between packets", I fail to see how L4S improves upon FQ in
> that
> > regard.
> > >> >
> > >> >
> > >> fairness still can be "abused", so by itself it might not be
> sufficient,
> > but
> > >> neither is L4S as it has at best stochastic guarantees, as a single
> queue
> > AQM
> > you (or
> > scheduling. When
> > >> > > Bob Briscoe
> >  http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Sun, 23 Jun 2019 02:07:08 +0300

> On 23 Jun, 2019, at 1:09 am, David P. Reed <[email-masked]> wrote:
> 
> per-flow scheduling is appropriate on a shared link. However, the end-to-end argument would suggest that the network not try to divine which flows get preferred.
> And beyond the end-to-end argument, there's a practical problem - since the ideal state of a shared link means that it ought to have no local backlog in the queue, the information needed to schedule "fairly" isn't in the queue backlog itself.  If there is only one packet, what's to schedule?

This is a great straw-man.  Allow me to deconstruct it.

The concept that DRR++ has empirically proved is that flows can be classified into two categories - sparse and saturating - very easily by the heuristic that a saturating flow's arrival rate exceeds its available delivery rate, and the opposite is true for a sparse flow.

An excessive arrival rate results in a standing queue; with Reno, the excess arrival rate after capacity is reached is precisely 1 segment per RTT, very small next to modern link capacities.  If there is no overall standing queue, then by definition all of the flows passing through are currently sparse.  DRR++ (as implemented in fq_codel and Cake) ensures that all sparse traffic is processed with minimum delay and no AQM activity, while saturating traffic is metered out fairly and given appropriate AQM signals.

> In fact, what the ideal queueing discipline would do is send signals to the endpoints that provide information as to what each flow's appropriate share is, and/or how far its current share is from what's fair.

The definition of which flows are sparse and which are saturating shifts dynamically in response to endpoint behaviour.

> Well, presumably the flows have definable average rates.

Today's TCP traffic exhibits the classic sawtooth behaviour - which has a different shape and period with CUBIC than Reno, but is fundamentally similar.  The sender probes capacity by increasing send rate until a congestion signal is fed back to it, at which point it drops back sharply.  With efficient AQM action, a TCP flow will therefore spend most of its time "sparse" and using less than the available path capacity, with occasional excursions into "saturating" territory which are fairly promptly terminated by AQM signals.

So TCP does *not* have a definable "average rate".  It grows to fill available capacity, just like the number of cars on a motorway network.

The recent work on high-fidelity ECN (including SCE) aims to eliminate the sawtooth, so that dropping out of "saturating" mode is done faster and by only a small margin, wasting less capacity and reducing peak delays - very close to ideal control as you describe.  But it's still necessary to avoid giving these signals unnecessarily to "sparse" flows, which would cause them to back off and thus waste capacity, but only to "saturating" flows that have just begun to build a queue.  And it's also necessary to protect these well-behaved "modern" flows from "legacy" endpoint behaviour, and vice versa.  DRR++ does that very naturally.

> Merely re-ordering the packets on a link is just not very effective at achieving fairness.

I'm afraid this assertion is simply false.  DRR++ does precisely that, and achieves near-perfect fairness.

It is important however to define "flow" correctly relative to the measure of fairness you want to achieve.  Traditionally the unique 5-tuple is used to define "flow", but this means applications can game the system by opening multiple flows.  For an ISP a better definition might be that each subscriber's traffic is one "flow".  Or there is a tweak to DRR++ which allows a two-layer fairness definition, implemented successfully in Cake.

> So the end-to-end approach would suggest moving most of the scheduling back to the endpoints of each flow, with the role of the routers being to extract information about the competing flows that are congesting the network, and forwarding those signals (via drops or marking) to the endpoints. That's because, in the end-to-end argument that applies here - the router cannot do the entire function of managing congestion or priority.

It must be remembered that congestion signals require one RTT to circulate from the bottleneck, via the receiver, back to the sender, and their effects to then be felt at the bottleneck.  That's typically a much longer response time (say 100ms for a general Internet path) than can be achieved by packet scheduling (sub-millisecond for a 20Mbps link), and therefore effects only a looser control (by fundamental control theory).  Both mechanisms are useful and complement each other.

My personal interpretation of the end-to-end principle is that endpoints generally do not, cannot, and *should not* be aware of the topology of the network between them, nor of any other traffic that might be sharing that network.  The network itself takes care of those details, and may send standardised control-feedback signals to the endpoints to inform them about actions they need to take.  These currently take the form of ICMP error packets and the ECN field, the latter substituted by packet drops on Not-ECT flows.

 - Jonathan Morton

---

From: "David P. Reed" <[email-masked]>
Date: Sat, 22 Jun 2019 18:09:37 -0400 (EDT)

Good point, but saturation is not congestion, in my view. Optimal state of a single bottleneck link is a queue of length <= 1. This can be maintained even under full load, by endpoint rate control. (Note that the "Little's Law" result in queueing theory is for a special case of queuing due to uncontrolled Poisson random arrivals. The traffic you refer to, downloading an update, is about as far from Poisson-random as possible. It's neither random nor Poisson).
 
But I take your point that there will be bottlenecked links that are special, and thus congestion control comes into play on those links.

In fact, what the ideal queueing discipline would do is send signals to the endpoints that provide information as to what each flow's appropriate share is, and/or how far its current share is from what's fair. A somewhat less optimal discipline to achieve fairness might also drop packets beyond the fair share. Dropping is the best way to signal the endpoint that it is trying to overuse the link.

Now what to do to determine fair share if the average number of packets in a saturated link's queue is <= 1?
 
Well, presumably the flows have definable average rates. So  good statistical estimators of the average rate of each flow exist. For example a table with a moving average of the rate of bytes on each flow can be maintained, which provides wonderful information about the recent history of capacities used by each flow. This allows signalling of overuse to the endpoints of each flow, either by dropping packets or by some small number of bits (e.g. ECN marking). Underuse can also be signalled.
 
We know (from pragmatic observation) that flows defined as source-host, dest-host pairs, have relatively stable demand over many packets, and many RTT's.  More specific flows (say, defined using TCP and UDP ports, the "extended header" that maybe should have been included in the IP layer) are less stable in some protocol usages we see.
 
However, what's clear is that something like fq_codel can be created that does not need to accumulate long queues on links to have an effect - obviously queues still result from transient overload, but one wants to avoid requiring queueing to build up in order to balance traffic fairly.
 
Ideally, in a situation where some definition of fairness needs to be implemented among flows competing for a limited resource, the queueing should build up at the source endpoint as much as possible, and yet the point where the conflict occurs may be deep within the network of networks.  However, once we start being able to balance loads across many independent paths (rather than assuming that there is exactly one path per flow), the definition of "fairness" becomes FAR more complicated, because the competing flows may not be between a particular source and destination pair - instead the traffic in each flow can be managed across different sets of multipaths.

The case of the "uplink" from a residential network is a common, but not general case. Once a residence has multiple uplinks (say in a Mesh neighborhood network or just in a complex wireless situation where multiple access points are reachable) the idea of localizing priority or congestion management in one box below the IP layer no longer works well at all.
 
On Saturday, June 22, 2019 4:47pm, "Jonathan Morton" <[email-masked]> said:

> > Pragmatic networks (those that operate in the real world) do not choose to
> operate with shared links in a saturated state. That's known in the phone business
> as the Mother's Day problem. You want to have enough capacity for the rare
> near-overload to never result in congestion.
> 
> This is most likely true for core networks. However, I know of several real-world
> networks and individual links which, in practice, are regularly in a saturated
> and/or congested state.
> 
> Indeed, the average Internet consumer's ADSL or VDSL last-mile link becomes
> saturated for a noticeable interval, every time his operating system or game
> vendor releases an update. In my case, I share a 3G/4G tower's airtime with
> whatever variable number of subscribers to the same network happen to be in the
> area on any given day; today, during midsummer weekend, that number is
> considerably inflated compared to normal, and my available link bandwidth is
> substantially impacted as a result, indicating congestion.
> 
> I did not see anything in your argument specifically about per-flow scheduling for
> the simple purpose of fairly sharing capacity between flows and/or between
> subscribers, and minimising the impact of elephants on mice. Empirical evidence
> suggests that it makes the network run more smoothly. Does anyone have a concrete
> refutation?
> 
> - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 24 Jun 2019 22:31:56 +0300

> On 24 Jun, 2019, at 9:57 pm, David P. Reed <[email-masked]> wrote:
> 
> TCP doesn't have a "natural sawtooth" - that is the response of TCP to a particular "queueing discipline" in a particular kind of a router - it would respond differently (and does!) if the router were to drop packets randomly on a Poisson basis, for example. No sawtooth at all.

I challenge you to show me a Reno or CUBIC based connection's cwnd evolution that *doesn't* resemble a sawtooth, regardless of the congestion signalling employed.  And I will show you that it either has severe underutilisation of the link, or is using SCE signals.  The sawtooth is characteristic of the AIMD congestion control algorithm.

 - Jonathan Morton

---

From: Jonathan Morton <[email-masked]>
Date: Mon, 24 Jun 2019 23:14:06 +0300

> On 24 Jun, 2019, at 10:50 pm, David P. Reed <[email-masked]> wrote:
> 
> My overall point here is that you seem to live in a world of academic-like purity - all TCP connections are essentially huge file transfers, where there are no delays on production or consumption of packets at the endpoint, there is no multiplexing or scheduling of processes in the endpoint operating systems, etc.

On the contrary, I've found that per-flow queuing algorithms like DRR++ cope naturally and very nicely with all sorts of deviations from the ideal, including for example the wild variations in goodput and RTT associated with wifi links.

These are exactly the kinds of complication that I imagine - and not merely in the abstract but through observation - that a pure end-to-end approach would have great difficulty in accommodating elegantly.

 - Jonathan Morton

---

From: Luca Muscariello <[email-masked]>
Date: Mon, 24 Jun 2019 23:25:35 +0200

On Mon, Jun 24, 2019 at 9:32 PM Jonathan Morton <[email-masked]>
wrote:

> > TCP doesn't have a "natural sawtooth" - that is the response of TCP to a
> particular "queueing discipline" in a particular kind of a router - it
> would respond differently (and does!) if the router were to drop packets
> randomly on a Poisson basis, for example. No sawtooth at all.
>
> I challenge you to show me a Reno or CUBIC based connection's cwnd
> evolution that *doesn't* resemble a sawtooth, regardless of the congestion
> signalling employed.  And I will show you that it either has severe
> underutilisation of the link, or is using SCE signals.  The sawtooth is
> characteristic of the AIMD congestion control algorithm.
>

Jonathan,
even if it is news to nobody,
AIMD does not necessarily converge to a limit cycle (e.g. a sawtooth). It
depends on  some regularity conditions of the AIMD law and congestion
feedback too.
For instance,  AIMD delay based congestion control or AIMD with certain
 ECN laws may display no limit cycles. Still under certain conditions.
Just to recall that the problem is a little more complex than that in
general.

---

From: "David P. Reed" <[email-masked]>
Date: Sat, 22 Jun 2019 18:25:13 -0400 (EDT)

Given the complexity of my broader comments, let me be clear that I have no problem with the broad concept of diffserv being compatible with the end-to-end arguments. I was trying to lay out what I think is a useful way to think about these kinds of issues within the Internet context.
 
Similarly, per-flow scheduling as an end-to-end concept (different flows defined by address pairs being jointly managed as entities) makes great sense, but it's really important to be clear that queue prioritization within a single queue at entry to a bottleneck link is a special case mechanism, and not a general end-to-end concept at the IP datagram level, given the generality of IP as a network packet transport protocol. It's really tied closely to routing, which isn't specified in any way by IP, other than "best efforts", a term that has become much more well defined over the years (including the notions of dropping rather than storing packets, the idea that successive IP datagrams should traverse roughly the same path in order to have stable congestion detection, ...).
 
Per-flow scheduling seems to work quite well in the cases where it applies, transparently below the IP datagram layer (that is, underneath the hourglass neck). IP effectively defines "flows", and it is reasonable to me that "best efforts" as a concept could include some notion of network-wide fairness among flows. Link-level "fairness" isn't a necessary precondition to network level fairness.
 
On Saturday, June 22, 2019 5:10pm, "Brian E Carpenter" <[email-masked]> said:

> > - Jerry Saltzer and I were the primary authors of the End-to-end argument
> protocols. Dave Clark got involved significantly later than all those decisions,
> supervisor, I was his student, and I operated largely independently, taking input
> end-to-end arguments, but he understands (as we all did) that it was a design
> *principle* and not a perfectly strict rule. That said, it's a rule that has a
> strong foundational argument from modularity and evolvability in a context where
> the system has to work on a wide range of infrastructures (not all knowable in
> advance) and support a wide range of usage/application-areas (not all knowable in
> advance). Treating the paper as if it were "DDC" declaring a law is just wrong. He
> across diverse implementations. But his focus was primarily on interoperability,
> not other things. So ideas in the IP protocol like "TOS" which were largely
> placeholders for not-completely-worked-out concepts deferred to the future were
> left till later.
> 
> Yes, well understood, but he was in fact the link between the e2e paper and the
> differentiated services work. Although not a nominal author of the "two-bit" RFC,
> he was heavily involved in it, which is why I mentioned him. And he was very
> > - It is clear (at least to me) that from the point of view of the source of
> an IP datagram, the "handling" of that datagram within the network of networks can
> vary, and so that is why there is a TOS field - to specify an interoperable,
> meaningfully described per-packet indicator of differential handling. In regards
> to the end-to-end argument, that handling choice is a network function, *to the
> > Congestion management, however, is not achievable entirely and only within
> source-destination flows exceed the capacity of the network of networks to satisfy
> all demands.
> >
> > The network can only implement *certain* general kinds of mechanisms that may
> > 1) admission controls. These are implemented at the interface between the
> source entity and the network of networks. They tend to be impractical in the
> Internet context, because there is, by a fundamental and irreversible design
> choice made by Cerf and Kahn (and the rest of us), no central controller of the
> entire network of networks. This is to make evolvability and scalability work. 5G
> (not an Internet system) implies a central controller, as does SNA, LTE, and many
> > 2) signalling congestion to the endpoints, which will respond by slowing
> their transmission rate (or explicitly re-routing transmission, or compressing
> their content) through the network to match capacity. This response is done
> *above* the IP layer, and has proven very practical. The function in the network
> is reduced to "congestion signalling", in a universally understandable meaningful
> mechanism: packet drops, ECN, packet-pair separation in arrival time, ... 
> This limited function is essential within the network, because it is the state of
> the path(s) that is needed to implement the full function at the end points. So
> congestion signalling, like ECN, is implemented according to the end-to-end
> argument by carefully defining the network function to be the minimum necessary
> > 3) automatic selection of routes for flows. It's perfectly fine to select
> different routes based on information in the IP header (the part that is intended
> to be read and understood by the network of networks). Now this is currently
> *rarely* done, due to the complexity of tracking more detailed routing information
> at the router level. But we had expected that eventually the Internet would be so
> well connected that there would be diverse routes with diverse capabilities. For
> implemented with IP, but not using TCP, which requires very low end-to-end
> latency. Thus, one would expect that TCP would not want any packets transferred
> over a path via Mars, or for that matter a geosynchronous satellite, even if the
> preferably along a path that has at most 200 ms. RTT, *even if that leads to
> congestion signalling*, while another TOS might say "send this path over the most
> "capacious" set of paths, ignoring RTT entirely. (these are just for illustration,
> That may well have been the original intention, but it was hardly mentioned at all
> in the diffserv WG (which I co-chaired), and "QOS-based routing" was in very bad
> odour at that time.
>  
> >
> > Queueing management to share a single queue on a path for multiple priorities
> of traffic is not very compatible with "end-to-end arguments". There are any
> number of reasons why this doesn't work well. I can go into them. Mainly these
> Oh, but it has, in lots of local deployments of voice over IP for example. It's
> what I've taken to calling a limited domain protocol. What has not happened is
> > it's NOT interoperable because the diversity of traffic between endpoints is
> hard to specify in a way that translates into the network mechanisms. Of course
> endpoints that want to specify an end-to-end goal don't have a way to understand
> the impact of those parameters on a specific queue that is currently congested.
> 
> > Instead, the history of the Internet (and for that matter *all* networks,
> even Bell's voice systems) has focused on minimizing queueing delay to near zero
> throughout the network by whatever means it has at the endpoints or in the design.
> state of the network is very lightly loaded indeed, in order to minimize RTT.
> Consequently, focusing on somehow trying to optimize the utilization of the
> network to 100% is just a purely academic exercise. Since "priority" at the packet
> level within a queue only improves that case, it's just a focus of (bad) Ph.D.
> theses. (Good Ph.D. theses focus on actual real problems like getting the queues
> down to 1 packet or less by signalling the endpoints with information that allows
> them to do their job).
> >
> >  
> >
> mechanics of the network of networks, it is those things that actually have
> implementable meaning in the network of networks when processing the IP datagram.
> The rest is "content" because the network of networks doesn't need to see it.
> >
> >  
> >
> > Thus, don't put anything in the IP header that belongs in the "content" part,
> just being a signal between end points. Some information used in the network of

---

From: "David P. Reed" <[email-masked]>
Date: Mon, 24 Jun 2019 14:57:58 -0400 (EDT)

Jonathan - all of the things you say are kind of silly. An HTTP 1.1 protocol running over TCP is not compatible with this description, except in "fantasyland".
 
I think you are obsessed with some idea of "proving me wrong". That's not productive.
 
If you have actual data describing how HTTP 1.1 connections proceed over time that disagrees with my observation, show them. Preferably taken in the wild.
 
I honestly can't imagine that you have actually observed any system other than the constrained single connection between a LAN and a residential ISP.

So you seem to see routers are part of TCP. That's not the way the Internet is designed.

 
On Saturday, June 22, 2019 7:07pm, "Jonathan Morton" <[email-masked]> said:

> > per-flow scheduling is appropriate on a shared link. However, the end-to-end
> argument would suggest that the network not try to divine which flows get
> preferred.
> > And beyond the end-to-end argument, there's a practical problem - since the
> ideal state of a shared link means that it ought to have no local backlog in the
> queue, the information needed to schedule "fairly" isn't in the queue backlog
> itself. If there is only one packet, what's to schedule?
> 
> The concept that DRR++ has empirically proved is that flows can be classified into
> two categories - sparse and saturating - very easily by the heuristic that a
> saturating flow's arrival rate exceeds its available delivery rate, and the
> opposite is true for a sparse flow.
> 
> An excessive arrival rate results in a standing queue; with Reno, the excess
> arrival rate after capacity is reached is precisely 1 segment per RTT, very small
> next to modern link capacities. If there is no overall standing queue, then by
> definition all of the flows passing through are currently sparse. DRR++ (as
> implemented in fq_codel and Cake) ensures that all sparse traffic is processed
> with minimum delay and no AQM activity, while saturating traffic is metered out
> fairly and given appropriate AQM signals.
> 
> > In fact, what the ideal queueing discipline would do is send signals to the
> endpoints that provide information as to what each flow's appropriate share is,
> and/or how far its current share is from what's fair.
> 
> The definition of which flows are sparse and which are saturating shifts
> dynamically in response to endpoint behaviour.
> 
> Today's TCP traffic exhibits the classic sawtooth behaviour - which has a
> different shape and period with CUBIC than Reno, but is fundamentally similar. 
> The sender probes capacity by increasing send rate until a congestion signal is
> fed back to it, at which point it drops back sharply. With efficient AQM action,
> a TCP flow will therefore spend most of its time "sparse" and using less than the
> available path capacity, with occasional excursions into "saturating" territory
> which are fairly promptly terminated by AQM signals.
> 
> So TCP does *not* have a definable "average rate". It grows to fill available
> capacity, just like the number of cars on a motorway network.
> 
> The recent work on high-fidelity ECN (including SCE) aims to eliminate the
> sawtooth, so that dropping out of "saturating" mode is done faster and by only a
> small margin, wasting less capacity and reducing peak delays - very close to ideal
> control as you describe. But it's still necessary to avoid giving these signals
> unnecessarily to "sparse" flows, which would cause them to back off and thus waste
> capacity, but only to "saturating" flows that have just begun to build a queue. 
> And it's also necessary to protect these well-behaved "modern" flows from "legacy"
> endpoint behaviour, and vice versa. DRR++ does that very naturally.
> 
> It is important however to define "flow" correctly relative to the measure of
> fairness you want to achieve. Traditionally the unique 5-tuple is used to define
> "flow", but this means applications can game the system by opening multiple flows.
> For an ISP a better definition might be that each subscriber's traffic is one
> "flow". Or there is a tweak to DRR++ which allows a two-layer fairness
> definition, implemented successfully in Cake.
> 
> > So the end-to-end approach would suggest moving most of the scheduling back
> to the endpoints of each flow, with the role of the routers being to extract
> information about the competing flows that are congesting the network, and
> forwarding those signals (via drops or marking) to the endpoints. That's because,
> in the end-to-end argument that applies here - the router cannot do the entire
> function of managing congestion or priority.
> 
> It must be remembered that congestion signals require one RTT to circulate from
> the bottleneck, via the receiver, back to the sender, and their effects to then be
> felt at the bottleneck. That's typically a much longer response time (say 100ms
> for a general Internet path) than can be achieved by packet scheduling
> (sub-millisecond for a 20Mbps link), and therefore effects only a looser control
> (by fundamental control theory). Both mechanisms are useful and complement each
> other.
> 
> My personal interpretation of the end-to-end principle is that endpoints generally
> do not, cannot, and *should not* be aware of the topology of the network between
> them, nor of any other traffic that might be sharing that network. The network
> itself takes care of those details, and may send standardised control-feedback
> signals to the endpoints to inform them about actions they need to take. These
> currently take the form of ICMP error packets and the ECN field, the latter
> substituted by packet drops on Not-ECT flows.
> 
> - Jonathan Morton

---

From: "David P. Reed" <[email-masked]>
Date: Mon, 24 Jun 2019 15:50:45 -0400 (EDT)

Please!
On Monday, June 24, 2019 3:31pm, "Jonathan Morton" <[email-masked]> said:

> particular "queueing discipline" in a particular kind of a router - it would
> respond differently (and does!) if the router were to drop packets randomly on a
> Poisson basis, for example. No sawtooth at all.
> 
> I challenge you to show me a Reno or CUBIC based connection's cwnd evolution that
> *doesn't* resemble a sawtooth, regardless of the congestion signalling employed. 
> And I will show you that it either has severe underutilisation of the link, or is
> using SCE signals. The sawtooth is characteristic of the AIMD congestion control
> algorithm.
 
Of course AIMD responds with a sawtooth to a router dropping occasional packets as congestion signalling. Are you trying to pretend I'm an idiot? And various kinds of SACK cause non-sawtooth responses.

TCP sources don't work like that in practice.

---

From: "David P. Reed" <[email-masked]>
Date: Tue, 25 Jun 2019 17:05:37 -0400 (EDT)

So, go for it, then. I wish you well.
 
On Monday, June 24, 2019 4:14pm, "Jonathan Morton" <[email-masked]> said:

> > My overall point here is that you seem to live in a world of academic-like
> purity - all TCP connections are essentially huge file transfers, where there are
> no delays on production or consumption of packets at the endpoint, there is no
> multiplexing or scheduling of processes in the endpoint operating systems, etc.
> 
> On the contrary, I've found that per-flow queuing algorithms like DRR++ cope
> naturally and very nicely with all sorts of deviations from the ideal, including
> for example the wild variations in goodput and RTT associated with wifi links.
> 
> These are exactly the kinds of complication that I imagine - and not merely in the
> abstract but through observation - that a pure end-to-end approach would have
> great difficulty in accommodating elegantly.
> 
> - Jonathan Morton
> 
>

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 26 Jun 2019 14:48:32 +0200

> On Jun 23, 2019, at 00:09, David P. Reed <[email-masked]> wrote:
> 
> [...]
>  
[...]

Excuse my stupidity, but the "only one single packet" case is the theoretical limiting case, no? 
Because even on a link not running at capacity this effectively requires a mechanism to "synchronize" all senders (whose packets traverse the hop we are looking at), as no other packet is allowed to reach the hop unless the "current" one has been passed to the PHY otherwise we transiently queue 2 packets (I note that this rationale should hold for any small N). The more packets per second a hop handles the less likely it will be to avoid  for any newcomer to run into an already existing packet(s), that is to transiently grow the queue.
Not having a CS background, I fail to see how this required synchronized state can exist outside of a few steady state configurations where things change slowly enough that the  seemingly required synchronization can actually happen (given that the feedback loop e.g. through ACKs, seems somewhat jittery). Since packets never know which path they take and which hop is going to be critical there seems to be no a priori way to synchronize all senders, heck I fail to see whether it would be possible at all to guarantee synchronized behavior on more than one hop (unless all hops are extremely uniform).
I happen to believe that L4S suffers from the same conceptual issue (plus overly generic promises, from the RITE website:
"We are so used to the unpredictability of queuing delay, we don’t know how good the Internet would feel without it. The RITE project has developed simple technology to make queuing delay a thing of the past—not just for a select few apps, but for all." this seems missing a conditions apply statement)

Best Regards
	Sebastian

---

From: "David P. Reed" <[email-masked]>
Date: Wed, 26 Jun 2019 12:31:46 -0400 (EDT)

It's the limiting case, but also the optimal state given "perfect knowledge".
 
Yes, it requires that the source-destination pairs sharing the link in question coordinate their packet admission times so they don't "collide" at the link. Ideally the next packet would arrive during the previous packet's transmission, so it is ready-to-go when that packet's transmission ends.
 
Such exquisite coordination is feasible when future behavior by source and destination at the interface is known, which requires an Oracle.
That's the same kind of condition most information theoretic and queueing theoretic optimality requires.
 
But this is worth keeping in mind as the overall joint goal of all users.
 
In particular,  "link utilization" isn't a user goal at all. The link is there and is being paid for whether it is used or not (looking from the network structure as a whole). Its capacity exists to move packets out of the way. An ideal link satisfies the requirement that it never creates a queue because of anything other than imperfect coordination of the end-to-end flows mapped onto it. That's why the router should not be measured by "link utilization" anymore than a tunnel in a city during commuting hours should be measured by cars moved per hour. Clearly a tunnel can be VERY congested and moving many cars if they are attached to each other bumper to bumper - the latency through the tunnel would then be huge. If the cars were tipped on their ends and stacked, even more throughput would be achieved through the tunnel, and the delay of rotating them and packing them would add even more delay.
 
The idea that "link utilization" of 100% must be achieved is why we got bufferbloat designed into routers. It's a worm's eye perspective. To this day, Arista Networks brags about how its bufferbloated feature design optimizes switch utilization ([ https://packetpushers.net/aristas-big-buffer-b-s/ ]( https://packetpushers.net/aristas-big-buffer-b-s/ )). And it selects benchmarks to "prove" it. Andy Bechtolsheim apparently is such a big name that he can sell defective gear at a premium price, letting the datacenters who buy it discover that those switches get "clogged up" by TCP traffic when they are the "bottleneck link". Fortunately, they are fast, so they are less frequently the bottleneck in datacenter daily use.
 
In trying to understand what is going on with congestion signalling, any buffering at the entry to the link should be due only to imperfect information being fed back to the endpoints generating traffic. Because a misbehaving endpoint generates Denial of Service for all other users.
 
Priority mechanisms focused on protecting high-paying users from low-paying ones don't help much - they only help at overloaded states of the network. Which isn't to say that priority does nothing - it's just that stable assignment of a sharing level to priority levels isn't easy.  (See Paris Metro Pricing, where there are only two classes, and the problem of deciding how to manage the access to the "first class" section - the idea that 15 classes with different metrics can be handled simply and interoperably between differently managed autonomous systems seems to be an incredibly impractical goal).
Even in the priority case, buffering is NOT a desirable end user thing.
 
My personal view is that the manager of a network needs to configure the network so that no link ever gets overloaded, if possible. The response to overload should be to tell the relevant flows to all slow down (not just one, because if there are 100 flows that start up roughly at the same time, causing MD on one does very little. This is an example of something where per-flow stuff in the router actually makes the router helpful in the large scheme of things. Maybe all flows should be equally informed, as flows. Which means the router needs to know how to signal multiple flows, while not just hammering all the packets of a single flow.  This case is very real, but not as frequently on the client side as on the "server side" in "load balancers" and such like.
 
My point here is simple:
 
1) the endpoints tell the routers what flows are going through a link already. That's just the address information. So that information can be used for fairness pretty well, especially if short term memory (a bloom filter, perhaps) can track a sufficiently large number of flows.
 
2) The per-flow decisions related to congestion control within a flow are necessarily end-to-end in nature - the router can only tell the ends what is going on, but the ends (together - their admissions rates and consumption rates are coupled to the use being made) must be informed and decide. The congestion management must combine information about the source and the destination future behavior (even if it is just taking recent history and projecting it as an estimate of future behavior at source and destination). Which is why it is quite natural to have routers signal the destination, which then signals the source, which changes its behavior.
 
3) there are definitely other ways to improve latency for IP and protocols built on top of it  - routing some flows over different paths under congestion is one. call the per-flow routing. Another is scattering a flow over several paths (but that seems problematic for today's TcP which assumes all packets take the same path).
 
4) A different, but very coupled view of IP is that any application-relevant buffering shoujld be driven into the endpoints - at the source, buffering is useful to deal with variability in the rate of production of data to be sent. At the destination, buffering is useful to minimize jitter, matching to the consumption behavior of the application.  But these buffers should not be pushed into the network where they cause congestion for other flows sharing resources.
So buffering in the network should ONLY deal with the uncertainty in resource competition.
 
This tripartite breakdown of buffering is protocol independent. It applies to TCP, NTP, RTP, QUIC/UDP, ...  It's what we (that is me) had in mind when we split UDP out of TCP, allowing UDP based protocols to manage source and destination buffering in the application for all the things we thought UDP would be used for - packet speech, computer-computer remote procedure calls (what would be QUIC today), SATNET/interplanetary Internet connections , ...).
 
Sadly, in the many years since the late 1970's the tendency to think file transfers between infinite speed storage devices over TCP are the only relevant use of the Internet has penetrated the router design community. I can't seem to get anyone to recognize how far we are from that.  No one runs benchmarks for such behavior, no one even measures anything other than the "hot rod" maximum throughput cases.
 
And many egos seem to think that working on the hot rod cases is going to make their career or sell product.  (e.g. the sad case of Arista).

On Wednesday, June 26, 2019 8:48am, "Sebastian Moeller" <[email-masked]> said:

> Because even on a link not running at capacity this effectively requires a
> mechanism to "synchronize" all senders (whose packets traverse the hop we are
> looking at), as no other packet is allowed to reach the hop unless the "current"
> one has been passed to the PHY otherwise we transiently queue 2 packets (I note
> that this rationale should hold for any small N). The more packets per second a
> hop handles the less likely it will be to avoid for any newcomer to run into an
> already existing packet(s), that is to transiently grow the queue.
> Not having a CS background, I fail to see how this required synchronized state can
> exist outside of a few steady state configurations where things change slowly
> enough that the seemingly required synchronization can actually happen (given
> that the feedback loop e.g. through ACKs, seems somewhat jittery). Since packets
> never know which path they take and which hop is going to be critical there seems
> to be no a priori way to synchronize all senders, heck I fail to see whether it
> would be possible at all to guarantee synchronized behavior on more than one hop
> (unless all hops are extremely uniform).
> "We are so used to the unpredictability of queuing delay, we don’t know how
> good the Internet would feel without it. The RITE project has developed simple
> technology to make queuing delay a thing of the past—not just for a select
> few apps, but for all." this seems missing a conditions apply statement)
> 
> Best Regards
> Sebastian

---

From: "David P. Reed" <[email-masked]>
Date: Wed, 26 Jun 2019 12:53:02 -0400 (EDT)

A further minor thought, maybe one that needs not be said:
 
Flows aren't "connections". Routers are not involved in connection state management, which is purely part of the end to end protocol. Anything about "connections" that a router might need to know to handle a packet should be packaged into the IP header of each packet in a standard form. Routers can "store" this information associated with the source, destination pair if they want, for a short time, subject to well understood semantics when they run out of storage. This fits into an end-to-end argument as an optiimization of a kind, as long as the function of such information is very narrowly and generally defined to benefit all users of IP-based protocols.
 
For example, remembering the last time a packet of a particular flow was received after forwarding it, for a short time, to calculate fairness, that seems like a very useful idea, as long as forgetting the last time of receipt is not unfair.
 
This use of the flow's IP headers to carry info into router queueing and routing decisions is analogous to the "Fate Sharing" principle of protocol design that DDC describes. Instead of having an independent control plane protocol, which has all kinds of problems with synchronization and combinatorial problems of packet loss, "Fate Sharing" of protocol information is very elegant.
On Wednesday, June 26, 2019 12:31pm, "David P. Reed" <[email-masked]> said:

 

 

 
 
My point here is simple:

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 27 Jun 2019 09:49:53 +0200

Hi David,

thanks for your response.

> On Jun 26, 2019, at 18:31, David P. Reed <[email-masked]> wrote:
> 

	Ah, great, I had feared I had missed something.

> In particular,  "link utilization" isn't a user goal at all. The link is there and is being paid for whether it is used or not (looking from the network structure as a whole). Its capacity exists t…

	+1; this is the core of the movement under the "bufferbloat" moniker put latency back into the spot light where it belongs (at least for common inter-active network usage, bulk transfer is a different kettle of fish). Given the relative low rates of common internet access links, running at capacity, while not a primary goal, still becomes common enough to require special treatment to keep latency under load increase under control. Both FQ solutions and L4S offer remedies for that case. (Being a non-expert home-user myself this case also is prominent on my radar, my ISPs backbone and peerings/transits being well managed the access link is the one point where queueing happens, just as you describe).

>  
> The idea that "link utilization" of 100% must be achieved is why we got bufferbloat designed into routers.

	While I do not describe to this view (and actually are trading in "top-speed" to keep latency sane) a considerable fraction of home-users seem obsessed in maxing out their access links and compare achievable rates; whether such behaviour shoud be encouraged is a different question.

> It's a worm's eye perspective. To this day, Arista Networks brags about how its bufferbloated feature design optimizes switch utilization (https://packetpushers.net/aristas-big-buffer-b-s/). And it selects benchmarks to "prove" it. Andy Bechtolsheim apparently is such a big name that he can sell defective gear at a premium price, letting the datacenters who buy it discover that those switches get "clogged up" by TCP traffic when they are the "bottleneck link". Fortunately, they are fast, so they are less frequently the bottleneck in datacenter daily use.
>  

	This is a good point, and one of the reasons, why I conceptually like flow queueing, as that gives the tools to allow to isolate bad actors, "trust, but verify" comes to mind as a principle. I also add that the _only_ currently known  L4S rolll-out target (low latency docsis) actually mandates a mechanism they call "queue protection" which to me looks pretty much like it is a FQ system that carefully tries to not call itself FQ (it monitors the length of flows and if they exceed something pushes them into the RFC3168 queue, which to this layman means it need to separately track the packets for each flow in the common queue to be able to re-direct them).

>  
> Priority mechanisms focused on protecting high-paying users from low-paying ones don't help much - they only help at overloaded states of the network.

	In principle I agree, in practice things get complicated; mixing latency-indifferent capacity-devouring applications like bit-torrent with say VoIP packets (fixed rates, but latency sensitive) over too narrow a link will make it clear that giving the VoIP packet precedence/priority over the bulk-transfer packet is a sane policy (that becomes an issue due to the difficulty of running a narrow link below capacity). I am sure you are aware of all of this, I just need to spell it out for my thinking process.

> Which isn't to say that priority does nothing - it's just that stable assignment of a sharing level to priority levels isn't easy.  (See Paris Metro Pricing, where there are only two classes, and the problem of deciding how to manage the access to the "first class" section - the idea that 15 classes with different metrics can be handled simply and interoperably between differently managed autonomous systems seems to be an incredibly impractical goal).

	+1; any prioritization scheme should be extremely simple so that an end-user can make predictions about its behavior easily. Also IMHO 3 classes of latency behaviour will go a long way, "normal", "don-t care", "important" should be enough (L4S IMHO only offers "important" and normal, so does not offer to easily down-grade say bulk background transfers like bit-torrent (which is going to be an issue with bit-torrent triggering on ~100 induced latency increase with L4S's RFC3168 queue using a PIE offspring to keep induced latency << 100ms, but I digress).)

> Even in the priority case, buffering is NOT a desirable end user thing.

	+1; IMHO again a reason for fq, misbehaving flows will not spoil the fun for everybody else.

>  
> My personal view is that the manager of a network needs to configure the network so that no link ever gets overloaded, if possible. The response to overload should be to tell the relevant flows to all slow down (not just one, because if there are 100 flows that start up roughly at the same time, causing MD on one does very little.
> This is an example of something where per-flow stuff in the router actually makes the router helpful in the large scheme of things. Maybe all flows should be equally informed, as flows. Which means the router needs to know how to signal multiple flows, while not just hammering all the packets of a single flow.  This case is very real, but not as frequently on the client side as on the "server side" in "load balancers" and such like.
>  
> My point here is simple:
>  

	In an ideal world the router would also signal the sender as that will at least half the time it takes for the congestion information to reach the most relevant party; but as I understand this is a) not generally possible and b) prone to abuses.

> 3) there are definitely other ways to improve latency for IP and protocols built on top of it  - routing some flows over different paths under congestion is one. call the per-flow routing. Another…

	This is about re-ordering, no? 

> So buffering in the network should ONLY deal with the uncertainty in resource competition.

	This, at least in my understanding, is one of the underlaying ideas of the L4S approach, so how is your take on how well L4S archives that goal?

> This tripartite breakdown of buffering is protocol independent. It applies to TCP, NTP, RTP, QUIC/UDP, ...  It's what we (that is me) had in mind when we split UDP out of TCP, allowing UDP based pr…

	Like many great insights that look obvious in retro-spect, I would guess that might have been controversial at its time?

> Sadly, in the many years since the late 1970's the tendency to think file transfers between infinite speed storage devices over TCP are the only relevant use of the Internet has penetrated the rout…

	I would guess, that this obsession might be market-driven, as long as customers only look at the top-speed numbers, increasing this number will be the priority.

Again thanks for your insights

	Sebastian

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 27 Jun 2019 09:53:39 +0200

Hi,

Am 26.06.19 um 18:31 schrieb David P. Reed:

> The idea that "link utilization" of 100% must be achieved is why we got
> bufferbloat designed into routers. It's a worm's eye perspective. To

You are right, but IMHO it is even worse, because it is an
artefact of the particular loss-based TCP congestion control
as it was designed and the correspondingly derived "BDP buffer size"
rule. The loss-based AIMD congestion control is able to keep up the
utilization then, because it uses the systematically built standing
queue during its back-off after a packet loss. That essentially keeps
the sending rate at the same level. However, now we know that avoiding
queuing delay is important, we can also design better congestion
controls that do not fill the available buffer capacity up to exhaustion
and that do not require standing queues to keep up link utilization.

However, as you also wrote, the positive and negative effects of
existing buffers depend a lot on the particular traffic pattern and this
has also changed much during the last decades. So I think that revising
the "buffer size" discussion could be useful...

Aside from that I find the SCE proposal very useful, because it allows
to provide an additional level of congestion signalling that could
be used by various congestion control schemes.

Regards,
 Roland

---

From: Sebastian Moeller <[email-masked]>
Date: Thu, 27 Jun 2019 09:54:21 +0200

Hi David,

> On Jun 26, 2019, at 18:53, David P. Reed <[email-masked]> wrote:
> 
> Flows aren't "connections". Routers are not involved in connection state management, which is purely part of the end to end protocol. Anything about "connections" that a router might need to know to handle a packet should be packaged into the IP header of each packet in a standard form.

	I read this, that your are not opposed to using IP packet data to convey information to intermediate routers then? In a way (and please correct me if this is wrong /too simplistic), L4S intends to use the ECT(1) codepoint for enspoints to signal to router's their behavior towards CE congestion signals (reduce window/rate by 50% versus a smaller step down).

> Routers can "store" this information associated with the source, destination pair if they want, for a short time, subject to well understood semantics when they run out of storage. This fits into an end-to-end argument as an optiimization of a kind, as long as the function of such information is very narrowly and generally defined to benefit all users of IP-based protocols.

	Okay, that I read as fq-syatems are not in violation of e2e then.

Best Regards
	Sebastian

> The idea that "link utilization" of 100% must be achieved is why we got bufferbloat designed into routers. It's a worm's eye perspective. To this day, Arista Networks brags about how its bufferbloated feature design optimizes switch utilization (https://packetpushers.net/aristas-big-buffer-b-s/). And it selects benchmarks to "prove" it. Andy Bechtolsheim apparently is such a big name that he can sell defective gear at a premium price, letting the datacenters who buy it discover that those switches get "clogged up" by TCP traffic when they are the "bottleneck link". Fortunately, they are fast, so they are less frequently the bottleneck in datacenter daily use.
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 27 Jun 2019 09:48:57 +0100

> be used by various congestion control schemes.
I know from the work on using ECT(0) with RFC8511, that you can already 
get a lot of benefit from treating ECN as a first indication of 
congestion and loss as a much more serious signal. I may have missed 
something recently. I recall attempts to do 3-level marking of ECNin the 
past,but  when I last looked through the litterature I did not find any 
lasting deployment resulted or modern research  (please send pointers if 
I overlooked something).

I am therefore  rather skeptical that research on SCE is ready  I'd be 
curious to see what extra value is gained from the SCE approach.

Gorry

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 28 Jun 2019 08:33:13 +1200

On 27-Jun-19 19:49, Sebastian Moeller wrote:
...
> a considerable fraction of home-users seem obsessed in maxing out their access links and compare achievable rates; whether such behaviour shoud be encouraged is a different question.

I think this is encouraged by, or is even a direct result of, so called "speed tests" for use by consumers (such as https://www.speedtest.net/), and the way connectivity "speed" has been used as a marketing tool. At least where I live, "speed" is the main marketing tool for switching users to fibre instead of copper. No doubt it will be used as the main marketing tool for 5G too.

It's almost as if those marketing people don't understand queueing theory.

    Brian

---

From: "David P. Reed" <[email-masked]>
Date: Thu, 27 Jun 2019 17:31:01 -0400 (EDT)

It's even worse. The FCC got focused on max speeds back in the day as its only way to think about Internet Access service. And I was serving on the FCC Technological Advisory Committee and also in its Spectrum Policy Task Force, then later involved in the rather confused discussions of Network Neutrality, where again "speed" in the "up-to" sense was the sole framing of the discussion.
 
Because it was mostly lawyers and lobbyists (not network engineers), this focus on max speed as the sole measure of quality ended up with a huge distortion of the discussion, strongly encouraged by the lobbyists who love confusion.
 
That said, max speed plays a role at all time scales in minimizing response time, but queuing delay has no constituency, even though its impact is FAR worse in real situations.
 
If the FCC and regulators (or even the DoD communications management layers)  ever start talking about queueing delay in shared network services, I will die of shock.
 
But we did have one HUGE temporary success. The speed test at DSL Reports measures lag under load, and calls it bufferbloat, and gives a reasonably scaled score.
 
When I talk to people who are interested in quality of Internet service, I point them at DSL Reports' speed test.
 
That is a big win.  However, marketers of Internet access services don't compete to get good scores at DSL Reports. Even "business" providers provide crappy scores.
 
Comcast Business in the South Bay does very poorly on bufferbloat for its high speed business services, for example. This is based on my measurements at my company.  I know some very high executives there, and Comcast is the only real game in town for us, so I tried to get the folks in Philadelphia to talk to the local managers. Turns out the local managers just refused to listen to the headquarters execs. They saw no monetary benefit in fixing anything (going from DOCSIS 2 to DOCSIS 3.1 which had already been on the market for several years would have fixed it, probably).
 
On Thursday, June 27, 2019 4:33pm, "Brian E Carpenter" <[email-masked]> said:

> > a considerable fraction of home-users seem obsessed in maxing out their
> access links and compare achievable rates; whether such behaviour shoud be
> encouraged is a different question.
> 
> I think this is encouraged by, or is even a direct result of, so called "speed
> tests" for use by consumers (such as https://www.speedtest.net/), and the way
> connectivity "speed" has been used as a marketing tool. At least where I live,
> "speed" is the main marketing tool for switching users to fibre instead of copper.
> No doubt it will be used as the main marketing tool for 5G too.
>

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Fri, 28 Jun 2019 09:49:39 +0200

"David P. Reed" <[email-masked]> writes:

> It's even worse. The FCC got focused on max speeds back in the day as
> its only way to think about Internet Access service. And I was serving
> on the FCC Technological Advisory Committee and also in its Spectrum
> Policy Task Force, then later involved in the rather confused
> discussions of Network Neutrality, where again "speed" in the "up-to"
> sense was the sole framing of the discussion.
>  
> Because it was mostly lawyers and lobbyists (not network engineers),
> this focus on max speed as the sole measure of quality ended up with a
> huge distortion of the discussion, strongly encouraged by the
> lobbyists who love confusion.
>  
> That said, max speed plays a role at all time scales in minimizing
> response time, but queuing delay has no constituency, even though its
> impact is FAR worse in real situations.
>  
> If the FCC and regulators (or even the DoD communications management
> layers) ever start talking about queueing delay in shared network
> services, I will die of shock.
>  
> But we did have one HUGE temporary success. The speed test at DSL
> Reports measures lag under load, and calls it bufferbloat, and gives a
> reasonably scaled score.

The Netflix test at fast.com does as well now (although it's under the
"more info" button, so not as visible by default).

-Toke