Subject: [OPSAWG] tsw/ospa: Internet phone-in-QoS ?

From: Toerless Eckert <[email-masked]>
Date: Tue, 14 Apr 2020 21:46:41 +0200

I have been somewhat following how in the face of COVID-19,
the appropriate way to manage congestion control in the Internet 
seem to be heads of countries reaching out to the one large content provider
they know (Netflix) and ask him to reduce bandwidth pressure on the
Internet. Of course, heads of states with differently aged children
would know that Disney+ or Apple might be other relevant streaming
providers to reach out to, but alas, we have forgotten to elect 
those heads of states on such key criteria.

That was of course tongue in cheek of course, but i was somewhat surprised
that nobody took up the opportunity so far to ask something like "how are we
 doing on Net Neutrality" ?, or "what the heck would we actually want it to be" ?

I can see a lot of operational short term workarounds to
approximate solutions less silly than phoning CEOs of random companies,
but it really strikes me as highly strange that events like the
ones we're in right now should not have us re-think to what extend
our current presumed strategy is sufficient beyond workarounds:

- Best effort is all we need
- The Internet MUST NOT be able to further distinguish traffic.
- Its just a matter of more money to eliminate inacceptable congestion.

Is this what we want to proliferate ?
Am i the only one wondering about this  ?

Obnoxious as i am i think i should quit my Netflix account and instead
subscribe to other streaming services for the time being, small enough
 not to be subject to phone-in-QoS. Hmm... HBO or Showtime ??

Or is the solution really: We MUST only have few big quasi-monopolies, because
our heads of state only have that much time to control Internet QoS through
their phone calls ? Aka: the evolution of the Internet to where we
are today is really a great thing, especially to manage QoS the way we can ?

Cheers
    Toerless

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 14 Apr 2020 23:44:10 +0300

> On 14 Apr, 2020, at 10:46 pm, Toerless Eckert <[email-masked]> wrote:
> 
> our current presumed strategy is sufficient…

I'm pretty sure that if ISPs implemented congestion control measures correctly at layer 3, there would be no need to take traffic-source-specific actions so high up the stack (beyond the machine layers) to merely ensure that backhaul networks and peering arrangements are not flooded into oblivion.  I'm talking about simple, well-understood measures such as:

1: Implement AQM at every potential bottleneck, to limit effect of excess traffic on latency & reliability.  In this context, even WRED without ECN is better than nothing, but doing better would be nice.

2: Share bottleneck capacity fairly between subscribers, so that one household's heavy traffic doesn't unduly impact the service to other households.  Want to download three Steam games, five Netflix streams and a 500-peer BitTorrent swarm all at once?  Go right ahead, but your next-door neighbour will get just as much bandwidth for his videoconference call about keeping a factory production line going.

If those two measures were widely implemented, there would be no need for data caps, and streaming services' existing quality adjustment algorithms would automatically adjust to match available capacity.

There are already published RFCs detailing all the necessary technology to make this work.  It's already available in many end-hosts and CPE boxes.  It's just not widely deployed and switched on in ISPs' networks, where it can do the most good.  But there *are* a few ISPs who have done it, and thus might be good sources of practical expertise on the subject, if only the industry at large was willing to listen.

 - Jonathan Morton

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 15 Apr 2020 02:18:09 +0200

Thanks, Jonathan,

a) I have a hard time believing that most bigger SP access
region paths (outside the home) do not at least have WRED (or better).
(aka: paths between national streaming clous services and 90%
 of subscribers).

b) Even within an ISP, i can easily see how the ISP would
(in the absence of other policies) like to give subscribers who
have bought more bandwidth also a value add for that bandwidth.

Same bandwidth under cross-subscriber congestion may
not be supporting the business goals of the ISP best.

c)  Even in the absence of this "not every subscriber is equal"
problem, it is not clear to me that bandwidth is used in
accordance to e.g.: publiic policies if all subscribers get
the same share - whether they use entertainment or are supporting 
critical functions".  Medical staff working from home giving video
sessions to patients for example.

We do have public policies applied to other infrastructures
that are fairly orthogonal to their funding. Emergency
vehicles on roads, or commuter lanes for example. We already
had similar problems to corona during the fires and fire
departments in california (oh sure, why not let fire dpartments
pay all-year ridiculous ISP prices when they need the
traffic volume only in regional emergencies...).

d) All the "simple" policies ultimately will AT BEST work in
the IBM-Server/Terminal, oops: cloud-service vs subscriber
application model, rwhee you try to come up with some
(IMHO impossible without policy distinction) recognition
of subscribers, but no limits on servers. Of course, if
you would recognize "entertainment" vs. "critical infrastructure"
servers you could have another set of simple workaround.
But what if you want to have more resilient applications
based on a model of peers. Where you do not need to rely
on specific designated services in the cloud but build
your eg.: resilient ad-hoc conference app solely between
subscriber IP addresses ...

Cheers
    toerless

(else, why would a subscriber buy that more bandwidr

On Tue, Apr 14, 2020 at 11:44:10PM +0300, Jonathan Morton wrote:
> > our current presumed strategy is sufficient???
>

---

From: "Rodney W. Grimes" <[email-masked]>
Date: Tue, 14 Apr 2020 23:27:04 -0700 (PDT)

> (aka: paths between national streaming clous services and 90%

The model there is "through lambda" at it.  They do not
believe in either AQM or ECN, they truely never expect
the network at that level to experience congestion, and
hence to date pretty much it is "drop" as a signal.

When trying to "sell" ECN concepts to network operators
at this level I usually get told "our network never
experiences congestion because we overprovision."

You may find it hard to believe, but it is a reality.

But, given light of the current situation perhaps it is time
to go back and talk to some of those operators and see if
they now understand that an event may occur that overwhelms
the overprovision and that ECN and AQM's are infact something
worth looking at getting deployed.

> traffic volume only in regional emergencies...).

Another business model I see on "congested" subscriber issues
is "its good when my customers link congests and has problems,
they come to me to buy more bandwidth, I really do not care
to solve this problem for them."

That from the mouth of a continent sized telco/internet
provider.

-- 
Rod Grimes                                                 [email-masked]

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 15 Apr 2020 09:44:46 +0200

Hi Toerless,

> On Apr 14, 2020, at 21:46, Toerless Eckert <[email-masked]> wrote:
> 
> Internet.

	A misconception, it was the EU's commissioner for the internal market Thierry Breton (https://ec.europa.eu/commission/commissioners/2019-2024/breton_en) who technically is not a head of a country. And the twitter messaging with Netflix and Co. was the "PR leg" of a joint statement of the EU commission and the Body of European Regulators for Electronic Communications (BEREC)(https://berec.europa.eu/eng/document_register/subject_matter/berec/others/9236-joint-statement-from-the-commission-and-the-body-of-european-regulators-for-electronic-communications-berec-on-coping-with-the-increased-demand-for-network-connectivity-due-to-the-covid-19-pandemic) aimed at european ISPs giving them guidance how to deal with potential congestion due to shifted traffic pattern caused by the mitigation approaches to limit COVID-19 spread. Reading that makes it clear that the EU accepted that in the current situation, the goal is not to let the consumers feel the quality differences between different ISPs but give ISP exceptional temporary permits to shed network load in line with the EU's net neutrality regulation.
In a sense the EU was presenting the thumb-screws and laid out under which conditions they might be used. Most of the streaming media companies read the signs correctly and decided to pro-actively reduce their bandwidth use on their own terms, instead of risking indifferent packet dropping by ISPs...
	IMHO both sides, the political/regulatory and the private enterprises, acted quite rationally and efficient. The fact that there is currently no wide-spread network overload does not make it a bad idea to pro-actively take measures to reduce known bandwidth hogs... Honestly, that is what I expect from halfway decent policy makers, address issues before the house is on fire (but hey, I also consider how the Y2K situation resolved, not to be a sign of undeserved alarmism, but rather successful mitigation after realization of the importance of the issue).

> Of course, heads of states with differently aged children

	Nope, at the time of the twitter message disney+ had not started serving Europe yet, Disney came around promising to pro-actively reduce their streaming bandwidth by 25% even before starting.

> doing on Net Neutrality" ?, or "what the heck would we actually want it to be" ?

	Sorry, the joint statement explicitly tackled the issue of how to shed overload in compliance with the EU's net neutrality rules and regulations. Now, you might not agree with the taken decision, but that is different from the questions you posed, no?

> approximate solutions less silly than phoning CEOs of random companies,

	Again, that was the PR side, the "meat" is/was in the clarification to European ISPs how to shed load in compliance with the EU's NN rules.

> - Best effort is all we need

	That is not true today, ISPs and CDN's as far as I know already use SLAs to prioritize e.g. VoIP traffic at peering points, so we already have more than best effort for the few things where it matters?

> - The Internet MUST NOT be able to further distinguish traffic.

	IMHO that is a sane approach because traffic carries very little reliable information to meaningfully distinguish traffic. Sure in an ideal world an overloaded hop would divine which packets are least important and target queuing/dropping to those while keeping important traffic fast. Alas, the required oracle seems to be amiss... Also note that NN rules where pushed, because ISPs where playing games with treating traffic differentially with the sole goal of extracting more revenue/income out of the same amount of traffic, is that really what we want?

> - Its just a matter of more money to eliminate inacceptable congestion.

	That hinges on the definition of "inacceptable", but sure money will allow to increase "bandwidth" and hence reduce congestion for the same amount of traffic (or move the point of congestion to some other place).

> Is this what we want to proliferate ?

	What alternative do you propose instead?

> not to be subject to phone-in-QoS. Hmm... HBO or Showtime ??

	Mmh, Showtime only serves the US, and HBO only serves parts of Europe, but assuming you get their content streamed to a receiver in Europe, according to EU commission and BEREC that data could still be subject to throttling (I guess you would need a VPN and hence in practice would avoid being limited).

> their phone calls ?

	No, we have big quasi-monopolies because economists and so-inclined politicians for decades have chiseled away at antitrust regulations... (I find it delicately ironic that the same people who preach "free market" typically are the first that welcome approaches to side-step the market. It is the same "experts" that simultaneously only see limits to what is okay for a company to do by regulations and the law AND work hard to get rid of regulations and laws, but I digress). 

> Aka: the evolution of the Internet to where we

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 15 Apr 2020 11:15:13 +0200

Hi Jonathan,

> On Apr 14, 2020, at 22:44, Jonathan Morton <[email-masked]> wrote:
> 

	I disagree a bit, we are witnessing a policy decision by the EU that assumed purely recreational streaming traffic shall take a back-seat to assumed work related "home-office" traffic IFF the network experiences congestion. This is as much about how to distribute available bandwidth between different traffic "classes" as it is about avoiding the undesirable side-effects of congestion, just dealing with the second is not sufficient to solve the first.

> 1: Implement AQM at every potential bottleneck, to limit effect of excess traffic on latency & reliability.  In this context, even WRED without ECN is better than nothing, but doing better would be…

	+1

> 2: Share bottleneck capacity fairly between subscribers, so that one household's heavy traffic doesn't unduly impact the service to other households.  Want to download three Steam games, five Netfl…

	+1; and not because equal sharing is optimal in any sense, but because it is the best an intermediate hop can do with its limited information (and even there people complain that per-something queueing is too expensive computationally to do at scale in high speed gear)...
	in a sense fair sharing is the "democracy" of schedulers with Winston Churchill's citation (gently ammended) applicable:
"No one pretends that democracy/FQ is perfect or all-wise. Indeed it has been said that democracy/FQ is the worst form of Government/scheduling except for all those other forms that have been tried from time to time.…"

> If those two measures were widely implemented, there would be no need for data caps, and streaming services' existing quality adjustment algorithms would automatically adjust to match available cap…

	It is argued, that most streamers use adaptive streaming already, so in essence will scale back under load (but that does not work as well as it should, and due to the existence of over-sized and under-managed queues in the internet creates transient latency excursions that make interactive traffic types like vide-conferencing and gaming suffer).

> There are already published RFCs detailing all the necessary technology to make this work.  It's already available in many end-hosts and CPE boxes.  It's just not widely deployed and switched on in…

	+1; but again that is mainly orthogonal to the EU/BEREC action that ruffled a few feathers in the tech community, no?

Best Regards
	Sebastian

---

From: Sebastian Moeller <[email-masked]>
Date: Wed, 15 Apr 2020 11:44:25 +0200

Hi Toerless,

> On Apr 15, 2020, at 02:18, Toerless Eckert <[email-masked]> wrote:
> 
> Thanks, Jonathan,
> 

	Wasn't the issue with RED, that it requires some tuning and nobody knows how to automate the tuning, and that it increases the RTT-dependent throughput inequality if flows of different RTTs share a congested hop?

> not be supporting the business goals of the ISP best.

	Well, currently ISPs have mostly zero control over bandwidth sharing at congested hops, so even equal fair sharing will be an improvement, and will also not require to propagate per-subscriber data to all potentially congestable hops. I would guess the current system where the edge restricts each user to at max his/her subscribed rate will work reasonable enough even when core links and transit points resort to flow- or IP-fairness.  

> sessions to patients for example.

	Well, the challenge for anything "better" than N-tupel-fairness (with different N) requires to propagate per-packet-information to the potentially congested hops. I believe this is done with-in reason using diffserv markings and PHBs, but that does not scale to differentiating "entertainment or [...] critical functions" in a generic fashion.

> vehicles on roads, or commuter lanes for example.

	But there we teach each individual controller/driver to yield to higher priority traffic and we sanction invalid dressing-up as the police...

> We already

	Unfettered "free" market capitalism, or rather old-school Manchester early-stage capitalism hiding behind the fig-leaf of a free market. With an intentional misinterpretation of "free" referring to free of regulation and not free information (only then is a market a resource optimizer), but I digress.

---

From: Jonathan Morton <[email-masked]>
Date: Wed, 15 Apr 2020 18:07:42 +0300

> On 15 Apr, 2020, at 12:44 pm, Sebastian Moeller <[email-masked]> wrote:
> 

That's why I specified WRED, which includes some automation of that tuning.  It's not great, but it's something.  Better algorithms exist, and should be used where available.

It's important to keep the scale of the problem in perspective.  Internet paths have a median of about 80ms RTT, baseline.  Adding a few milliseconds to that is no big deal, and a good AQM can keep the queue down to 5ms already.  An old AQM might only control the queue to a few tens of milliseconds, which is still basically acceptable.  But an uncontrolled FIFO is in the hundreds to even tens of thousands of milliseconds, depending on how cheap memory is for the vendor - and above about 2000ms, DNS just stops working.

> Well, currently ISPs have mostly zero control over bandwidth sharing at congested hops, so even equal fair sharing will be an improvement…

Correct for wireline, at least.  I've noticed my local 4G tower has low latency even when heavily congested, though, provided I keep my traffic shaped to below the share of airtime available to me.  So some ISPs manage to share bandwidth at bottlenecks, even if they don't implement AQM.

> Well, the challenge for anything "better" than N-tupel-fairness (with different N) requires to propagate per-packet-information to the potentially congested hops. I believe this is done with-in rea…

I think the important point here is that reliably defining whose traffic is "more equal" than the rest - in the Animal Farm sense - is a Hard Problem, one that has been tackled many times over the years without any significant success.  It's also one that we don't even *need* to solve, as long as the "fair share" is sufficient to let the critical applications work.  In the example I gave, that is probably true.

It also has the benefit of keeping the masses entertained indoors, which in the current crisis is the critical factor for solving the wider problem sooner and with fewer casualties.

Diffserv and its ilk may then be useful for improving the utility of a *single* subscriber's service, working within that fair-share.  The important distinction is, that subscriber has the right to decide which of his own traffic is more important to himself.  He does not have the right to decide that his traffic is more important than his neighbour's.

 - Jonathan Morton

---

From: Kyle Rose <[email-masked]>
Date: Wed, 15 Apr 2020 12:29:03 -0400

On Wed, Apr 15, 2020 at 11:08 AM Jonathan Morton <[email-masked]>
wrote:

> I think the important point here is that reliably defining whose traffic
> is "more equal" than the rest - in the Animal Farm sense - is a Hard
> Problem, one that has been tackled many times over the years without any
> significant success.  It's also one that we don't even *need* to solve, as
> long as the "fair share" is sufficient to let the critical applications
> work.
>

Agreed. ISTM that this is the only reasonable way to address
underprovisioned links.

If a link is underprovisioned temporarily:
 * Small flows are not the problem, so we can give them low latency with no
penalty under any circumstance. FQ does this without overallocating
(unnecessarily reserving) capacity to these flows.
 * Large flows that are congestion-controlled and rapidly capacity-seeking
will always try to fill up the pipe, so the best you can do in the short
term is to divide some portion of the capacity equally among them in a
value-free manner. Low latency is not generally required for these (in
fact, Reno et al. preclude it), so FQ seems fine.
 * Large flows that are either completely unresponsive or that slowly
respond to congestion (e.g., clients dropping from 4K to HD on sustained
buffer underruns) are the flows that really want to be low latency but have
to suffer in the case of a heavily constrained pipe because not doing so
would be unfair to the other classes of flows at the bottleneck. Maybe the
priority should be to reduce rapidly capacity-seeking flows first, but not
enough to create the perverse incentive for all large flows to ignore
congestion signals.

If the link turns out to be perpetually underprovisioned:
 * The network needs to fix that by adding capacity, period. There's no
other solution. While doing this will result in capacity-seeking flows
sending more traffic, there is no link anywhere that would have unbounded
demand at any given time: there is always some other link that would become
new bottleneck. If you're the bottleneck and you aren't an access network,
you're probably doing it wrong.

Kyle