Subject: Re: [arch-d] A Public Option for the Core

From: Brian E Carpenter <[email-masked]>
Date: Wed, 12 Aug 2020 10:53:09 +1200

(Bcc to the IETF list and Cc to the architecture-discuss list)

Thanks for circulating this.

On 12-Aug-20 06:50, Lars Eggert wrote:
> Hi,
> 
> Scott Shenker et al. just presented a pretty thought-provoking "public option" for the Internet's core backbone at SIGCOMM: https://dl.acm.org/doi/abs/10.1145/3387514.3405875

Hmm.

"The technical Internet community has long embraced the notion
that the Internet should be application-neutral; that notion later
became known as network neutrality (a term coined in [57])."

I dispute that assertion. "Network neutrality" is a slippery term,
but certainly there is antipathy between some interpretations of it
and RFC2474 and other QoS technologies.

"We choose instead to initially create the POC’s backbone network out
of a set of leased lines, and use the interconnections to one or more
ISPs as a fallback if the POC’s backbone does not have sufficient
connectivity."

That sounds like a fairly accurate description of the Internet in
about 1995, when ISPs had emerged but the de facto backbone was still
largely non-profit and/or settlement-free. And note that non-profit
transit networks existed then, and actively migrated to a for-profit
model after about 1998. See https://en.wikipedia.org/wiki/EBONE for
an example.

So I think history would likely repeat itself if this proposal was
adopted.

Maybe it's time to update https://tools.ietf.org/html/draft-carpenter-metrics
Reactions to that draft from the then major transit ISPs were interestingly
negative.

   Brian

> 
> If you scroll down (or go to https://dl.acm.org/doi/abs/10.1145/3387514.3405875#sec-supp), his recorded talk video gives a high-level overview.
> 
> (PDF and video *should* be open access and work for me, but you never know with the ACM...)
> 
> Lars
>

---

From: John Grant <[email-masked]>
Date: Wed, 12 Aug 2020 12:52:19 +0100

> became known as network neutrality (a term coined in [57])."
It isn't application-neutral, though; the service it provides is much 
better suited to applications that don't care about time, so can easily 
tolerate queuing delays and retransmission of lost packets. It's fine 
for fetching web pages, or downloading media files and playing them out 
locally, but the service we get with video conferencing etc can be 
pretty rubbish.

-- 
John Grant
Nine Tiles, Cambridge, England
+44 1223 862599 and +44 1223 511455
http://www.ninetiles.com

---

From: Jared Mauch <[email-masked]>
Date: Wed, 12 Aug 2020 08:08:23 -0400

> On Aug 11, 2020, at 6:53 PM, Brian E Carpenter <[email-masked]> wrote:
> 

Looking at this 20+ years in the future, there’s some interesting dynamics going on, which many of us are interested in for the centralization discussions that are ongoing.

There’s the usual last-mile vs major city dynamics at play when it comes to cost.  Construction is expensive, but the fiber isn’t the expensive bit, it’s the dirt movement that costs quite a bit.

With more traffic traversing private backbones vs public ones, there is also a lack of measurements that are feasible.  You can’t measure from within the Akamai backbone as it’s our private network.  Public networks, you can utilize things like RIPE Atlas or similar, but there are significant limitations to it.  Cellular network measurement either require some applications at scale that run in the background, but the various charging cultures that exist globally around voice/sms/data pose challenges.

Public sources of routing data are generally available from the major backbones (route-views, ripe ris, etc) but not for the private clouds (Microsoft, google, amazon, ovh, ibm/oracle).  

The costs of running these backbones are often much lower than public and allow us to optimize for our business use cases in a way a public network can’t solve for everyone.

This is in part a natural economic progression.  Why cloud computing?  Because power is expensive at my house, so I put data on a VM somewhere else for $5/mo.  If I put it in a colocation facility I may pay up to 2500/mo for colocation plus the server and bandwidth costs, plus then I have to deal with swapping failed disk vs $5/mo as a service.

We don’t often talk about these economic factors that drive the shift, but look at the architecture.  It’s not devoid of economic costs or impacts.  Someone is paying for all that compute at Google/Facebook and power, it’s often not a direct expense we see, like we don’t see the credit card fees that merchants pay.

Why are things like neutral IX not successful?  It requires people to sacrifice and value their time very low.  It’s not a surprise that some of the larger IX points operate their own networks to extend their reach (their own private backbones).  Not everyone can interconnect at the speeds necessary at the neighborhood level, there must be some aggregation and it may not be close.

My personal telecom expenses are relatively fixed over time, but the bits/second tend to go up over time.

I am hoping speeds continue to go up, costs go down and the risk of centralization of the network follows the same risk evaluations that people have done in 2020 with their supply chain risks and how they were often centralized.  Diversity is helpful, but if we are always driving for the lowest cost as a consumer, it will take a few shocks to restore that diversity and supply chain depth and balance the ecosystem.

- jared

---

From: Scott Shenker <[email-masked]>
Date: Wed, 12 Aug 2020 06:19:30 -0700

> On Aug 11, 2020, at 3:53 PM, Brian E Carpenter <[email-masked]> wrote:
> 

We did not mean to imply that the two are notions are identical, and indeed we discuss the no-QoS interpretation of network neutrality later in the paper. However, I think it is undeniable that conceptually the notion of network neutrality is the descendant of application-neutrality, which was the intent of this sentence. I apologize if our writing was unclear.
> So I think history would likely repeat itself if this proposal was

The world of today is very different from what it was in 1995, particularly in terms of the market forces at work. We discuss our reasoning for why our proposal might become a reality later in the paper (in a section called “Are We Crazy?”), and I would be interested in your reaction. Do you think the market forces we identify might lead to a different result now?

More generally, as an academic working on issues related to the Internet, I view it as my mission to identify designs and paradigms that would make for a better Internet, and to explore whether there are plausible paths to such eventualities. We recognize that our proposed change is a long-shot. In fact, almost all attempts to effect large-scale changes in the Internet (whether its design or its structure) are likely to fail.   My responsibility as a researcher is to not let that stop me, and my mission as a member of the Internet community is to try to make such changes happen, despite great odds. I hope that people on this list and elsewhere read this paper in that spirit.

—Scott

> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 12 Aug 2020 15:20:07 +0200

Inline

On Wed, Aug 12, 2020 at 08:08:23AM -0400, Jared Mauch wrote:
> Looking at this 20+ years in the future, there???s some interesting dynamics going on, which many of us are interested in for the centralization discussions that are ongoing.
> 
> There???s the usual last-mile vs major city dynamics at play when it comes to cost.  Construction is expensive, but the fiber isn???t the expensive bit, it???s the dirt movement that costs quite a bit.
> 
> With more traffic traversing private backbones vs public ones, there is also a lack of measurements that are feasible.  You can???t measure from within the Akamai backbone as it???s our private network.  Public networks, you can utilize things like RIPE Atlas or similar, but there are significant limitations to it.  Cellular network measurement either require some applications at scale that run in the background, but the various charging cultures that exist globally around voice/sms/data pose challenges.

At the same time, operators try to make money by selling data about their customers
including amounts of traffic or even location on a mobile network. 

> The costs of running these backbones are often much lower than public and allow us to optimize for our business use cases in a way a public network can???t solve for everyone.

Indeed, in many cases a lot of business including that from Akamai is built on
expoiting value from underlying infrastructure by layering on top of it
and creating a service that sells at a higher margin. Often by market
vertical monopolization financed for years on loss by large investments to
grow and replace competition before the service breaks even.

Aka: Internet technologies have enabled a food chain not unlike the one
we see in biology where arguably the human species is on top. And like the
humans, one can argue that companies on the top of the vertical internet
food chain also destroy the eco system they exploit.

The Internet economy is quite the darwinistic playground. Lets see how
long it will take to realize the need for an evolution beyond that. Like
wee did for most other infrastructures and aspects of economy.

> We don???t often talk about these economic factors that drive the shift, but look at the architecture.  It???s not devoid of economic costs or impacts.  Someone is paying for all that compute at Google/Facebook and power, it???s often not a direct expense we see, like we don???t see the credit card fees that merchants pay.

If the "We" means IETF, i agree, and it would be lovely if we would have more
of an economic discussion track.

> Why are things like neutral IX not successful?  It requires people to sacrifice and value their time very low.  It???s not a surprise that some of the larger IX points operate their own networks to extend their reach (their own private backbones).  Not everyone can interconnect at the speeds necessary at the neighborhood level, there must be some aggregation and it may not be close.

Wondering: 

I am sure you can study how to build roads and highways somewhere. Is there any
public study course for these core aspects of the Internet economy ? Is there
even enough public data to build such course work ?

> I am hoping speeds continue to go up, costs go down and the risk of centralization of the network follows the same risk evaluations that people have done in 2020 with their supply chain risks and h…

I think we already have good steps beyond network darwinism on crucial points
in some countries. Such as the last-mile subscriber ownership model to create
more LMA competition and even diversity. And we see a lot of darwinism to
counter such trends, such as the history of DSL vectoring to further
monopolization.

I am more excited/worried about access than core, and when i am worried about
core it is more about the IMHO complete unkown best balance between transparency
and lawfulness. The old Internet meme "complying to all regulations by
offering services from another country" is still a core unresolved problem
statement, and a POC from a single country doesn't change that.

Toerless

-- 
---
[email-masked]

---

From: Christian Huitema <[email-masked]>
Date: Wed, 12 Aug 2020 08:37:13 -0700

> The world of today is very different from what it was in 1995, particularly in terms of the market forces at work. We discuss our reasoning for why our proposal might become a reality later in the…

The market forces are leading to the "hollowing of the core". Last time
I checked, over 50% of the top Internet servers were served by a dozen
or less top ASes, belonging to large CDN or large content hosters, both
of which may well be providing their edge points and delivery networks.
Your paper does not address that.

It seems that you are very concerned with market players using access to
"the core" as a choke point to enforce their own version of "pay to
play". This has been an issue a few years ago, but I am not sure it is
an issue now. What I see instead is local ISP making arrangements like
"zero rating" with large content providers. This is a play with local
access bandwidth, not with core transit bandwidth.

We also have evidence of local ISP prioritizing access to Facebook,
YouTube or Netflix without any commercial arrangement with these content
providers. Apparently, the reasoning is that "ISP quality" is measured
by the populace as "can I easily stream this popular content", and so
ISPs use tricks to improve delivery of the most popular content and
compete with other ISPs. (I remember these discussions as part of the
MARNEWS workshop.) That kind of feedback loop is not affected by the
state of core transit.

> More generally, as an academic working on issues related to the Internet, I view it as my mission to identify designs and paradigms that would make for a better Internet, and to explore whether the…

Bravo! But then, what about a public option for the edge?

-- Christian Huitema

---

From: Scott Shenker <[email-masked]>
Date: Wed, 12 Aug 2020 11:47:28 -0700

To avoid misconceptions, I would encourage everyone who is interested to look at the 20 minute video online (https://dl.acm.org/doi/abs/10.1145/3387514.3405875#sec-supp <https://dl.acm.org/doi/abs/10.1145/3387514.3405875#sec-supp>). Or if you really want to punish yourself, you can read the paper at (https://dl.acm.org/doi/pdf/10.1145/3387514.3405875 <https://dl.acm.org/doi/pdf/10.1145/3387514.3405875>).

Christian, the hollowing out of the core is precisely what we are addressing (in the video, it is the first motivating problem). Moreover, we are structuring the POC to encourage competition and open access at the edge. So you may not agree with our proposed approach, but I think we are trying to address the problems you raised!

—Scott

> On Aug 12, 2020, at 8:37 AM, Christian Huitema <[email-masked]> wrote:
> 
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 16 Aug 2020 12:02:56 +1200

> We did not mean to imply that the two are notions are identical, and indeed we discuss the no-QoS interpretation of network neutrality later in the paper. However, I think it is undeniable that con…

I'd go a bit further. We don't want application-type neutrality, because (e.g.) video streams require different service characteristics from (e.g.) software update distribution. What we want is application-service-provider neutrality, so that streaming from CarpenterTube gets the same service as streaming from ShenkerTube. I know that the authors know this, but the policy-level debates have often been unclear on this point.

> The world of today is very different from what it was in 1995, particularly in terms of the market forces at work. We discuss our reasoning for why our proposal might become a reality later in the…

As you say: "Our alternative is to cleanly separate transit and last-mile-delivery" and indeed that would be a good thing. I appreciate this comment: "If more and more LMPs find the POC attractive (and the success of
IXPs in Europe suggest that this is not farfetched)...". As it happens it was Olivier Martin in my group at CERN who set up the CERN neutral IXP in [thinks hard] late 1995, and it's still going strong today (https://cixp.net/about-cixp). In the world of 1995/96, ISPs jumped onto to the CIXP precisely because it was non-profit and neutral.

My question is whether today's LMPs are able to take independent decisions to migrate to the POC. The difference from 1995 is surely that most LMPs are not independent so cannot take such a decision.

Incidentally I would expect a real deployment of the POC to start by peering with as many of the non-profit IXPs as possible. That would create an instant flash mob of POC users. Successful peering is its own best advertisement.
 
> More generally, as an academic working on issues related to the Internet, I view it as my mission to identify designs and paradigms that would make for a better Internet, and to explore whether the…

Absolutely, and thanks for working on this.

   Brian

> .

---

From: Joseph Touch <[email-masked]>
Date: Sat, 15 Aug 2020 18:48:18 -0700

> On Aug 15, 2020, at 5:02 PM, Brian E Carpenter <[email-masked]> wrote:
> 
>>> On Aug 11, 2020, at 3:53 PM, Brian E Carpenter <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>>> Scott Shenker et al. just presented a pretty thought-provoking "public option" for the Internet's core backbone at SIGCOMM: https://dl.acm.org/doi/abs/10.1145/3387514.3405875 <https://dl.acm.org/doi/abs/10.1145/3387514.3405875>
>>> 
>>> Hmm.
>>> 

It’s presumptive at best to claim anything in one voice about any community; this one perhaps moreso (esp. on the main list recently)...

> I'd go a bit further. We don't want application-type neutrality, because (e.g.) video streams require different service characteristics from (e.g.) software update distribution.

I agree only that there are many interpretations of the term, many at odds with each other.

Again, “we” is hard to define, but it certainly doesn’t include me...

> What we want is application-service-provider neutrality, so that streaming from CarpenterTube gets the same service as streaming from ShenkerTube.

*I* want apps to be able to get different service from the network and to pay differently for them if needed, but *never* to have the network infer or enforce that mapping. Just like the USPS, variable fee for variable service, but there’s absolutely no reason the price of a letter should be determined by who wrote it or what it says. If a user wants to run software update over low-latency, slightly lossy service, that’s their decision; same for someone who wants to stream over best-effort.

Providers that do otherwise only force us to run a whole Internet *inside* a single port number - e.g., as is starting with the merging of DNS and HTTP over 443.

Joe

---

From: John Grant <[email-masked]>
Date: Sun, 16 Aug 2020 11:02:05 +0100

On 16/08/2020 02:48, Joseph Touch wrote:
> *I* want apps to be able to get different service from the network and 
> to pay differently for them if needed, but *never* to have the network 
> infer or enforce that mapping.
Of course, if you're going to do that -- have the application negotiate 
a particular kind of service rather than have the network "inspect" the 
packets and guess what service is needed -- you need a control plane 
protocol (let's call it "signalling") to do the negotiation, and you no 
longer have the connectionless/stateless paradigm on which IP is founded.