Subject: [arch-d] Comments on draft-lazanski-consolidation

From: Watson Ladd <[email-masked]>
Date: Thu, 12 Nov 2020 19:00:04 -0800

Dear all,

As someone who works on a team that's implemented all three of the
protocols explicitly listed, it's a bit awkward for me to comment on
the draft. But comment I will.

The brief summary is that I think this draft suffers from trying to do
too much, and if it was split up into section 5 and the rest it would
be much more convincing and productive.

Let me note that aws-us-east-1 having a bad day shouldn't be a news
item. That alone is evidence that there is consolidation in the
provision and location of computer services. But that doesn't really
have a lot to do with protocol changes, and there aren't many examples
of protocols changing in response to consolidation in the draft.

I'm not sure what to make of section 4.2. It discusses the end to end
principle, then says the availability of storage and computation
changes that. NFS has existed for a long time, that doesn't mean the
Internet provides storage. It means servers on the Internet provide
storage. ISPs have always consolidated traffic, at least at the
metropolitan level. They bring it to IXs or their transit links. 5G
isn't changing that. The end to end principle is a principle of
network design, and it isn't the only way to design networks: circuit
switching and store-and-forward networks didn't do it, in part due to
technical limitations. What is the edge to edge principle that this
draft refers to?

There have always been computers on the Internet, providing services
to people. Anyone who has ssh'd into a cluster to run their codes is
using data processing over the Internet. That these services might be
anycasted or spread over multiple sites is not new. It doesn't make
the network intelligent in the way the end-to-end principle argues
against.

All of this change doesn't involve new protocols.

And then the draft discusses ECH, DoH, Privacy Pass as protocols that
could lead to further consolidation. DoH has been discussed
considerably in a variety of fora: I'll simply note that you can pick
your resolver with DNS over UDP as well, and it's not really a
protocol level issue how it was deployed, which the draft
acknowledges. "Centralized" resolvers like 8.8.8.8 existed before DoH.

ECH is I think probably the draft's strongest argument. Any sort of
anonymity service benefits from having more usage, so them that has,
gets. This is why e.g. Tor is prefered over less popular solutions.
It's worth considering these incentives but I think that there is, and
always has been, a variety of shared hosting options. There are some
very real technical difficulties here, and I think all of us would
welcome well thought out proposals. But doing nothing about SNI
leakage is not an option.

Privacy pass I think has been misinterpreted in this draft.. The
multiple server case is really not central, it's a possible deployment
model. The deployment that currently exists and many of the ideas that
have been tossed around are of the single server model: the verifier
only trusts a single server to hand out the token that is then used,
akin to the usher taking a ticket at a movie theater.  Different
deployments use different servers, and there isn't a privacy issue.
The draft is not saying there should be four servers worldwide.

In conclusion I think this draft needs to be split up and each part
further developed and clarified to be a good basis for further
discussion.

Sincerely,
Watson

-- 
Astra mortemque praestare gradatim

---

From: Christian Huitema <[email-masked]>
Date: Thu, 12 Nov 2020 21:14:57 -0800

> leakage is not an option.

Watson's point is confirmed by market analysis. There is significant
competition in the CDN market -- the ascension of Fastly from 'small
startup" to "significant player" shows that. In fact, there is also
competition in the "cloud hosting" market -- AWS and Azure are big, but
there is a variety of other players, from Google Cloud to IBM, Oracle,
Linode, Digital Ocean and many others. I did an analysis of that here:
https://huitema.wordpress.com/2020/08/09/can-internet-services-hide-in-crowds/,
but there is better data available here:
https://www3.cs.stonybrook.edu/~mikepo/papers/esni.asiaccs20.pdf.

As for ECH, the analysis shows that most of these "top million" sites
are running on dedicated IP addresses, even when serviced by VPNs. Yes,
ECH requires "shared address" hosting to work, but no, there is no
evidence that big services are leveraging shared address provisioning to
grab market share.

-- Christian Huitema

---

From: Eliot Lear <[email-masked]>
Date: Fri, 13 Nov 2020 09:14:04 +0100

Hi Watson,

> On 13 Nov 2020, at 04:00, Watson Ladd <[email-masked]> wrote:
> 
> Dear all,
> 
> provision and location of computer services.

Yes.

> But that doesn't really

I might put this differently.  There are natural tendencies to consolidation in any market, regardless of protocols, simply because of economies of scale.  But our protocols support that economic structure, and that is deeply ingrained in our culture, to the point where I often quote Mike O’Dell, a former IETF area director and an Internet pioneer: “The only real problem is scaling.”  Load balancing is an absolute requirement for scale, and so SNI came to be.  So we addressed that to scale, and that enabled market consolidation, because without it CDNs could not offer secure services.  

The same design can be seen in our mail systems.  The MX record provides for all manner of scaling.  And for consolidation.  Gmail, the system you use, could hardly function without it.  The same with short-lived TTLs.

There are other examples to me that seem to me to have created consolidation due to lack of scale. For instance, NAT and private addressing limited services to be delivered from the home, and once in place, SPs built entire business models around that assumption.  Some of this might change over time with IPv6, but the status quo provides for a lot of inertia.

Dynamic addresses build on this.  Without DHCP, it would not be scalable for SPs to rotate addresses, thereby eliminating unwanted service offerings.  And then there are services that spot that, and correct for it.  Whole markets built out around this need for scale that a protocol delivered.

In each of these cases, the market pressure was there for work, our tradition and processes of rough consensus and running code enabled the work, and people showed up with code.

> 
>  not sure what to make of section 4.2. It discusses the end to end

NFS being a massive pain to handle, with all sorts of history with poor performance on WANs, kernel hangs, and all the PTSD that brought to administrators.  Probably we should leave that example on the cutting room floor.  

On the other hand, SSH is more interesting.  SSH has no key management system to speak of, which is why attacking key servers led to a fundamental failure.  This is particularly ironic, given how it was billed initially as a web of trust.  You see no formal consolidation there, but nevertheless there was a fail.  Conversely and perhaps worth study, you don’t see quite that same sort of failure with X.509 certs, even though there is consolidation in the browser market.  There are huge numbers of CAs in browser stores.  What would happen if those policies got too restrictive?  What are the economic incentives in play?

> acknowledges. "Centralized" resolvers like 8.8.8.8 existed before DoH.

Users choosing DNS servers is a rarity.  The only real question is who else chooses.  Is it the service provider, the home router vendor, or the browser?  By creating DoH, browsers have challenged the other two.  We ought not deny that reality.  However, I also don’t think we should replay good/bad here.

To me, the draft needs to also look at malware and how consolidation helps/hurts.  I think it’s also probably worth while to look at the recent O365 downtime, how two-sided markets can hurt scale and stability, what what standards’ role are to prevent that risk.

Eliot