Subject: [arch-d] Fwd: New Version Notification for draft-nottingham-avoiding-internet-centralization-01.txt

From: Mark Nottingham <[email-masked]>
Date: Sun, 9 Jan 2022 20:18:07 +1100

Hello everyone,

Happy New Year! 

This is a bit more polished, thanks to some holiday time, some quarantine time, and feedback received. 

Responses and suggestions still most welcome.

Cheers,

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-nottingham-avoiding-internet-centralization-01.txt
> Date: 9 January 2022 at 8:14:43 pm AEDT
> To: "Mark Nottingham" <[email-masked]>
> 
> 
> A new version of I-D, draft-nottingham-avoiding-internet-centralization-01.txt
> has been successfully submitted by Mark Nottingham and posted to the
> IETF repository.
> 
> Name:		draft-nottingham-avoiding-internet-centralization
> Revision:	01
> Title:		Centralization and Internet Standards
> Document date:	2022-01-09
> Group:		Individual Submission
> Pages:		20
> URL:            https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-01.txt
> Status:         https://datatracker.ietf.org/doc/draft-nottingham-avoiding-internet-centralization/
> Html:           https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-01.html
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-nottingham-avoiding-internet-centralization
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-nottingham-avoiding-internet-centralization-01
> 
> Abstract:
>   Despite being designed and operated as a decentralized network-of-
>   networks, the Internet is continuously subjected to forces that
>   encourage centralization.
> 
>   This document offers a definition of centralization, explains why it
>   is undesirable, identifies different types of centralization,
>   catalogues limitations of common approaches to controlling it, and
>   explores what Internet standards efforts can do to address it.
> 
> 
> 
> 
> The IETF Secretariat
> 
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Vittorio Bertola <[email-masked]>
Date: Mon, 10 Jan 2022 12:17:43 +0100 (CET)

> Il 09/01/2022 10:18 Mark Nottingham <[email-masked]> ha scritto:
> 
> 
> Hello everyone,
> 
> Happy New Year! 
> 
I don't know if you had a chance to have a look at the new draft I posted a few days ago to the model-t list:

https://datatracker.ietf.org/doc/draft-bertola-everything-but-the-user/

It's a take (a personal position) on the discussion on a possible expansion of the RFC 3552 Internet threat model. It looks like we are attacking similar problems from different angles; especially the problem of centralisation - which IMHO is significantly worsened by having a threat model which considers non-user-controlled endpoint elements as out of scope, rather than as potentially adversarial intermediaries much like the in-network ones - and the discussion on multistakeholderism.

On this latter point, commenting on S Moonesamy's observation and your recent reply, I would also say that the IETF is currently not a fully multistakeholder entity, not for the lack of formalized constituencies but for the lack of significant participation by many of the non-technical Internet constituencies. However, whether it makes sense for non-technical stakeholders to participate in a technical SDO - and how - is up for discussion, and as you will find in the draft, my suggestion is that non-technical stakeholders should come into play in selected situations, only when the non-technical impact of the new standard looks relevant.

--

Vittorio Bertola | Head of Policy & Innovation, Open-Xchange
[email-masked]  mailto:[email-masked]
Office @ Via Treviso 12, 10144 Torino, Italy

---

From: Dominique Lazanski <[email-masked]>
Date: Mon, 10 Jan 2022 17:27:23 +0100

Also, Mark and I have a slightly updated take on our draft here:

https://datatracker.ietf.org/doc/draft-lazanski-consolidation/ <https://datatracker.ietf.org/doc/draft-lazanski-consolidation/>

It seems that there is common ground among all three and also some big differences. However, its clear to me that there are technical, economic and policy related issues that are intertwined and need to be discussed further. The unintended consequences of legislation in, say, Russia (for example) is based in consolidation taking place on US platforms. These can’t be ignored.

Looking forward to a further discussion!

Dominique 

> On 10 Jan 2022, at 12:17, Vittorio Bertola <vittorio.bertola=[email-masked]> wrote:
> 
> 
> https://datatracker.ietf.org/doc/draft-bertola-everything-but-the-user/ <https://datatracker.ietf.org/doc/draft-bertola-everything-but-the-user/> 
> 
> [email-masked] <mailto:[email-masked]> 
> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss

---

From: Watson Ladd <[email-masked]>
Date: Mon, 10 Jan 2022 20:43:34 -0800

On Mon, Jan 10, 2022 at 8:27 AM Dominique Lazanski
<[email-masked]> wrote:
> https://datatracker.ietf.org/doc/draft-lazanski-consolidation/

This draft makes a number of misleading and poorly substantiated
statements about DoH that easily could have been corrected by checking
against the published RFC. Given that several of the authors were
vocal participants in the rather ugly and extended debate about DoH, I
expect better. A sampling

- "This change in the look up process is forcing the technology to
develop in a way which has narrowed the ability for companies and
small industries to do DNS look ups without updating out of date
hardware and software, therefore disenfranchising developing countries
and smaller companies without  big budgets."

The  addition of a protocol does not narrow the ability to use the
older protocol. You would have to show revolvers turning off 53 in
favor of DoH. That hasn't happened.

- "By routing the DNS over HTTPS, it becomes much easier to track user
activity through the use of cookies."

Actual text from RFC 8484: " HTTP cookies SHOULD NOT be accepted by
DOH clients unless they are explicitly required by a use case." This
would seem to run against the previous statement. It also ignores
further privacy improvements that DoH makes possible such as ODoH.

A large number of statements about the CDN market are also not true,
and easily checkable public sources could have shed some much more
interesting light on them. CDN providers counteract centralization on
some layers by making it much easier for smaller networks to remain
online and serve content. They also can be centralizing if they get
much market share, but it's very difficult to understand the market
share of CDNs across top domains.

Unless there is a workable discovery protocol, and the people calling
for it should be the ones to propose it, I don't think we should hold
off on real privacy improvements for real users as a result. I think
there are real problems to think about in this space, but making this
a referendum on the biggest privacy improvement for real users since
Firesheep is not going to help have that conversation.

> It seems that there is common ground among all three and also some big differences. However, its clear to me that there are technical, economic and policy related issues that are intertwined and ne…

I'm not aware of any large consequences of laws in Russia caused by US
consolidation. That definitely doesn't mean there haven't been any,
but it might be useful to point to examples.

-- 
Astra mortemque praestare gradatim

---

From: Watson Ladd <[email-masked]>
Date: Mon, 10 Jan 2022 21:38:25 -0800

> Responses and suggestions still most welcome.

I think this is a very useful document, I do have a few small quibbles.
"Similarly, the need for coordination in the Web's trust model brings
centralization risk, because a Certificate Authority (CA) can control
communication between the Web sites that they sign certificates for
and users whose browsers trust the CA's root certificates."

This isn't quite the case; a CA can sign for any website (ignoring
CAA, etc.) and then intercept. However techniques like CT make it much
more visible.

ISP choice varies heavily by region: it's a classic natural monopoly
as the costs of building out a network exceed the service provision
massively. For applications I think it's worth saying something about
the way capturing value and improvements are related. It is because a
centralized protocol can have value capture that the central
participant is motivated to invest in it. For a decentralized protocol
new application improvements can depend on other participants, making
it harder to realize the value. The other point I would make is
network effects are related to switching costs: if it is easy to
switch to another network or use both, then the value of having
everyone on the network is less than if it is inherently difficult to
switch. In that regards the Internet can create a situation  where
"the competition is a click away" and a seemingly unassailable
advantage vanishes overnight.

---

From: Eliot Lear <[email-masked]>
Date: Tue, 11 Jan 2022 07:33:28 +0100

Watson,

Full disclosure- my name is on the draft, but it is a draft, and it 
needs work.  In this regard, I think you have jumped the gun in your 
review.  Please see below.

> favor of DoH. That hasn't happened.

I don't think you have to show that resolvers are turning off port 53, 
but rather simply that the query traffic has shifted from port 53.  
Certainly that has, to some extent or another, happened.

> would seem to run against the previous statement.
I think the statement requires further qualification: if the SHOULD is 
ignored by both parties, then that risk exists.  Other tracking risks 
exist as well, but then...
>   It also ignores
That is a remedy to that form of tracking.
> share of CDNs across top domains.

I think the fundamental risk here is concentration of resolver 
providers.  What would happen to load if a Big Cloud Provider Who Offers 
DoH Today had a systemic failure (we have certainly seen such things)?  
This can of course be remediated by having a large and diverse selection 
of providers to spread load, so that no one provider can cause an 
excessive load shift.

By the way, your own statement is baseless.  That is, if you are going 
to correct someone, please be so kind as to at least cite some sources, 
since you claim they are readily available.  Not that the draft 
shouldn't support its own claims, of course.

> Firesheep is not going to help have that conversation.

Where is anyone being held back?

> but it might be useful to point to examples.

Indeed examples are good.  By the way, The Verge had an article about 
the Swiss Army doing its own version of IM because of risks associated 
with WhatsApp being a US company.  Yes, there are a number of 
entertaining aspects to that bit of information; and I'm confident the 
Verge doesn't have the story quite right.  But at least enjoy the 
entertainment as it comes.

Eliot

---

From: Eliot Lear <[email-masked]>
Date: Tue, 11 Jan 2022 07:58:37 +0100

Several points:

On 11.01.22 06:38, Watson Ladd wrote:
> massively.

Be careful with your use of "natural monopoly".  We have earlier work 
from Daniel Karrenberg et al on market concentration.  In the interim, 
different markets have played out in different ways, and we are seeing 
new offerings such as Star Link and crop up.

Mark specifically decided not to go there in his draft, though I am not 
certain that he can entirely avoid it.  In particular, market 
concentration clearly implies all the risks that Mark mentions regarding 
centralization, both in terms of geographical concentration and platform 
concentration.

One aspect I think would be useful to at least mention is zero-rating, 
and whether it has centralizing effects.

>   For applications I think it's worth saying something about
> it harder to realize the value.

This is a *very* good point, but we also have to understand the base 
case.  That is- nobody is going to offer a product unless they can make 
money and stay competitive.  Standards that permit little or no 
differentiation provide limit first-mover advantage. Thus we see, for 
instance, the rise of the app and the fall of the protocol.  That too 
may be worth discussing – in some context. Again, I’m not sure whether 
Mark wants it in his work.

> The other point I would make is

I am not sure what you mean by “The Internet” here.

Eliot

---

From: Watson Ladd <[email-masked]>
Date: Mon, 10 Jan 2022 23:47:41 -0800

> Certainly that has, to some extent or another, happened.

That's not what the sentence says: it says that ability to do DNS
lookups (not serve queries) via port 53 has been reduced. A shift in
traffic doesn't mean anything to the traffic that remains.

<snip>
> of providers to spread load, so that no one provider can cause an

Why does DoH make this worse than DNS? 8.8.8.8 works just fine over
53, and you have the same concentration issue. I think this a more
general thing than with DNS and should be a risk of concentration:
failures might remove substantial capacity.
Yes, AWS-East-1, I'm looking at you.

> shouldn't support its own claims, of course.

I retract that, there aren't very clear contradictions. I'll link some
sources that complicate the picture
https://w3techs.com/technologies/overview/proxy shows that the most
common reverse proxy is none. Only 20% of sites use Cloudflare.
Unfortuately it doesn't seem to have historical data, but Netcraft
does, but alas only for server, and they had 10% for Cloudflare
recently https://news.netcraft.com/archives/2021/11/23/november-2021-web-server-survey.html.
These are substantial percentages!

The problem is that these measurements only look at sites, not
traffic. And traffic is highly skewed, with many large sites running
their own CDN, and many CDNs only competing for high traffic sites or
in particular niches. I don't really know what the market looks like:
a handful of competitors as it says in the draft is too few.  Probably
the best way is to try to pull it out from IX records, but off-nets
complicate it. It's not an easy number to get.

My recommendation would be to focus on sites, as it's more observable
and substantiated. And there's a more interesting dynamic: DDoS
attacks make it difficult to run small networks, filtering services
and CDN make it easier to run websites, but at the cost of having a
substantial impact for the rest of the world if it goes down and a lot
of sites impacted.

The other technical point is about CDN geolocation being disrupted by
DoH. This doesn't actually happen: the CDNs that do rely on geographic
IP assigment see some degradation even without /24 propagated, but
it's not big as the destination of the DoH resolver is close
networkwise to the query. No different from what happens with Do53,
and the same solutions like explicit client indication work. Useless
is way overstating it.

> Where is anyone being held back?

Nobody is, because the IETF process hasn't stopped ODOH and DOH
because people want discovery. That doesn't stop people from arguing
otherwise. Your draft argues that we need more review of protocols
then the already ponderously slow IETF consensus process, focused on
the claimed operational problems, that protocol design has economic
implications we need to review, etc. These seem like extremely
difficult things to actually get to solid conclusions on.

---

From: Mat Ford <[email-masked]>
Date: Tue, 11 Jan 2022 08:32:18 +0000

> On 11 Jan 2022, at 07:47, Watson Ladd <[email-masked]> wrote:
> 
> I'll link some
> and substantiated.

You can get some insight (based on W3 Techs data) into concentration in the market for CDNs over time based on Top (1M, 10k, 1k) sites here: https://pulse.internetsociety.org/centralization

It’s a pretty heavily concentrated marketplace although less so for top 1k sites. The picture when one considers the jurisdiction of the CDN provider is even more heavily skewed.

Mat

---

From: Eliot Lear <[email-masked]>
Date: Tue, 11 Jan 2022 10:21:57 +0100

> Yes, AWS-East-1, I'm looking at you.

Absolutely true.  DoH *can* have a concentrating effect, but it's not 
the only thing that can.  And just because it can doesn't make it Bad 
Technology.  I'm not sure there is *any* scalable protocol mechanism 
that *doesn't* suffer have a concentrating impact.  In fact, my working 
thesis is that scale and centralization walk hand in hand, but I'm happy 
to be proved wrong.

Eliot

---

From: Vittorio Bertola <[email-masked]>
Date: Tue, 11 Jan 2022 11:25:18 +0100 (CET)

> Il 11/01/2022 06:38 Watson Ladd <[email-masked]> ha scritto:
> 

Please do not take offense, but this is also an unsubstantiated claim. Perhaps it was so in the 90's, and still the choice in physical infrastructure is limited, though in big cities in Europe it can happen to be reached by multiple fiber networks. But all in all, now and here in a second-tier European city, I can get Internet access from one of more than 50 fiber providers, three major 4/5G mobile operators plus at least a dozen virtual ones, and a handful of satellite / wide-area-wireless providers, and most of them give pretty good service (or not worse than the others, if you prefer). If I move 100 km north or 100 km west, I get into two other European countries where I have similar ranges of choice among completely different and mostly unrelated companies.

But unless I want to embark in a painful odyssey of crippled experiences, I can only choose among two mobile operating systems, a couple of search engines and half a dozen browsers throughout the entire Western world.

-- 
Vittorio Bertola | Head of Policy & Innovation, Open-Xchange
[email-masked] 
Office @ Via Treviso 12, 10144 Torino, Italy

---

From: Andrew Campling <[email-masked]>
Date: Tue, 11 Jan 2022 10:44:55 +0000

On 11.01.22 05:38, Watson Ladd <[email-masked]> wrote:

> ISP choice varies heavily by region: it's a classic natural monopoly as the costs of building out a network exceed the service provision massively. 

It's important to distinguish between ISPs and network operators: the latter build and run networks whereas the former provide Internet services over those networks.  Clearly some companies do both but, at least in some markets, there are large numbers of ISPs that provide service over other companies' networks (via wholesale access).  In such markets at least it is not correct to suggest that ISPs are natural monopolies.  

Separately, some markets are providing increasing evidence that network operation is not a natural monopoly either.  The most common examples are where (broadband) Internet connectivity is available using a range of competing technologies including fixed telco networks, cable networks, fixed wireless, LEO satellite etc.  In the UK there are increasing instances of properties in urban areas being served by multiple FTTP networks, each of which may be offering Internet connectivity via multiple ISPs.

Noting the above, I'd concede that there is some evidence that fixed network connectivity in rural areas may be a natural monopoly, anything beyond that is highly dependent on local market conditions, the regulatory environment etc.

Andrew

---

From: S Moonesamy <[email-masked]>
Date: Tue, 11 Jan 2022 02:34:21 -0800

Hi Dominique,
>https://datatracker.ietf.org/doc/draft-lazanski-consolidation/
[snip]

>Looking forward to a further discussion!

Thank you for pointing to your draft.

Section 2 of the draft mentions "player" concentration.  Shouldn't 
that be described as market concentration?

I may be misinterpreting that section; it looks like "consolidation" 
and "concentration" are used interchangeably.

The draft states that economies of scale led to consolidation of 
email services [1].  It does not include information about how that 
conclusion was reached.  Was it influenced by protocol design, 
"cloud" technologies, or something else?

The draft uses DoH as an example of how protocol design promotes 
consolidation/market concentration and argues that there should have 
been better consultation [2] prior to the standardization 
decision.  Who drove the deployment of the protocol?

Regards,
S. Moonesamy

1. Many years ago, I disagreed about what was written in 
draft-tschofenig-post-standardization-00.  I'd say, with hindsight, 
that my opinion was incorrect.

2. This is probably related to participation in "international" 
standardizing bodies .

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 11 Jan 2022 11:54:16 +0100

> Please do not take offense, but this is also an unsubstantiated claim. Perhaps it was so in the 90's, and still the choice in physical infrastructure is limited, though in big cities in Europe it c…

And, if the fiber is available to use (either as dark fiber or transmission of some kind) the end user can choose from multiple ISPs even if there is a monopoly regarding fiber.

   Patrik

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 11 Jan 2022 19:40:46 +0100

On Tue, Jan 11, 2022 at 10:21:57AM +0100, Eliot Lear wrote:
> In fact, my working thesis is that scale and centralization walk hand
> in hand, but I'm happy to be proved

Probably depends on the structure of the organizations/groups/entities that scale.
The more hierarchical they are, the more centralized they will scale.

But: i would suggest to use words like concentration, monopolization,
homogenization and similar. Centralization is very unspecific and
has a lot of beneficial aspects whereas the above three are better
pointing (IMHO) to the candidate issues.

---

From: Watson Ladd <[email-masked]>
Date: Wed, 12 Jan 2022 08:59:42 -0800

> Please do not take offense, but this is also an unsubstantiated claim. Perhaps it was so in the 90's, and still the choice in physical infrastructure is limited, though in big cities in Europe it c…

I was using natural monopoly as statement about the economics of
building the network, not a statement about outcomes.
> But unless I want to embark in a painful odyssey of crippled experiences, I can only choose among two mobile operating systems, a couple of search engines and half a dozen browsers throughout the e…

And what could the IETF have done to change those outcomes and how
does that matter to what the IETF does? If we're talking about
browsers where there are now just three engines with any usage, and
two multiplatform, that has a very different interaction with
standardization than with operating systems. And that's again
different from search or video delivery.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Jan 2022 18:16:09 +1100

Hi Watson,

> On 11 Jan 2022, at 4:38 pm, Watson Ladd <[email-masked]> wrote:
> 

Yes - I'll attempt to add some nuance here. Tracked at:
  https://github.com/mnot/avoiding-internet-centralization/issues/20

> it harder to realize the value.

For me, this gets back to Moxie's observations. Beyond noting the tradeoff here, is there something we can say about how to address it, beyond "build standards that actually meet people's needs" (which 6.4 brushes against)?

> advantage vanishes overnight.

Yes - see <https://github.com/mnot/avoiding-internet-centralization/issues/14>. 

Thanks for the feedback!

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Jan 2022 18:17:09 +1100

Hi Vittorio,

Thanks for the reference. I've been thinking a lot about intermediation recently as well, but don't know that I come to the same conclusions. I'll try to write something up (but it may take some time).

Cheers,

> On 10 Jan 2022, at 10:17 pm, Vittorio Bertola <[email-masked]> wrote:
> 
>