Subject: [arch-d] Fwd: New Version Notification for draft-nottingham-avoiding-internet-centralization-02.txt

From: Mark Nottingham <[email-masked]>
Date: Thu, 10 Feb 2022 13:07:35 +1100

Hi everyone,

More updates, both in response to feedback received and after further consideration.

Cheers,

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-nottingham-avoiding-internet-centralization-02.txt
> Date: 10 February 2022 at 1:06:30 pm AEDT
> To: "Mark Nottingham" <[email-masked]>
> 
> 
> A new version of I-D, draft-nottingham-avoiding-internet-centralization-02.txt
> has been successfully submitted by Mark Nottingham and posted to the
> IETF repository.
> 
> Name:		draft-nottingham-avoiding-internet-centralization
> Revision:	02
> Title:		Centralization and Internet Standards
> Document date:	2022-02-09
> Group:		Individual Submission
> Pages:		21
> URL:            https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.txt
> Status:         https://datatracker.ietf.org/doc/draft-nottingham-avoiding-internet-centralization/
> Html:           https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.html
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-nottingham-avoiding-internet-centralization
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-nottingham-avoiding-internet-centralization-02
> 
> Abstract:
>   Despite being designed and operated as a decentralized network-of-
>   networks, the Internet is continuously subjected to forces that
>   encourage centralization.
> 
>   This document offers a definition of centralization, explains why it
>   is undesirable, identifies different types of centralization,
>   catalogues limitations of common approaches to controlling it, and
>   explores what Internet standards efforts can do to address it.
> 
> 
> 
> 
> The IETF Secretariat
> 
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 10 Feb 2022 17:07:09 +1300

Mark,

I agree that "Blockchains Are Not Magical", but I'm curious. Why did "Distributed consensus protocols" in bullet #1 of that section become just "Distributed consensus"? And why does the bullet now seem to imply that distributed consensus requires a blockchain?

Maybe "Distributed Consensus Protocols" even needs a section of its own.

(I'm no expert, but Heidi Howard might have an opinion. https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-935.pdf )

Regards
    Brian Carpenter

On 10-Feb-22 15:07, Mark Nottingham wrote:
>> *From: [email-masked] <mailto:[email-masked]>
>> *Subject: **New Version Notification for draft-nottingham-avoiding-internet-centralization-02.txt*
>> *Date: *10 February 2022 at 1:06:30 pm AEDT
>> *To: *"Mark Nottingham" <[email-masked] <mailto:[email-masked]>>
>>
>>
>> Name:draft-nottingham-avoiding-internet-centralization
>> Revision:02
>> Title:Centralization and Internet Standards
>> Document date:2022-02-09
>> Group:Individual Submission
>> Pages:21
>> URL: https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.txt <https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.txt>
>> Status: https://datatracker.ietf.org/doc/draft-nottingham-avoiding-internet-centralization/ <https://datatracker.ietf.org/doc/draft-nottingham-avoiding-internet-centralization/>
>> Html: https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.html <https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.html>
>> Htmlized: https://datatracker.ietf.org/doc/html/draft-nottingham-avoiding-internet-centralization <https://datatracker.ietf.org/doc/html/draft-nottingham-avoiding-internet-centralization>
>> Diff: https://www.ietf.org/rfcdiff?url2=draft-nottingham-avoiding-internet-centralization-02 <https://www.ietf.org/rfcdiff?url2=draft-nottingham-avoiding-internet-centralization-02>
>>
>> Abstract:
> Mark Nottingham https://www.mnot.net/ <https://www.mnot.net/>
> 
> 
> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss
>

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 10 Feb 2022 15:14:11 +1100

I'd have to check the commit logs, but I don't remember that as a conscious decision -- it may have been an unintentional change as the result of editorial work.

Regarding distributed consensus -- I think the relevant aspect is when that consensus is spread across distinct (and often, unknown) actors. In practice, distributed consensus is more often used between actors who are under control of the same authority, to improve reliability against various technical failures, but not to address issues surrounding authority itself. There are exceptions, of course, and if we can find better terminology to make this distinction, that'd be very helpful.

Thanks for the extra reading :)

Cheers,

> On 10 Feb 2022, at 3:07 pm, Brian E Carpenter <[email-masked]> wrote:
> 
> Mark,
>

---

From: Eliot Lear <[email-masked]>
Date: Thu, 10 Feb 2022 10:37:42 +0100

Mark,

Is there a good example of a system that was centralized and has been 
decentralized?

Eliot

On 10.02.22 03:07, Mark Nottingham wrote:
>> *From: [email-masked]
>> *Subject: **New Version Notification for 
>> draft-nottingham-avoiding-internet-centralization-02.txt*
>> *To: *"Mark Nottingham" <[email-masked]>
>>
>>
>> A new version of I-D, 
>> draft-nottingham-avoiding-internet-centralization-02.txt
>> URL: 
>> https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.txt
>> Status: 
>> https://datatracker.ietf.org/doc/draft-nottingham-avoiding-internet-centralization/
>> Html: 
>> https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-centralization-02.html
>> Htmlized: 
>> https://datatracker.ietf.org/doc/html/draft-nottingham-avoiding-internet-centralization
>> Diff: 
>> https://www.ietf.org/rfcdiff?url2=draft-nottingham-avoiding-internet-centralization-02
>>
>> Abstract:

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 10 Feb 2022 12:36:22 +0100

On 2022-02-10, at 10:37, Eliot Lear <[email-masked]> wrote:
> 
> Is there a good example of a system that was centralized and has been decentralized?

I’m not Mark, but I would point to DNS.
(Yes, that was in the pre-commercial phase of the Internet.)

Grüße, Carsten

---

From: Eliot Lear <[email-masked]>
Date: Thu, 10 Feb 2022 12:41:35 +0100

Yes, I was hoping for something post-1985 ;-)

---

From: Dirk Trossen <[email-masked]>
Date: Thu, 10 Feb 2022 12:52:42 +0000

Eliot,

Not sure if your question was limited to 'communication system' or even an example of Internet technology - if not, the energy grid comes my mind, moving from many decades of centralization (a POP like model) to a decentralized system, including localized markets. 

Best,

Dirk

---

From: Vittorio Bertola <[email-masked]>
Date: Thu, 10 Feb 2022 14:54:24 +0100 (CET)

> Il 10/02/2022 12:41 Eliot Lear <[email-masked]> ha scritto:
> 
>  

Around 2000 there was a wave of decentralization around media. Before then you only had centralized (and rudimentary) music/video download services, then suddenly everyone started to exchange files, and later streaming packets, over peer-to-peer, decentralized architectures. Unfortunately the industry did not seem to like that loss of control, so we are now back to centralized services and/or "decentralization" made through a huge number of servers scattered around, all owned by a single CDN provider chosen among a set of very few.

Perhaps the bottom line is that decentralization at the application layer only happens when end-users have the freedom to do it on their own.

-- 
Vittorio Bertola | Head of Policy & Innovation, Open-Xchange
[email-masked] 
Office @ Via Treviso 12, 10144 Torino, Italy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 11 Feb 2022 08:57:01 +1100

It's an interesting question, and as mentioned I think the answer depends on whether you're limiting the search to Internet things.

How do you see the answer informing this document?

Cheers,

> On 10 Feb 2022, at 8:37 pm, Eliot Lear <[email-masked]> wrote:
> 
> Mark,
>

---

From: Hesham ElBakoury <[email-masked]>
Date: Thu, 10 Feb 2022 17:58:01 -0800

There is raft consensus protocol from Stanford.
https://www.usenix.org/node/184041.

Hesham

On Wed, Feb 9, 2022, 8:08 PM Brian E Carpenter <[email-masked]>
wrote:

> Mark,
>
> I agree that "Blockchains Are Not Magical", but I'm curious. Why did
> "Distributed consensus protocols" in bullet #1 of that section become just
> "Distributed consensus"? And why does the bullet now seem to imply that
> distributed consensus requires a blockchain?
>
> (I'm no expert, but Heidi Howard might have an opinion.
> https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-935.pdf )
>
> Regards
>     Brian Carpenter
>
> <
> <
> <
> <
> <

---

From: Guntur Wiseno Putra <[email-masked]>
Date: Fri, 11 Feb 2022 10:49:11 +0700

Dear Eliot, Mark &
Architecture-Discuss,

If by the terms "single control" and "homogeneous system" we may
identify/anticipate such "centralized operations", there was ever a thought
on  shifting "a homogeneous system to a heterogeneous one":

"Current multi-network systems are fairly small (tens of networks maximum)
and largely designed by and under control of a single organization. (We
shall call this "homogeneous" internetworking.) Basic interconnection is
supported by simple hierarchical addressing and routing procedures employed
uniformly throughout the system [1,4,10,13]. Interconnections of different
multi-network systems (heterogeneous internetworking) are just beginning to
be made, largely by ad hoc means".

C.A. Sunshine, "Addressing Problems in Multi-Network System", IEN#178,
April 1981

https://www.rfc.org/ien/ien178/

There may be RFCs and other IENs as historical records dealing with the
matters of "centralization" and "decentralization".

Regard,
Guntur Wiseno Putrs

Pada Kamis, 10 Februari 2022, Eliot Lear <[email-masked]> menulis:

> URL:            https://www.ietf.org/archive/id/draft-
> nottingham-avoiding-internet-centralization-02.txt
> Html:           https://www.ietf.org/archive/id/draft-
> nottingham-avoiding-internet-centralization-02.html
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-
> nottingham-avoiding-internet-centralization
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-
> nottingham-avoiding-internet-centralization-02
>
> Abstract:
> Architecture-discuss mailing [email-masked]://www.ietf.org/mailman/listinfo/architecture-discuss
>
>

---

From: Eliot Lear <[email-masked]>
Date: Fri, 11 Feb 2022 09:01:42 +0100

Hi Mark,

First I want to reiterate something I think I've said to you before: I 
truly admire your leading the charge in this area. People may forget, 
but I certainly haven't, that you were raising these concerns as early 
as 2014, and maybe earlier.

My answer to you rambles just a bit, but I hope it is helpful anyway.

> How do you see the answer informing this document?

I think it gives us a strong indicator as to how much of §5 (modulo 
§5.6) you should include or how you should couch that section or 
otherwise organize the work.  There is, I think, a lot more to say in 
§4, and it may lead you to want to review how you discus/present the 
earlier sections.

What would happen if a popular/dominant platform app store broke down or 
was compromised?  What enabled that form of centralization?  There are 
certainly cybersecurity benefits in having firm control over what code 
gets run on billions of end point devices, but there are also risks.  
Isn't that a form of centralization?  This area is a whopper because 
from an IOT standpoint we tell all manufacturers to provide signed 
updates to their devices.  But right now as I write this to you, I am 
fuming over one platform provider stopping support update for a 
particular device.  Is there something to be said about trust anchors?

There is an entire area of exploration one could enter around 
cyber-risks, scale, and centralization.  Certificates are a *great* area 
because we just saw a fail in which LE invalidated 2 million 
certificates because of a problem with TLS-ALPN-01.  Is 2 million a 
large # these days? Could it have been worse?  LE has turned the 
certificate market on its ear, but this has created challenges of its 
own.  The browser cert cache seems an interesting area to explore.  In 
fact there was a recent paper about the sort of garbage that accumulates 
in there.  I'm sorry that I can't find the cite for that, but I assert 
that stronger controls on those caches may lead to MORE centralization.

We have recently discussed Eduroam as a federation, by the way, and I 
think understanding its success factors and risks is also worthy of 
exploration, since it is, so far as I know, the largest Internet 
federation in existence.  Is there a centralization risk with Eduroam 
and do alternatives to Eduroam exist? (Hint: yes they do, but they're 
obviously smaller). What enables those alternatives? (Hint: it's an IETF 
protocol object.) This might argue for keeping some parts of §5.  Oh and 
that example is LOADED with intermediaries.

Another area you might want to explore is the 3g/5g roaming 
architecture.  Are there centralization risks there?  If not, how were 
they mitigated?  BTW, they have their own centralized naming 
mechanisms.  There are SO many intermediaries in play here that truly I 
wonder how the system functions so well, but it does.

There are also policy aspects you could delve into, at the risk of 
getting away from IETFy stuff and into crypto-currency examples.  You're 
somewhat there already in §4.3 (which I quite like).  For instance, the 
payment networks (Visa, Mastercard, AMEX,...) exhibit both 
centralization and decentralization aspects, and operate largely as 
federations.  Governments outside of the US HATE some of that.  What are 
the scale factors that drove those networks, and what has kept them in 
place?  I predict that would strengthen your point in §5.6.  There are a 
wealth of references for that.  You might want to have a look at [1] as 
a good start, but there's a lot of Visa/Mastercard envy out there. 
Again, lots of intermediaries in play here.

The DNS example *is* enlightening.  That was a case where the 
centralized authorities for the name space (Jon and Jake) very much 
wanted to get out of the business.  And we have another brewing example 
of that: there have been ramblings that NIST might want to decentralize 
the National Vulnerability Database.[2] Again, the central point in that 
case might have an interest in getting out of the operational side of 
things. In both of those cases, governments are involved and the 
motivations are often different from those of private concerns.[3]  That 
begins to help us zoom in on at least one factor in limiting 
centralization, but it's not a protocol aspect.

This leads me back to §5.6.  You touch on scale at several points in the 
document, but not in depth.  To me at least, it seems worthy of 
exploring centralization as the flipside of O'Dell's Axiom: “The only 
real problem is scaling.”  Cloud computing is one example.  CDNs are 
another.  Reputation management in the face of non-prearranged 
communications such as EMail is another.   Is scale in and of itself a 
centralization risk?

I'll pause here to simply say that I think there's a lot of work here 
both done and to be done.  I encourage you to keep going. This leads me 
to ask another question: what do you and others want to have happen with 
this draft?

Eliot

[1] 
https://foreignpolicy.com/2021/08/26/future-of-money-dollar-decline-central-bank-digital-currency-crypto-sanctions-china-russia/

[2] https://nvd.nist.gov/

[3] https://www.youtube.com/watch?v=XJYmyYzuTa8

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 14 Feb 2022 14:07:16 +1100

Hi Eliot,

> On 11 Feb 2022, at 7:01 pm, Eliot Lear <[email-masked]> wrote:
> 
> Hi Mark,
> 
> First I want to reiterate something I think I've said to you before: I truly admire your leading the charge in this area.  People may forget, but I certainly haven't, that you were raising these concerns as early as 2014, and maybe earlier.

Thanks - but I'm not sure I remember what you're referring to :)

> I think it gives us a strong indicator as to how much of §5 (modulo §5.6) you should include or how you should couch that section or otherwise organize the work.  There is, I think, a lot more to say in §4, and it may lead you to want to review how you discus/present the earlier sections.

Could you double-check your section references because there is no current 5.6...

> What would happen if a popular/dominant platform app store broke down or was compromised?  What enabled that form of centralization?  There are certainly cybersecurity benefits in having firm control over what code gets run on billions of end point devices, but there are also risks.  Isn't that a form of centralization?  This area is a whopper because from an IOT standpoint we tell all manufacturers to provide signed updates to their devices.  But right now as I write this to you, I am fuming over one platform provider stopping support update for a particular device.  Is there something to be said about trust anchors?

One area that I've been thinking about expanding upon is the notion that there are different constraints on substitution -- especially regarding time and effort (money). If changing to another provider of a function you need is expensive and/or very time-consuming (noting that different functions have different relevant timescales), you're effectively locked into it. 

However, it's important to ask if that is best characterised as 'centralization', or if it is just friction that makes a market less perfectly competitive, and therefore easier to centralise? I.e., should this be a top-level 'kind' of centralization in s 4, or is it merely a factor that can influence one of those kinds (likely, s 4.3)?

My initial though is that it's the latter. I want to resist labelling everything bad on the Internet as 'centralization'...

> There is an entire area of exploration one could enter around cyber-risks, scale, and centralization.  Certificates are a great area because we just saw a fail in which LE invalidated 2 million certificates because of a problem with TLS-ALPN-01.  Is 2 million a large # these days? Could it have been worse?  LE has turned the certificate market on its ear, but this has created challenges of its own.  The browser cert cache seems an interesting area to explore.  In fact there was a recent paper about the sort of garbage that accumulates in there.  I'm sorry that I can't find the cite for that, but I assert that stronger controls on those caches may lead to MORE centralization.

I just realised we have overlapping expansions of 'LE' -- Law Enforcement and Let's Encrypt. Hmm. 

One way (maybe not the right way) to think about this is that the scale isn't always relevant. If I try to access a Web site and it's down because of [some CA problem], it doesn't matter whether or not that CA has twenty customers or twenty million customers -- the resource that I need (whether I'm the client or server) is still down. So "big scale == bad centralisation" doesn't really help (especially when the bigger CA is -- to make a crude generalisation -- likely to be more available).

The interesting question for me is how this risk can be realistically mitigated -- using all of the potential tools including not only technical means (e.g., having backups/redundancy) but also social, economic, and legal (if a CA can take sites down it's bad in all three ways), and a combination of them (e.g., assuring that it's easy to switch).

> We have recently discussed Eduroam as a federation, by the way, and I think understanding its success factors and risks is also worthy of exploration, since it is, so far as I know, the largest Internet federation in existence.  Is there a centralization risk with Eduroam and do alternatives to Eduroam exist? (Hint: yes they do, but they're obviously smaller). What enables those alternatives? (Hint: it's an IETF protocol object.) This might argue for keeping some parts of §5.  Oh and that example is LOADED with intermediaries.

S 5.1 doesn't say that federation is doomed to centralization, FWIW. Again, as protocol designers we can't really constrain an *instance* of a federalisable (word?) protocol, whether it's Eduroam or any of the various chat platforms that use XMPP (last I heard). What we can do is make it possible to federate, and look for other levers to encourage it.

Note that I can't add my home wifi network to Eduroam (AIUI), so it's not _broadly_ federated anyway...

> Another area you might want to explore is the 3g/5g roaming architecture.  Are there centralization risks there?  If not, how were they mitigated?  BTW, they have their own centralized naming mechanisms.  There are SO many intermediaries in play here that truly I wonder how the system functions so well, but it does.

It's in interesting case, but it's informed by the telco way of working, which is pretty NOT Internet. That said, it might inform some expansion of 'inherited centralization'...

> There are also policy aspects you could delve into, at the risk of getting away from IETFy stuff and into crypto-currency examples.  You're somewhat there already in §4.3 (which I quite like).  For instance, the payment networks (Visa, Mastercard, AMEX,...) exhibit both centralization and decentralization aspects, and operate largely as federations.  Governments outside of the US HATE some of that.  What are the scale factors that drove those networks, and what has kept them in place?  I predict that would strengthen your point in §5.6.  There are a wealth of references for that.  You might want to have a look at [1] as a good start, but there's a lot of Visa/Mastercard envy out there.  Again, lots of intermediaries in play here.

This is explored pretty effectively in the WEAPONIZED-INTERDEPENDENCE reference.  

> The DNS example is enlightening.  That was a case where the centralized authorities for the name space (Jon and Jake) very much wanted to get out of the business.  And we have another brewing example of that: there have been ramblings that NIST might want to decentralize the National Vulnerability Database.[2]  Again, the central point in that case might have an interest in getting out of the operational side of things. In both of those cases, governments are involved and the motivations are often different from those of private concerns.[3]  That begins to help us zoom in on at least one factor in limiting centralization, but it's not a protocol aspect.
> 
> This leads me back to §5.6.  You touch on scale at several points in the document, but not in depth.  To me at least, it seems worthy of exploring centralization as the flipside of O'Dell's Axiom: “The only real problem is scaling.”  Cloud computing is one example.  CDNs are another.  Reputation management in the face of non-prearranged communications such as EMail is another.   Is scale in and of itself a centralization risk?

Perhaps (it'd help to know what you think 5.6 is :).

> I'll pause here to simply say that I think there's a lot of work here both done and to be done.  I encourage you to keep going.

Yes. I currently feel like the core of the document is substantially there, but there are a number of areas that could be extended upon / filled out / have more examples. At the same time, this document feels like it's best kept at a high level, and I'd like to keep it focused -- both to ease the reader's task, and to avoid going too far out on a limb. It's a tricky balance.

>   This leads me to ask another question: what do you and others want to have happen with this draft?

Personally, I'm torn between trying for the Independent stream and the IAB stream (and have had an informal chat with or two IAB members about the latter).  If it were to go to the IAB stream, I'd want to keep it focused on the relationship between the Internet standards community and centralization, and most definitely would NOT want it to become a kitchen sink for everything to do with centralization. 

Cheers,

> Eliot
> 
> [1] https://foreignpolicy.com/2021/08/26/future-of-money-dollar-decline-central-bank-digital-currency-crypto-sanctions-china-russia/
> 
> <OpenPGP_0x87B66B46D9D27A33.asc>

---

From: Eliot Lear <[email-masked]>
Date: Mon, 14 Feb 2022 10:15:43 +0100

Hi Mark,

On 14.02.22 04:07, Mark Nottingham wrote:
> Hi Eliot,
>
>> On 11 Feb 2022, at 7:01 pm, Eliot Lear<[email-masked]>  wrote:
>>
>> Hi Mark,
>>

You raised the risks of centralization at an open mic session at an IAB 
meeting around that time.

> Could you double-check your section references because there is no current 5.6...

Embarrassingly I seem to have borrowed from your -00 for that. You have 
since split §4 into two, and reorganized a bit.  My point is that I 
agree with you that there are indeed substantial limits to 
centralization.  However, see below.

> However, it's important to ask if that is best characterised as 'centralization', or if it is just friction that makes a market less perfectly competitive, and therefore easier to centralise? I.e.,…

There are (at least) two bodies of work that relate to your point.  
Karrenberg/Vest discusses European ISP concentration.  If you are 
considering locality as part of centralization, then market 
concentration is important.  I do think it would be interesting to see 
an update on the earlier work to see how things have changed, and why, 
but I don't see you needing to do that for this draft.

The second is the hourglass model itself.  What is the middle of the 
hourglass?  Previous work has shown, and we have all seen, that it is 
incredibly difficult to supplant that middle.  As RFC 7305 highlights, 
that is true not only for IP ([Akhshabi11]), but also in biology (see 
[Meyer13]).  This is relevant to your work only in as much as it relates 
to IPv4 versus IPv6. Hal Varian gave an excellent presentation in 2008 
about how v6 is NOT a substitutable resource for v4, which is a major 
reason why v6 has deployed at a snail's pace.  The question is whether 
that creates a centralization risk because of the cost of entry into IPv4.

I'm not so sure.  Just because I can't run a mail server in my home 
doesn't mean that I couldn't gain access to a wide selection of mail 
services.  This leads me to conclude that concentration is only a factor 
in as much as it (a) limits offerings from a very small number of 
players.  As your draft points out, there are indeed other 
considerations with mail.  There are a very small number of operating 
systems, and that market concentration to me is a far higher risk than 
anything else.

However, what I like about such discussion is that it gives you the 
opportunity to bound the work, not simply by saying, “here's where I 
stop”, but with a logical barrier.

> One way (maybe not the right way) to think about this is that the scale isn't always relevant. If I try to access a Web site and it's down because of [some CA problem], it doesn't matter whether or…

One might argue that LE are suffering from their own success.  As a 
not-for-profit, I don't think they ever intended to capture a 
substantial portion of the CA market, and yet they have.  In doing so 
one question is whether they are driving out other players. Arguably 
there are more important factors, like whether a large cloud provider as 
a trust anchor poses MORE of a risk or compounds that risk.  If we end 
up with a small number of players, the question then turns to what is in 
browser caches, and what the MTTR would be if a major player failed.  
Rinse and repeat for IOT devices and their update policies.

> The interesting question for me is how this risk can be realistically mitigated -- using all of the potential tools including not only technical means (e.g., having backups/redundancy) but also soc…

Yes, and this is where things get Ugly.  As I mentioned, there was a 
recent paper that pointed out all of the garbage that is ending up in 
people's certificate stores.  As we all know, there is a substantial 
habituation problem tied to cybersecurity risk with "Just Click Yes", 
and the browser people have done an admirable job in avoiding that.  
What this says to me is “let's not go there in the first place.”  So how 
can protocols like ACME foster good diversity and what is the role of 
private trust anchors?

> Note that I can't add my home wifi network to Eduroam (AIUI), so it's not _broadly_ federated anyway...

Well indeed!  What's interesting here is that the very basic Radius/EAP 
NAI provides for a means of federating that allows *anyone* to play from 
a protocol perspective.  That then moves us into the business 
discussion, and how usage gets paid for.  Eduroam has it easy in that 
universities are generally generous with their resources to visiting 
researchers and students so long as abuse is kept to a minimum.  Others 
may want other consideration, as allude.  What's interesting is that 
Eduroam is almost a counterargument to the notion that scale and 
centralization go hand-in-glove.  I write “almost” because they *have* 
captured the education sector.  I blame Klaas Wierenga (just because).

> This is explored pretty effectively in the WEAPONIZED-INTERDEPENDENCE reference.
Ah yes.  Sorry for not having read through.

> Yes. I currently feel like the core of the document is substantially there, but there are a number of areas that could be extended upon / filled out / have more examples. At the same time, this doc…

This is why I asked my question in the first place. I think there are a 
few "successes" where the risk of centralization has either abated or 
been minimized, but I am concerned that the advice you give in §6 may be 
premature, and may need a lot more exploration. As my friend Dave Meyer 
taught me, it's always good to leave something on the table for the next 
paper ;-)

> Personally, I'm torn between trying for the Independent stream and the IAB stream (and have had an informal chat with or two IAB members about the latter).  If it were to go to the IAB stream, I'd…

You are tackling a really important topic.  I've seen interest from many 
corners, and I hope the IAB will take that into account.  Let's face it: 
if AWS East or O365 going down can make the mainstream news, the topic 
is quite ripe.  But should that mean an input to a workshop, an RG, 
related conferences, an independent submission, or all of the above?

Eliot

[Meyer13] Meyer, D., "On the Complexity of Engineered Systems (and its 
effect on protocol deployment)", IAB ITAT Workshop, December 2013.

[Akhshabi11] Akhshabi, S., Dovrolis, C., "The Evolution of Layered 
Protocol Stacks Leads to an Hourglass-Shaped Architecture", Proceedings 
of the ACM SIGCOMM 2011.

---

From: Dirk Trossen <[email-masked]>
Date: Tue, 15 Feb 2022 16:40:49 +0000

Mark, all,

In relation to the 'blockchain' aspect in your draft, we have just released a draft looking into the impact of DLTs on provider aspects (see https://www.ietf.org/archive/id/draft-trossen-rtgwg-impact-of-dlts-00.txt). A number of challenges you point out, including privacy but also efficiency in particular, are the focus of this draft, also asking the question how network-level innovations may improve on the negative impact. This is just initial work, seeking feedback from others working on DLT to better understand those impacts and their dependency on the specific DLT approach being used.

As for the aspect of spreading consensus across distinct, often unknown actors, it might be worthwhile looking at DLT usage where the actors may well be under the same authority to realize distributed functionality, e.g., for storage purposes. This is indeed different from those applications where the aspect of distribution is the authority itself.

Best,

Dirk

-----Original Message-----
From: Architecture-discuss [mailto:[email-masked]] On Behalf Of Mark Nottingham
Sent: 10 February 2022 05:14
To: Brian E Carpenter <[email-masked]>
Cc: [email-masked]
Subject: Re: [arch-d] New Version Notification for draft-nottingham-avoiding-internet-centralization-02.txt

Cheers,

>>> Title:Centralization and Internet Standards Document date:2022-02-09 
>>> <https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-
>>> centralization-02.txt>
>>> Status: 
>>> <https://datatracker.ietf.org/doc/draft-nottingham-avoiding-internet
>>> -centralization/>
>>> Html: 
>>> <https://www.ietf.org/archive/id/draft-nottingham-avoiding-internet-
>>> centralization-02.html>
>>> Htmlized: 
>>> <https://datatracker.ietf.org/doc/html/draft-nottingham-avoiding-int
>>> ernet-centralization>
>>> Diff: 
>>> <https://www.ietf.org/rfcdiff?url2=draft-nottingham-avoiding-interne
>>> t-centralization-02>
>>> 
>>> Abstract:

---

From: Dirk Trossen <[email-masked]>
Date: Tue, 15 Feb 2022 16:46:27 +0000

...this is meant to read " impact of DLTs on provider networks" - apologies. 

-----Original Message-----
From: Architecture-discuss [mailto:[email-masked]] On Behalf Of Dirk Trossen
Sent: 15 February 2022 17:41
To: Mark Nottingham <[email-masked]>; Brian E Carpenter <[email-masked]>
Cc: [email-masked]
Subject: Re: [arch-d] New Version Notification for draft-nottingham-avoiding-internet-centralization-02.txt

Mark, all,

Best,

Dirk

-----Original Message-----
From: Architecture-discuss [mailto:[email-masked]] On Behalf Of Mark Nottingham
Sent: 10 February 2022 05:14
To: Brian E Carpenter <[email-masked]>
Cc: [email-masked]
Subject: Re: [arch-d] New Version Notification for draft-nottingham-avoiding-internet-centralization-02.txt

Cheers,

---

From: "Bajpai, Vaibhav" <[email-masked]>
Date: Wed, 16 Feb 2022 10:54:17 +0000

Hi everybody,

To provide some data-driven input to the discussion, thought to share 
our paper that came online yesterday. We present long-term trends and 
present state of Web consolidation by analysing massive datasets 
that are available in the public domain.

-- Vaibhav

----------------
An Empirical View on Consolidation of the Web

ACM DL - https://doi.org/10.1145/3503158
personal copy - https://bit.ly/3oPHYTc

abstract:

The majority of Web content is delivered by only a few companies that provide
Content Delivery Infrastructures (CDIs) such as Content Delivery Networks
(CDNs) and cloud hosts. Due to increasing concerns about trends of
centralization, empirical studies on the extent and implications of resulting
Internet consolidation are necessary. Thus, we present an empirical view on
consolidation of the Web by leveraging datasets from two different measurement
platforms. We first analyze Web consolidation around CDIs at the level of
landing webpages, before narrowing down the analysis to a level of embedded
page resources. The datasets cover 1(a) longitudinal measurements of DNS
records for 166.5 M Web domains over five years, 1(b) measurements of DNS
records for Alexa Top 1 M over a month and (2) measurements of page loads and
renders for 4.3 M webpages, which include data on 392.3 M requested resources.
We then define CDI penetration as the ratio of CDI-hosted objects to all
measured objects, which we use to quantify consolidation around CDIs. We
observe that CDI penetration has close to doubled since 2015, reaching a lower
bound of 15% for all .com, .net, and .org Web domains as of January 2020.
Overall, we find a set of six CDIs to deliver the majority of content across
all datasets, with these six CDIs being responsible for more than 80% of all
221.9 M CDI-delivered resources (56.6% of all resources in total). We find
high dependencies of Web content on a small group of CDIs, in particular, for
fonts, ads, and trackers, as well as JavaScript resources such as jQuery. We
further observe CDIs to play important roles in rolling out IPv6 and TLS 1.3
support. Overall, these observations indicate a potential oligopoly, which
brings both benefits but also risks to the future of the Web.
----------------

------------------------------
Vaibhav Bajpai, www.vaibhavbajpai.com

> On 10. Feb 2022, at 03:07, Mark Nottingham <[email-masked]> wrote:
> 
> Hi everyone,
>

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 16 Feb 2022 14:47:53 -0500

One general danger is that we don't have good metrics of which factors are
most important and which outcomes are most harmful. This can lead to the
"my least-favorite bad thing is the cause of this new problem", without a
lot of quantitative evidence. Or, conversely, "while my favorite good thing
may have something to do with the problem, I'm reluctant to name it since
they are the good guys".

The LE case below is a good example of "free" driving out diversity. We see
this in most areas - we get one or two dominant open-source implementations
of each protocol, and that basically terminates the market for building
others, in particular any commercial offerings, outside niche areas.
("Nobody was ever fired for recommending nginx as the webserver.") And
Google Android being "free as in beer" clearly has limited the potential
for alternative mobile operating systems not tied to a dominant hardware
vendor. But clearly, the emergence of well-tested and familiar open source
tools has been beneficial, both technically and economically. But for
reliability, it doesn't matter much if all versions of log4j are operated
by one entity or thousands - except that it's probably quicker to fix fewer
instances.

At this point, I suspect the best we (collectively) can do is to identify
factors and causality chains, but I think it's premature to say "here are
five dominant causes and if we fix these, the world will be better".

On Mon, Feb 14, 2022 at 4:16 AM Eliot Lear <[email-masked]> wrote:

> On 11 Feb 2022, at 7:01 pm, Eliot Lear <[email-masked]> <[email-masked]> wrote:
>
> Hi Mark,
>
> since split §4 into two, and reorganized a bit.  My point is that I agree
> with you that there are indeed substantial limits to centralization.
> However, see below.
>
>
> considering locality as part of centralization, then market concentration
> is important.  I do think it would be interesting to see an update on the
> earlier work to see how things have changed, and why, but I don't see you
> needing to do that for this draft.
>
> incredibly difficult to supplant that middle.  As RFC 7305 highlights, that
> is true not only for IP ([Akhshabi11]), but also in biology (see
> [Meyer13]).  This is relevant to your work only in as much as it relates to
> IPv4 versus IPv6. Hal Varian gave an excellent presentation in 2008 about
> how v6 is NOT a substitutable resource for v4, which is a major reason why
> v6 has deployed at a snail's pace.  The question is whether that creates a
> centralization risk because of the cost of entry into IPv4.
>
> services.  This leads me to conclude that concentration is only a factor in
> as much as it (a) limits offerings from a very small number of players.  As
> your draft points out, there are indeed other considerations with mail.
> There are a very small number of operating systems, and that market
> concentration to me is a far higher risk than anything else.
>
> opportunity to bound the work, not simply by saying, “here's where I stop”,
> but with a logical barrier.
>
> not-for-profit, I don't think they ever intended to capture a substantial
> portion of the CA market, and yet they have.  In doing so one question is
> whether they are driving out other players.  Arguably there are more
> important factors, like whether a large cloud provider as a trust anchor
> poses MORE of a risk or compounds that risk.  If we end up with a small
> number of players, the question then turns to what is in browser caches,
> and what the MTTR would be if a major player failed.  Rinse and repeat for
> IOT devices and their update policies.
>
> habituation problem tied to cybersecurity risk with "Just Click Yes", and
> the browser people have done an admirable job in avoiding that.  What this
> says to me is “let's not go there in the first place.”  So how can
> protocols like ACME foster good diversity and what is the role of private
> trust anchors?
>
> a protocol perspective.  That then moves us into the business discussion,
> and how usage gets paid for.  Eduroam has it easy in that universities are
> generally generous with their resources to visiting researchers and
> students so long as abuse is kept to a minimum.  Others may want other
> consideration, as allude.  What's interesting is that Eduroam is almost a
> counterargument to the notion that scale and centralization go
> hand-in-glove.  I write “almost” because they *have* captured the
> education sector.  I blame Klaas Wierenga (just because).
>
>
> few "successes" where the risk of centralization has either abated or been
> minimized, but I am concerned that the advice you give in §6 may be
> corners, and I hope the IAB will take that into account.  Let's face it: if
> AWS East or O365 going down can make the mainstream news, the topic is
> quite ripe.  But should that mean an input to a workshop, an RG, related
> conferences, an independent submission, or all of the above?
>
> Eliot
>
> Protocol Stacks Leads to an Hourglass-Shaped Architecture", Proceedings of
> the ACM SIGCOMM 2011.
>

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 16 Feb 2022 12:05:31 -0800

On Wed, Feb 16, 2022 at 11:48 AM Henning Schulzrinne <[email-masked]>
wrote:

> The LE case below is a good example of "free" driving out diversity.
>

Just as a level set... While it's true that LE is the largest CA in terms
of number of certificates issued, there are still quite a few CAs with
large market shares:
https://w3techs.com/technologies/overview/ssl_certificate

-Ekr

---

From: Christian Huitema <[email-masked]>
Date: Wed, 16 Feb 2022 13:16:17 -0800

> may have something to do with the problem, I'm reluctant to name it since
Yes indeed. Moral hazard and all that...
> by one entity or thousands - except that it's probably quicker to fix fewer

Henning, I am not sure about what you are really saying here. When nginx 
was developed, apache was the "dominant open-source implementation", yet 
nginx did succeed. Another example of such competition would be 
llvm/xlang versus gcc. Or the multiple projects competing with some 
success against openssl. For me the "log4j" example points at something 
else than concentration: the widespread practice of reusing open source 
components in large projects, without fully mastering the quality and 
security implications of additional dependencies.

But back to "free" versus "diversity". Yes, competing against free is 
rather hard. That's a business model issue. Any big system will incur 
costs and require funding. The big examples of free services seem to 
fall in three categories: funded by grants and donations (e.g., Signal); 
funded through a "freemium" model (e.g., Zoom); and funded by 
advertisements (e.g., Whatsapp). Aside from the conglomerate of 
Facebook, Instagram and Whatsapp, these are also examples of fairly 
competitive systems. I think this is an example where the technology 
does not lead to too much concentration, despite obvious network 
effects. The barrier of entry is low, with operating costs and revenues 
both growing with the number of users.

In contrast, we have examples like gmail in which a single free service 
pretty much blocks competition from anybody but Microsoft. This is a 
case where the barrier to entry is rather high, largely because the high 
investment required to deal with spam and other security issues. The 
operating cost do raise with the number of users, but not quite linearly 
-- spam control, for example, becomes easier when a larger number of 
users enables better gathering of information. The theoretical example 
of that is the pure software market, in which the marginal cost of an 
additional user is near zero. In that case, the dominant player has more 
revenue and can innovate at a faster pace than competitors with fewer 
customers. I wonder how many of our network protocols have that 
characteristic?

> five dominant causes and if we fix these, the world will be better".

Yes.

-- Christian Huitema

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 16 Feb 2022 18:13:11 -0500

> success against openssl.

Right - but for most protocols, the "market" of open-source implementations
would be considered highly concentrated or close to it, similar to (say)
the CDN or cloud market. For example, for web servers (
https://w3techs.com/technologies/overview/web_server), the HHI is
approximately 2,200. (This is a pure server count and is not weighted by
actual web usage, so this should be treated with caution.)

> For me the "log4j" example points at something

That, but also that log4j had become the dominant logging tool.

> both growing with the number of users.

Depends on what you define as concentration. For messaging, at least within
each geographic region, there are usually two dominant applications
(Whatsapp and iMessage, say, but you could also count three, via RCS), even
if, globally, the usage is more distributed across applications.

Same for videoconferencing - it's basically Zoom, WebEx, Microsoft Teams
and Google Meet, and then a long tail. Better than a monopoly, but similar
to the kind of concentration in cloud services or operating systems (across
all platforms), i.e., the typical "3 services have 3/4 of the market, the
rest split the remaining quarter". Two of these are loss-leaders for
platforms, not standalone money-making products.

> customers. I wonder how many of our network protocols have that

For consumer email, it's pretty similar to the others cited - three
providers (
https://email-verify.my-addr.com/list-of-most-popular-email-domains.php)
that split, in this case, pretty much exactly half the market by address
count. Thus, it's actually probably a bit less concentrated than (say) the
cloud or web server market.

Henning

---

From: Christian Huitema <[email-masked]>
Date: Wed, 16 Feb 2022 15:50:56 -0800

> cloud or web server market.
I trust your data. I saw different versions of that, but they were 
probably measuring something else than address count.

We should probably debate what metric we use to declare that a market is 
concentrated. If it takes just 3 providers to get to 50% market share 
but more than 100 to reach 90%, is the market really concentrated? I 
think that monopoly behavior happens when a dominant player has 80% of 
the market, that cartel-like behavior often happens when a market has 2 
or 3 dominant actors, and that competition thrives when there are 5 or 
more choices -- but I cannot quote research backing that on the top of 
my head.

Or, should we look at market dynamics? For example, the CDN market is 
pretty concentrated, but we saw Cloudflare emerge and get a rather big 
position, and more recently Fastly. I think that a market in which new 
player emerge and get a good business is a healthy market. By 
opposition, a market in which a few players keep the same dominating 
positions over time is probably ossified.

And of course, there are different consequences. For example, the DNS 
server market has 5 or 6 big players, so it looks more or less sane from 
an economic point of view. But then, having just one player suffer a big 
outage like Dyn did a few years ago had some pretty large impact on the 
Internet. Or, having a market ossify has pretty big impact on 
innovation. Or, having just a few big players in a communication field 
increases risks for censorship.

So maybe we should be really clear on what we are looking at.

-- Christian Huitema

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 16 Feb 2022 21:23:52 -0500

I'm using the US Department of Justice HHI thresholds.

> concentrated. If it takes just 3 providers to get to 50% market share but
> more than 100 to reach 90%, is the market really concentrated? I think that
> monopoly behavior happens when a dominant player has 80% of the market,
> that cartel-like behavior often happens when a market has 2 or 3 dominant
> actors, and that competition thrives when there are 5 or more choices --
> but I cannot quote research backing that on the top of my head.
>
> player emerge and get a good business is a healthy market. By opposition, a
> market in which a few players keep the same dominating positions over time
> is probably ossified.
>

There's a school of antitrust thought called dynamic competition (e.g.,
https://academic.oup.com/jcle/article/5/4/581/755200). It's favored, as one
might guess, by dominant incumbents since it's very hard to prove that
there won't be a future competitor ("a search alternative is only one click
away"). A countervailing notion is that while you do have some churn in the
early history of an industry, it tends to settle, with the US automobile
industry as an example - where you didn't really see changes until the
entry of Japanese competitors in the 1980s, i.e., the US market was "the
big three" (GM, Chrysler, Ford) from the early 1930s for about 50-60 years.

> server market has 5 or 6 big players, so it looks more or less sane from an
> economic point of view. But then, having just one player suffer a big
> Internet. Or, having a market ossify has pretty big impact on innovation.
> Or, having just a few big players in a communication field increases risks
> for censorship.
>

The outage impact seems separable - any player with more than trivial
market share is going to be noticeable unless they only supply some totally
niche market that nobody cares about, so this depends more on who they
serve as opposed to their size. (Don't know - Peloton outage?) Even a
Wordle outage would be news at this point, taking down half the Facebook
posts in its wake.

> So maybe we should be really clear on what we are looking at.

Agree.

Henning

---

From: Christian Huitema <[email-masked]>
Date: Wed, 16 Feb 2022 21:23:21 -0800

> I'm using the US Department of Justice HHI thresholds.
Looks like a good idea. mark, could we add that reference in some form 
to the next version of the draft?

Also, it would be really nice if we could collect a couple of statistics 
about market share for Internet services, and document whether the 
markets would be considered centralized according to criteria like the 
Herfindahl-Hirschman Index as used by the US DoJ Antitrust division 
(https://www.justice.gov/atr/herfindahl-hirschman-index). I think there 
is data available for a few interesting pieces, such as:

* web server software

* content distribution networks

* authoritative DNS servers

* recursive DNS resolvers

* email services

And probably a few more. I could see that as an appendix to the 
centralization draft.

-- Christian Huitema

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 25 Feb 2022 08:56:34 -0600

Hi, Christian,

I'm mostly reading this thread with interest, but on your question below
...

On Wed, Feb 16, 2022 at 3:16 PM Christian Huitema <[email-masked]>
wrote:

> customers.
> * I wonder how many of our network protocols have that characteristic?*
>

 I'm not sure what you're thinking of, re: network protocols, but if
transport protocols are included, I would think that QUIC (and the
constellation around QUIC, like MASQUE, etc.) might have that
characteristic (among many other characteristics).

Is that relevant to what you're thinking?

Best,

Spencer

---

From: Mat Ford <[email-masked]>
Date: Fri, 25 Feb 2022 16:25:21 +0000

> On 17 Feb 2022, at 05:23, Christian Huitema <[email-masked]> wrote:
> 
> 
> Looks like a good idea. mark, could we add that reference in some form to the next version of the draft?
> 
> Also, it would be really nice if we could collect a couple of statistics about market share for Internet services, and document whether the markets would be considered centralized according to criteria like the Herfindahl-Hirschman Index as used by the US DoJ Antitrust division (https://www.justice.gov/atr/herfindahl-hirschman-index). I think there is data available for a few interesting pieces, such as:
> 
> * web server software
> 

See https://pulse.internetsociety.org/centralization

We provide HHI measurements based on (W3Techs) surveys of top 10M, 10k and 1k websites for the following markets:

			10M	10k	1k
---------------------------------------------
Data center		212	754	957
Top Level Domain	3,004	3,437	4,493
SSL Certificate		2,873	3,401	3,819
DNS Server		507	1,422	1,316
CDN			6,593	3,947	3,724
Web Hosting		158	684	959

These results suggest TLD, SSL and CDN markets are all highly concentrated (which I don’t think will come as news to anyone). Also note that market concentration tends to increase when more popular content is the focus, except in the case of the CDN market.

Mat

> And probably a few more. I could see that as an appendix to the centralization draft.
> 
> -- Christian Huitema
>