Subject: Re: [arch-d] Is consolidation a protocol property?

From: Vittorio Bertola <[email-masked]>
Date: Mon, 2 Aug 2021 10:46:13 +0200 (CEST)

> Il 31/07/2021 02:10 Tommy Pauly <tpauly=[email-masked]> ha scritto:
> 
> I am sympathetic to the fact that consolidation is an interesting question when taken in the right context, but I would argue that it is a matter of *deployment* models, not protocols themselves. There can of course be protocol elements that are required to make certain deployment models possible, but there are many instances where deployment models do not require standardized protocols.

However, there are many instances where they do, and in that case it would be good if the IETF kept up with its own promises (like the RFC 3935 excerpt that Martin already quoted) and made sure to release protocols only when they have all the features that are necessary to support decentralized deployment models, and not just the centralized ones (which indeed tend to be simpler to conceive and deploy, and so will naturally tend to be technically available before the decentralized ones).

Then, of course there are vendors that - either because of a conscious business choice, or simply because they don't care about the issue - will not implement the protocol features that allow more decentralization. That is indeed something that if necessary should be dealt with elsewhere, possibly at the regulatory level, but it's not a reason not to have those features since the beginning.

More generally, the IETF needs to decide whether non-technical features of protocols (which inevitably include business models) are in scope. I've often heard people saying that we need to conceive protocols so that they offer the right business incentives, or they will not get implemented; and this is true. Consolidation effects are a necessary part of that discussion as well; a protocol may be conceived so to create business incentives for dominant players, which will promote its speedy implementation by companies that have the capital and the market share to make it quickly ubiquitous, but will also promote more consolidation, and less diversity both in the IETF and in the market.

-- 
Vittorio Bertola | Head of Policy & Innovation, Open-Xchange
[email-masked] 
Office @ Via Treviso 12, 10144 Torino, Italy

---

From: Jari Arkko <[email-masked]>
Date: Mon, 2 Aug 2021 15:39:01 +0300

Watson,

#include <what-martin-and-christian-said>

I think there’s agreement that consolidation and centralisation are largely economic issues (and maybe legal issues in some cases). Those aspects are not in IETF domain of expertise. 

However, not all topics fall neatly in separate areas. Consolidation and centralisation do in some cases have aspects that relate to technology. Such as:

- interoperability (what interfaces standardised? what parties can participate in the protocol?)
- pervasive monitoring (e.g., creating targets that increase the risk of pm)
- resilience
- user security impacts and security mechanisms (e.g., which parties have access to which data)

I think these are obviously fair game for discussion. In my view IETF works best when it considers multiple viewpoints and tries to understand what’s best for the Internet. Of course, it is not always clear what the answer is but we need to have those discussions.

My interpretation of the discussion in OHTTP was that it was not so much about whether consolidation in general but rather about whether the intended (or eventual) use cases would benefit (or suffer) from additional interoperability in the form of discovery*.

Jari

*) There might have been other topics to discuss beyond discovery, too, e.g., whether some guidance is needed for the construction of the actual payload HTTP requests, as those are likely to carry much richer data, including potentially user identifying data or fingerprinting opportunities. And that would have been important for items 2 and 4 above. But we didn’t get there because discovery was too popular angle.

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Aug 2021 16:10:37 +0200

The IETF concern around consolidation is one of resilience, and that 
sword cuts both ways.  On the one hand, if we were to sum the amount of 
uptime and downtime, and average that across the user base, I would be 
hard pressed to find an enterprise that can manage a service with the 
level of uptime that the Big Guys offer.

On the other hand, *every* service will at some point suffer a failure, 
and the scope of that failure for Big services will be wider than any 
individual failure.  And that is something that the IETF should consider.

There are some architectural axioms here:

  * Centralized enumerate services will have a larger scope of failure
    than discovered localized services, even if they have far a lower
    MTBF per capita.
  * Services not used, like muscles, will atrophy over time.
  * Failover can only occur at scale when the resources to handle that
    failover are available.
  * A distributed infrastructure can suffer a system wide failure in an
    implementation monoculture.

Now let's look at ohttp.  If small numbers of proxy services were used 
for general purposes, that would create a bottleneck around which 
failures might cause some issues.  But if the client merely falls back 
to not using ohttp, resiliency can be maintained, because very roughly 
speaking the same amount of resources on client and server are used with 
or without the ohttp proxy.  It's even a little bit cheaper on the 
server when people fall back because there's less encryption.  That is- 
there is no risk of atrophy in the system.

In the case of resolver queries, it's the opposite.  If resolver queries 
move from one discovered distributed infrastructure to an enumerated 
centralized infrastructure, then all three axioms kick in, and we are 
set up for a large scope failure at some point in time.  And that is why 
ADD is so important to get right.

Eliot

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 2 Aug 2021 16:38:36 +0200

Eric

I don't think that what you cited from my email violates that "impersonal discussions"
recommendation. What i said is purely about business goals of adopters of
technology and how they are sponsoring work in the IETF.  We are continuously
arguing about technologies by mean of "what is the use-case", and "where are the IETF
participants representing interest in this use case" and assessments such as "how will
this ever make money" or "Oh, this looks like a single vendor use-case interest, go away".
AFAIK, what i said is no different than what is being said in any of those discussions. 

Furthermore, we do attempt to design technical solutions against non-technical
metrics/goals: Privacy is no different as such a metric or goal as
freedom of user-choice, transparency, auditing or enabling/disabling of collusion.

Btw: If you think i am misrepresenting the business goals driving the proposed
technology, please explain. As written to the ohttp mailing list, without a
comprehensive description of scenarios, one is left guessing.

Cheers
    Toerless

On Fri, Jul 30, 2021 at 06:56:26PM -0700, Eric Rescorla wrote:
> On Fri, Jul 30, 2021 at 6:34 PM Toerless Eckert <[email-masked]> wrote:
> 
> > Ultimately, it is left up to the IETF community to make this a goal.
> > A community which is nowadays very much dominated by those
> > against whose interests such a goal would be. Which showed clearly
> > in this weeks meeting.
> >
> 
> It would be great if we could keep this conversation on a technical level,
> as indicated by S 2 of RFC 7154 and especially point 2.
> 
> -Ekr

-- 
---
[email-masked]

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Aug 2021 07:41:20 -0700

On Mon, Aug 2, 2021 at 7:11 AM Eliot Lear <[email-masked]> wrote:

> centralized infrastructure, then all three axioms kick in, and we are set
> up for a large scope failure at some point in time.
>

I don't think this is obviously true at all.

Consider, as an example, a design in which clients just iteratively tried
the main public resolvers (say, Google, CF, and Quad9) until it got an
answer [0]. Why do you believe that is more likely to produce a large scale
failure -- absent such a catastrophic failure that nothing would work --
than the current situation?

As an aside, none of the major client-side DoX deployments presently have
the failure mode you indicate:

- Firefox explicitly falls back to Do53 on resolution failure
- Chrome and Microsoft simply determine when the existing resolver is
likely to support secure transport.

-Ekr

[0[ This is probably not a great idea for other reasons, but that's not
relevant for now.

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Aug 2021 07:48:46 -0700

> AFAIK, what i said is no different than what is being said in any of those

I think most of the people working on O-HTTP have been clear that they
don't think
centralization is a good thing.  Your implication to the contrary on the
basis that you think
their employers benefit from centralization is what I'm talking about here.

-Ekr

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 2 Aug 2021 17:10:07 +0200

There was some good consumer reporting on the June 8 failure, nothing
how it primarily affected big web services vs. not most of the smaller
ones. Most consumers would not even understand any of these differences.

In the USA, FCC investigate quite well failures of the 911 system,
publishes reports of problem analysis and hands our useful fines that
may help operators to build better resilience into their operations.
I have a reference to one such report in rfc8994. Similar oversight
is of course applied to other critical infrastructure operations such
as the power grid.

As far as i understad it, web operations is not considered a criticial
infrastructure and as such there is no oversight or not even mandated
resilience metrics. And with consumers not being able to really have freedom
of choice, competition (as little as there is) is of no help.

Wrt to atropy: Within web-scale experiences, there is ongoing
invisible guinea-pigging of users into failure cases to harden th
system. Would be fun to imagine if/how this could be operationalized
with users. Something like a scheduled fire-alarm test... Please browse
now, but be aware you likely do not have all the privacy for the next
10 minutes... Not sure how well that can work.

Cheers
    Toerless

On Mon, Aug 02, 2021 at 04:10:37PM +0200, Eliot Lear wrote:
> The IETF concern around consolidation is one of resilience, and that sword
> cuts both ways.  On the one hand, if we were to sum the amount of uptime and
> downtime, and average that across the user base, I would be hard pressed to
> find an enterprise that can manage a service with the level of uptime that
> the Big Guys offer.
> 
> On the other hand, *every* service will at some point suffer a failure, and
> the scope of that failure for Big services will be wider than any individual
> failure.  And that is something that the IETF should consider.
> 
> Now let's look at ohttp.  If small numbers of proxy services were used for
> general purposes, that would create a bottleneck around which failures might
> cause some issues.  But if the client merely falls back to not using ohttp,
> resiliency can be maintained, because very roughly speaking the same amount
> of resources on client and server are used with or without the ohttp proxy. 
> It's even a little bit cheaper on the server when people fall back because
> there's less encryption.  That is- there is no risk of atrophy in the
> system.
> 
> centralized infrastructure, then all three axioms kick in, and we are set up
> for a large scope failure at some point in time.  And that is why ADD is so
> important to get right.
> 
> Eliot
> 

> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss

-- 
---
[email-masked]

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 2 Aug 2021 10:21:33 -0500

On this point in Martin's email,

On Fri, Jul 30, 2021 at 7:57 PM Martin Thomson <[email-masked]> wrote:

> I generally agree with what Tommy has said here, though I will note that
> the question can be important to the design of a protocol.  Privacy Pass
> have had this problem wherein aspects of the design might have limited the
> number of actors that can provide certain key functions in the system.  I
> think we have a reasonable solution there that doesn't result in any
> inherent bias toward having a limited set of actors, but it is existence
> proof of the issue.
>
> FWIW, any PKI would be another great example of inherent centralization
> bias.
>
> I tend to think that, like anything else in this space, awareness of the
> potential problem and discussion of its effects on designs is OK.  There is
> probably no single thing that can be said on the topic that applies in all
> cases, except what has already been said:
>
>    We embrace technical concepts such as
>    decentralized control, edge-user empowerment and sharing of
>    resources, because those concepts resonate with the core values of
>    the IETF community.
>
> -- RFC 3935, Section 4.1
>
> There is, of course, value in exploring specific cases in more detail and
> trying to extract common lessons.  I haven't seen any truly compelling work
> in that space yet.  It's hard.  People are still finding their way.
>

I agree that this would be valuable. Is there an obvious venue for that
exploration? I found choosing a home (PANRG) to be very useful for a
somewhat similar exploration that led to
https://datatracker.ietf.org/doc/rfc9049/ ("looking at specific proposals
for path-aware IETF protocols and trying to extract common lessons about
what goes wrong").

I understand the "hard" and "still finding their way" parts, but if people
knew where to cluster while they're fumbling in the dark, that might be
helpful.

Best,

SPencer

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 2 Aug 2021 17:55:57 +0200

On Mon, Aug 02, 2021 at 07:48:46AM -0700, Eric Rescorla wrote:
> don't think centralization is a good thing.  Your implication to the contrary on the
> basis that you think their employers benefit from centralization is what I'm talking about here.

I did not say that, i do not think that and i am not sure how you can think
that would be a logical implication. Maybe because to me, it is clear that
my personal opinions are not necessarily the same as the commercial goals 
of my employer. But when they are different wrt. to any of my IETF positions,
then i am very open about that stating e.g. when i like a proposal but can not
support it in the IETF for commercial reasons.

As said on the ohttp mailing list, it would be great to be very upfront to
document the actually targeted commercial scenarios and then have a discussion
about which alternative scenarios would require what additional/alternative
technology components and what problems we wouldn't know how to solve.

Yes, a lot of such work might be a lot more speculative or even research,
but if we all agree that decentralization is a good goal, then it would be
good to not only give it lip-service, but do  good faith effort to explore
and document this . And if contributors have to say they can not contribute to that
part of the effort because they have no sponsorship for their time for this,
that is perfectly fine.

Cheers
    Toerless

---

From: Eliot Lear <[email-masked]>
Date: Mon, 2 Aug 2021 21:10:48 +0200

Hi Eric,

I should have caveated my note, just a bit.  There are quite a few 
assumptions, not the least of which is how resolvers handle 
provisioning.  I make a basic economic assumption that overprovisioning 
is based on existing load, and that DDOS load is a factor of existing 
load- that's one I can't talk to.  But if those assumptions hold...

On 02.08.21 16:41, Eric Rescorla wrote:
>
> On Mon, Aug 2, 2021 at 7:11 AM Eliot Lear <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     In the case of resolver queries, it's the opposite.  If resolver
>     queries move from one discovered distributed infrastructure to an
>     enumerated centralized infrastructure, then all three axioms kick
>     in, and we are set up for a large scope failure at some point in
>     time.
>
>
> Consider, as an example, a design in which clients just iteratively 
> tried the main public resolvers (say, Google, CF, and Quad9) until it 
> got an answer [0].

If a big fish tries to flop to smaller fish, it's a lot different than 
the other way around.

> - Firefox explicitly falls back to Do53 on resolution failure

A little fish could probably get away with that, but in the scenario I'm 
outline, not only am I assuming that you guys are good at what you do, 
but so is everyone else, and such a failure happens rarely – in fact 
considerably more rarely than an  SP failure.  Let's also presume wider 
deployment of Do[TH]->enumerated  list (otherwise, who cares?).  
Economic efficiencies dictate that Do53 capacity would drop over time 
due to match demand, or at best stay flat in the face of increasing 
demand of DoH.  Big fish flops, say, 7 years from now.  Where's the 
excess capacity to handle that load?

Now... this can be remediated.  Here are a few ways:

  * Establish elastic backup contracts.  Someone would have to foot the
    bill on failure, but let's presume that it is a solvable problem,
    particularly for a big fish.  Now it's a matter of adding capacity
    to match load.  In a cloud-based world, this doesn't strike me as
    beyond the realm of possibility.
  * Here's an amusing way: hit the existing do53 servers with a query
    for each doh query, but just ignore the answer.  That way the do53
    servers maintain capacity.  That might not be considered friendly,
    tho, and it has privacy implications.  Of course, nothing says the
    QNAME in do53 has to be related to the QNAME in Do[HT] ;-)
  * Find a way to establish trust in the existing infrastructure. This
    is where I think ADD is going, tho we have a ways to get there.

The point is that the standards are enabling policies around who gets 
the queries at a macro level.  Standards can also enable policies to 
distribute that load.

Eliot