Subject: [Sidrops] I-D Action: draft-ietf-sidrops-prefer-rrdp-01.txt

From: [email-masked]
Date: Fri, 22 Oct 2021 01:53:45 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the SIDR Operations WG of the IETF.

        Title           : Resource Public Key Infrastructure (RPKI) Repository Requirements
        Authors         : Tim Bruijnzeels
                          Randy Bush
                          George Michaelson
	Filename        : draft-ietf-sidrops-prefer-rrdp-01.txt
	Pages           : 10
	Date            : 2021-10-22

Abstract:
   This document formulates a plan of a phased transition to a state
   where RPKI repositories and Relying Party software performing RPKI
   Validation will use the RPKI Repository Delta Protocol (RRDP)
   [RFC8182] as the preferred access protocol, and require rsync as a
   fallback option only.

   In phase 0, today's deployment, RRDP is supported by most, but not
   all Repositories, and most but not all RP software.

   In the proposed phase 1 RRDP will become mandatory to implement for
   Repositories, in addition to rsync.  This phase can start as soon as
   this document is published.

   Phase 2 will start once the proposed updates are implemented by all
   compliant Repositories.  In this phase RRDP will become mandatory to
   implement for all compliant RP software, and rsync will be required
   as a fallback option only.

   It should be noted that although this document currently includes
   descriptions and updates to RFCs for each of these phases, we may
   find that it will be beneficial to have one or more separate
   documents for these phases, so that it might be more clear to all
   when the updates to RFCs take effect.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-sidrops-prefer-rrdp/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-prefer-rrdp-01

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-sidrops-prefer-rrdp-01

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 22 Oct 2021 11:03:20 +0200

Dear working group,

We just uploaded a new version of the prefer-rrdp draft.

The document has had some updates based on working group member comments
and discussion between the co-authors. The uploaded version still lacks
final review by my co-authors, but.. after discussing we decided that it would
be good to replace the now expired -00 with this. This also makes it easier
for WG members to comment.

This version is not intended to be final. It's intended to support further
discussion. So feel free!

Important changes compared to -00:
- We no longer talk about post-rsync in this document
   (this can be done as a later and separate exercise)
- We included some more text on rsync fallback

I would like to ask you all to give it a read provide feedback.

Kind regards
Tim

> On 22 Oct 2021, at 10:53, [email-masked] wrote:
> 
> 
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Job Snijders <[email-masked]>
Date: Sun, 24 Oct 2021 17:58:52 +0200

Dear all,

On Fri, Oct 22, 2021 at 11:03:20AM +0200, Tim Bruijnzeels wrote:
> This version is not intended to be final. It's intended to support
> further discussion. So feel free!
> 

Before applauding and promoting RRDP and slapping ourselves on the
back... shouldn't we first attempt to resolve some flaws in the RRDP
protocol? I know some of us are aware how trivial it is to use RRDP to
cause RPs to have a very bad day.

In retrospect it seems "Fear, Uncertainty, and Doubt" about rsync was an
integral part of pushing the RRDP agenda. This has led to the
unfortunate situation where some of us may have gotten blindsided by the
new issues introduced by RRDP. Let's debunk some of the myths I've seen
floating around:

Lack of implementations? Seems there are multiple choices:
    https://rsync.samba.org/ (C), https://www.openrsync.org/ (privsep C),
    https://acrosync.com/ (C++).

Security? ... I believe the modern day RRDP implementations may have
    received less scrutiny and review than good ole' rsync. Having found
    vulnerabilities in some RRDP implementations myself, I'm going to
    chalk the 'security' argument up as a red herring.

Speed? In practise I oftentimes see RSYNC outperform RRDP. It should not
    come as a surprise that unmarshalling XML-wrapped Base64 encoded
    blobs and multiple HTTP roundtrips cost something somewhere.

Consistency? Similar to how data inconsistencies can be observed on
    poorly managed rsync servers, it is possible (and has happened in
    practise) to mess up the RRDP streams too. RRDP as a protocol does
    very little to help detect/overcome inconsistencies.

Debugability? Arguably direct immediate access via the rsync:// URIs to
    specific individual object beats having to download a snapshot or
    finding the right delta to get to the Signed Object of interest.

DoS? Comparing an anycasted HTTPS service to an unicasted RSYNC service
    is comparing apples and oranges. If folks truly were concerned about
    this, surely they can anycast the rsync service too?

Performance? It come to light that the world's largest repositories
    were served from terribly slow (wrong) backends, another issue at
    some point in time was lack of simple directory sharding in some
    repositories. It would be foolish to blame the protocol instead of
    the operator who simply misconfigure the service.

Both RRDP and RSYNC appear to bring unique problems and features to the
table. It seems to me that the best path forward is to encourage
everyone to run both rsync and rrdp. Eroding support for rsync, or
suggesting RRDP is an adequate replacement might harm the ecosystem.

Mono cultures have strong failure modes. Until a THIRD fetching
mechanism is defined (RRDP v2, or a flooding protocol like BGP?), I
believe we need both RSYNC and RRDP.

We should actively encourage operators to deploy both protocols.

Kind regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Sun, 24 Oct 2021 19:24:15 +0200

> On 24 Oct 2021, at 17:58, Job Snijders <[email-masked]> wrote:
> 

Actually.. that is exactly what this document says.

Currently RRDP is optional. This document seeks to make it mandatory,
and indeed preferred. Earlier versions of this document went further,
but this has been removed.

W.r.t. flooding using BGP. Geoff Huston has been advocating this idea
for some time and I agree that this should be investigated properly. I
think it could be a good option to flood the *relevant* data for BGP
in the messages themselves. To make this work we would need to do a
serious amount of work: e.g. a CA-router protocol so that relevant signed
data can easily be added, and perhaps and RTR++ like offloading of
received crypto information to validation software..

Still I think this should be the longterm game, because intrinsically
having to download all information from repositories, where most information
is irrelevant to the actual route, and any CA or repository can send RPs
on a wild goose chase is problematic. I don't think it an RRDP v2, 3 or 16
can solve it.

Tim

---

From: Job Snijders <[email-masked]>
Date: Sun, 24 Oct 2021 20:01:39 +0200

> Actually.. that is exactly what this document says.

Section 2 "Motivation" first goes on to spread FUD about rsync, and then
concludes that RRDP isn't perfect either. Eh.... yes, ok?
If the authors wish to make RRDP and RSYNC 'more equal', you'll either
have to be honest about the problems in RRDP, or the remove text
suggesting rsync is not adequate.

> but this has been removed.

I don't really see the point of making it mandatory, especially when the
benefits of RRDP appear to be somewhat exaggerated. Arguably RSYNC is
the simpler of the two protocols, making for a great mandatory
lowest common denominator. RRDP being optional, and preferred if
available (aka the current situation) seems to strike a good balance. No
new draft needed?

> for some time and I agree that this should be investigated properly.
> I think it could be a good option to flood the *relevant* data for BGP
> serious amount of work: e.g. a CA-router protocol so that relevant
> signed data can easily be added, and perhaps and RTR++ like offloading
> of received crypto information to validation software..

I'm skeptical about this working group's ability to pull off RPKIoBGP.
There are very few (too few?) people who work on *both* BGP and RPKI
implementations, *and* also have operational experience with both
technologies.

Hands-on experience with software development and BGP routers is a key
prerequisite for this type of work.

> on a wild goose chase is problematic. I don't think it an RRDP v2, 3 or 16

Yes, the fundamental flaw is that RRDP/RSYNC both are 'pull' protocols.
This means incentives are misaligned: in BGP if you send trash to a
peer, the peer will disconnect you, which in turn hurts the bad actor.
This is a wonderful property. On the other hand in the RPKI, there
barely are any repercussions for CAs when sending bad stuff to RPs.

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Mon, 25 Oct 2021 14:41:12 -0700

> mechanism is defined (RRDP v2, or a flooding protocol like BGP?),

speaking of mono-cultures and insanely tightly bound interdependence.

randy

---

From: Job Snijders <[email-masked]>
Date: Mon, 25 Oct 2021 23:44:53 +0200

> speaking of mono-cultures and insanely tightly bound interdependence.

Yes, I appreciate you waving a warning flag here and do not ignore it.

What about a separate flooding mesh, in spirit of MSDP?

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Mon, 25 Oct 2021 14:47:25 -0700

> What about a separate flooding mesh, in spirit of MSDP?

if we need three solutions to a problem that it is not clear that we
have, the mind boggles at how many we would need for a real problem. :)

randy

---

From: Job Snijders <[email-masked]>
Date: Mon, 25 Oct 2021 23:48:28 +0200

> have, the mind boggles at how many we would need for a real problem. :)

I'm arguing that three solutions need to exist, before one of those three
can be deprecated.

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Mon, 25 Oct 2021 14:52:24 -0700

> I'm arguing that three solutions need to exist, before one of those
> three can be deprecated.

and why not two or five?  i.e. what is the logical argument here?

btw, i agree that today's rrdp is weakly specified and often poorly
implemented.  but this does not make me want to rush to an Nth weakly
specified and poorly implemented protocol.

randy

---

From: Job Snijders <[email-masked]>
Date: Tue, 26 Oct 2021 00:02:11 +0200

> and why not two or five?  i.e. what is the logical argument here?

I argue the ecosystem needs at least two transports. Multiple
transportation options hopefully increases overall availability and
reliablity.

> implemented.  

Then why prefer it?

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Mon, 25 Oct 2021 20:21:03 -0700

> I argue the ecosystem needs at least two transports.

which it has

> Multiple transportation options hopefully increases overall
> availability and reliablity.

so a dozen would be lovely?  where is the trade-off curve?  and how many
will be mti?

perhaps this path is a bit of a mess.

>> btw, i agree that today's rrdp is weakly specified and often poorly

social compromise.

rand

---

From: Martin Hoffmann <[email-masked]>
Date: Tue, 26 Oct 2021 12:21:54 +0200

Job Snijders wrote:
> > > I'm arguing that three solutions need to exist, before one of
> > > those three can be deprecated.  
> > 

On the other hand, two transports definitely create additional
complexities in implementing software; deployment and operation of
repositories, and troubleshooting of inconsistencies in data sets.

I am not sure I agree that this cost is worth a hypothetical increase
in availability.

M.

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 27 Oct 2021 12:40:36 +0200

Job, group,

> On 24 Oct 2021, at 20:01, Job Snijders <job=[email-masked]> wrote:
> 

I understand that you have a different perspective, but by using the
term FUD I believe you are second-guessing my, and other people's
motivation. I prefer that we can have this discussion while respecting
each other's viewpoint.

The fist motivation is based on the experience (and measurements and
experiments) of repository operators. In the RPKI there is an asymmetry
in the amount of RPs vs repositories. With *recursive* rsync the server
needs to invest CPU and memory in a dialogue with each client to work
out the optimal minimal delta, i.e. optimise for bandwidth.

This is a scaling problem where server CPU and memory form a bottleneck.
Yes, you can invest lots in multiple servers, multiple PoPs etc, but
this asymmetry does not change fundamentally.

What RRDP does, is that it protects the server. By letting the server
publish immutable data (except for the notification file) the work is
pushed to the client side, where it can scale in parallel. You don't
have to use a CDN to benefit. A simple https proxy such as nginx can
serve these files from memory, and the server bottleneck is bandwidth,
not CPU/memory. The bandwidth bottleneck can be mitigated further by
using a CDN, or your own distributed caching proxies. This is magnitudes
easier and cheaper than scaling up rsync.

So, while RRDP may not always give an immediate performance benefit to
the clients, it does give great benefits to the server. The biggest
benefit to the RPs is indirect: the server is just much less likely to
become unavailable due to load or scaling issues.

To the second point. The rsync library support may not be an issue
for rpki-client, and the new openrsync implementation. But it is an
issue that has been felt by many other RP implementers.

All this said, if it helps to have a more constructive discussion
on the content of suggested changes to RFCs in this document, by
leaving the motivation out of the document itself, then that also
works for me.

> available (aka the current situation) seems to strike a good balance. No

First of all, let me remind you that this is a planning document. It has
been stated multiple times that it may never become RFC. If it is overtaken
by events then this is a feature, and not a bug.

That said, the document contains suggestion for updates to existing RFC
that I believe are still relevant:

- Fallback from RRDP to rsync

When RRDP was specified the idea was that it would be highely available
and equivalent to rsync content. Fallback was optional in the RRDP RFC. As
mentioned in 3.1 of prefer-rrdp the word 'could' was used in this context.

Since then the WG has made it clear that they want fallback to be mandatory.
I am not arguing against that here, but I believe that means that we still
need updates to the RRDP RFC to make this clear. And you will note that
there is some additional text, based on operator feedback, to ask RPs to
use preventive measures to prevent that they all fall back at the same time.

Furthermore it was found, that while it was clearly stated that the RRDP
files MUST be available - this availability was treated as optional in the
experiment that demonstrated that not all RPs perform the fallback to rsync.
Because of that I believe that an update to 6481 is also warranted to make
it clear to repository operators that all access mechanisms should be
treated as first class citizens.

- Mandatory for RPs

As said, the benefits of RRDP are mostly on the server side, but for that
repository operators need all RPs to support and prefer it. This seems
to be the case today, but since this support is not mandatory this could
change at any time. Because of this I think we need something like the
text suggested in section 3.3.

- Mandatory for repositories

This is discussed in section 3.2. Personally I care less about this,
because RPs need to support fallback to rsync anyway. I still think
that repositories would do very well to protect themselves by having

>>> We should actively encourage operators to deploy both protocols.

I am not sure that you actually disagree with this.

Kind regards,

Tim

---

From: Job Snijders <[email-masked]>
Date: Wed, 27 Oct 2021 13:48:22 +0200

On Wed, Oct 27, 2021 at 12:40:36PM +0200, Tim Bruijnzeels wrote:
> With *recursive* rsync the server needs to invest CPU and memory in a
> dialogue with each client to work out the optimal minimal delta, i.e.
> optimise for bandwidth.
> 

You are holding it up-side down. The RIRs and NIRs are in an excellent
position to absorb some of the computational cost. It is an excellent
property that server and client have a dialogue to optimize the
transfer. Why offload all cost to the RPs?

> What RRDP does, is that it protects the server.

Yes, and zero protection for RPs.

> the clients, it does give great benefits to the server.

I appreciate you admitting that RRDP is not performant.

> - Fallback from RRDP to rsync

Given the fact that RRDP is optional, it logically follows a fallback to
rsync is mandatory. Because rsync is the only mandatory-to-implement
protocol.

I don't see the purpose of this document. 

Kind regards,

Job

---

From: Ties de Kock <[email-masked]>
Date: Wed, 27 Oct 2021 14:06:28 +0200

Hi Job,

> On 27 Oct 2021, at 13:48, Job Snijders <job=[email-masked]> wrote:
> 

I would at least say that it gives some protection to RPs. After you (rightfully)
pointed out that RPs should do TLS validation, at least it prevents MITM attacks
where an attacker could selectively remove CAs from validation.

Pulling off the same for RRDP is harder (TLS certificates) and publicly visible
(CT logs).

I do acknowledge that RRDP has several weaknesses. A naïeve implementation
acts as if the server is trusted, and while that may have been feasible without
delegated RPKI, by now, it is a risk.

Kind regards,
Ties

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 27 Oct 2021 15:08:28 +0200

> On 27 Oct 2021, at 13:48, Job Snijders <[email-masked]> wrote:
> 

You forgot to quote a bit from my email:
>> The biggest benefit to the RPs is indirect: the server is just much less
>> likely to become unavailable due to load or scaling issues.

I don't think there is a need to be overly dramatic about the cost for
RPs. The delta is actually pre-calculated by the server.

The remaining cost is mainly because of the size of some notification
files, which has grown excessive in the wild. This is being addressed.

The main challenge for an RRDP server is to get the balance right
in the size of the notification file and the amount of deltas to support.
If the server keeps too many deltas then the download and parsing cost
becomes unreasonable. If it keeps too few then RPs who update infrequently
will be forced to use snapshots.

> Yes, and zero protection for RPs.

ahh.. the straw man, nice

> I appreciate you admitting that RRDP is not performant.

ahh.. the straw man, nice

> I don't see the purpose of this document. 

Except that 8182 specifically says 'could'. If you think that fallback is
mandatory then I don't see why would object to an update to 8182 that makes
this clear.

---

From: Job Snijders <[email-masked]>
Date: Wed, 27 Oct 2021 15:48:44 +0200

> RPs. The delta is actually pre-calculated by the server.

You appear to operate under the assumption that RRDP servers will always
behave correctly, and in the best interest of RPs. This doesn't hold
true. Software defects, bit flips, or even malicious attacks are
possible. 

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Wed, 27 Oct 2021 10:53:28 -0700

> experiments) of repository operators.

where can we find these data?  would be useful to cite it in the next
rev of draft-ietf-sidrops-rpki-rov-timing :)

at the 10,000km level, all PPs experience the same RP query profile.  i
am not aware of load issues being experienced by operators of PPs.
would love to hear of measurements.

randy

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 28 Oct 2021 10:34:03 +0200

> On 27 Oct 2021, at 19:53, Randy Bush <[email-masked]> wrote:
> 

https://www.ietf.org/proceedings/86/slides/slides-86-sidr-2.pdf
https://www.ietf.org/proceedings/89/slides/slides-89-sidr-6.pdf

> would love to hear of measurements.

Please feel free to add your own.

But bear in mind that the current load on a small PP where the majority
of RPs will use RRDP if it is available, is not the same as the potential
load of all RPs querying a large PP. The RIPE NCC repository today is 287MB
in 73k objects, and looking at how it has been growing, it can be expected
to get bigger. Especially if BGPSec would take off and/or ASPA is added.