Subject: Re: [Sidrops] I-D Action: draft-ietf-sidrops-prefer-rrdp-00.txt

From: Ties de Kock <[email-masked]>
Date: Fri, 26 Mar 2021 10:43:15 +0100

Hi all,

Following a discussion in the chat during the SIDROPS session, I think it is
better to take this to the list.

Currently, RRDP clients could (RRC8182, 3.4.5) fall back to alternative
repository access mechanisms (i.e. rsync) when there were any issues when
retrieving RPKI data from an RRDP repository server.

If this will change into a MUST without considering the timing of the fallback,
this will have an impact on the availability of rsync repositories. Let me build
the case of why RRDP clients falling back to rsync will act as a thundering herd
and will cause fallback to fail, and how we can mitigate that.

We observe that rsync clients update infrequently (~hourly) and RRDP clients
update frequently. The majority of clients in our logs check notification.xml
every 10 minutes. We also see that HTTP failures are temporally correlated,
clients detect a failure (in CDN or backend) at the same time. From this, it
follows that clients will observe a failure at the same time and will fall back
at the same time.

Scaling rsync for the total number of relying party instances, when spread out,
is achievable, even for a large repository. However, scaling rsync so it is
available when all clients fallback in a short period (we see ~1170 unique ips
retrieve notification.xml in a 10 minute window) is difficult.

Fallback will happen rarely, which implies that their copy of the repository is
potentially (very) outdated. A clean copy from rsync takes ~10-15 CPU-seconds
per client (and potentially a large number of IOPS hitting the backing storage)
on the server-side.

That is why I would like the draft to include text that recommends that fallback
is spread out. I would propose that when RRDP is failing (due to content or
connectivity), the clients retry RRDP until a set time at which they fall back
to rsync.

For a repository operator, it is optimal if all clients are spread out over
time. For fallback, this means that a client keeps trying rrdp for a
random delay from the uniform distribution of 0...rsync_update_interval after
the first failure before falling back to rsync.

The alternative is that, when clients act like a thundering herd, the rsync
repositories will not be available for an extended period. If clients retry at a
very similar interval, the repositories will be overloaded for an extended time.

Please let me know what you think of this mechanism,

Ties

> On 23 Feb 2021, at 08:44, Tim Bruijnzeels <[email-masked]> wrote:
> 
> Dear working group,
> 
> I have asked for 15 minutes on this subject at the upcoming WG session.
> 
> Differences in this version:
> - rename 'deprecate-rsync' -> 'prefer-rrdp' - as this is the first goal (phases 0-1-2)
> - discuss deprecate rsync as a post-phase-2 goal
> - start discussion of what would be needed to have transport agnostic names (no rsync in uris)
> - moved implementation status overview to the end
> 
> 
> Regards,
> Tim
> 
> 
> 
>> On 23 Feb 2021, at 00:45, [email-masked] wrote:
>> 
>> 
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>> This draft is a work item of the SIDR Operations WG of the IETF.
>> 
>>       Title           : Resource Public Key Infrastructure (RPKI) Repository Requirements
>>       Authors         : Tim Bruijnzeels
>>                         Randy Bush
>>                         George Michaelson
>> 	Filename        : draft-ietf-sidrops-prefer-rrdp-00.txt
>> 	Pages           : 13
>> 	Date            : 2021-02-22
>> 
>> Abstract:
>>  This document formulates a plan of a phased transition to a state
>>  where RPKI repositories and Relying Party software performing RPKI
>>  Validation will use the RPKI Repository Delta Protocol (RRDP)
>>  [RFC8182] as the only mandatory to implement access protocol.
>> 
>>  The first objective is to make RRDP the preferred access protocol,
>>  and require rsync as a fallback option only.  This will greatly
>>  reduce the operational burden and concerns for RPKI repository
>>  operators.
>> 
>>  In phase 0, today's deployment, RRDP is supported by most, but not
>>  all Repositories, and most but not all RP software.
>> 
>>  In the proposed phase 1 RRDP will become mandatory to implement for
>>  Repositories, in addition to rsync.  This phase can start as soon as
>>  this document is published.
>> 
>>  Once the proposed updates are implemented by all Repositories phase 2
>>  will start.  In this phase RRDP will become mandatory to implement
>>  for all RP software, and rsync will be required as a fallback option
>>  only.
>> 
>>  It should be noted that although this document currently includes
>>  descriptions and updates to RFCs for each of these phases, we may
>>  find that it will be beneficial to have one or more separate
>>  documents for these phases, so that it might be more clear to all
>>  when the updates to RFCs take effect.
>> 
>>  Furthermore, this document currently includes an early discussion of
>>  a future objective, which would be to change the RPKI standards such
>>  that names in RPKI objects are no longer tightly coupled to rsync.
>>  By using transport independent names and validation, we will obtain
>>  the agility needed to phase out rsync altogether and/or introduce
>>  other future access protocols.
>> 
>> 
>> The IETF datatracker status page for this draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-sidrops-prefer-rrdp/
>> 
>> There are also htmlized versions available at:
>> https://tools.ietf.org/html/draft-ietf-sidrops-prefer-rrdp-00
>> https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-prefer-rrdp-00
>> 
>> 
>> Please note that it may take a couple of minutes from the time of submission
>> until the htmlized version and diff are available at tools.ietf.org.
>> 
>> Internet-Drafts are also available by anonymous FTP at:
>> ftp://ftp.ietf.org/internet-drafts/
>> 
>> 
>> _______________________________________________
>> Sidrops mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/sidrops
> 
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Job Snijders <[email-masked]>
Date: Fri, 26 Mar 2021 17:15:42 +0100

On Fri, Mar 26, 2021 at 10:43:15AM +0100, Ties de Kock wrote:
> However, scaling rsync so it is available when all clients fallback in
> a short period (we see ~1170 unique ips retrieve notification.xml in a
> 10 minute window) is difficult.

Have you considered using a TCP proxy application like 'haproxy' to
distribute rsync clients across multiple rsync backends? One can even
anycast rsync servers to distribute load!

> Fallback will happen rarely, which implies that their copy of the
> repository is potentially (very) outdated. A clean copy from rsync
> takes ~10-15 CPU-seconds per client (and potentially a large number of
> IOPS hitting the backing storage) on the server-side.

Unless the validator constructs a cache as if the data was fetched via
rsync... It is possible to make a validator that'll have a 'warm' cache
when connecting to rsync after having connected to RRDP a few times.

The RIPE RPKI repository currently is only ~ 200 megabytes, I project
that in the next 4 years it won't reach 1 gigabyte. This is such a small
size that one can easily serve this from RAM in frontend caches and not
waste IOPS from 'backing storage'.

It appears Randy came up with a simple suggestion:
https://www.ietf.org/rfcdiff?url2=draft-ietf-sidrops-rpki-rov-timing-03

another consideration:

The rpki-client default installation on OpenBSD/Ubuntu/Debian will be
launched at a random minute of the hour, and will (for the lifetime of
the governing cron/systemd process) fetch at that choosen minute of the
hour.

Validator implementations which are started in 'oneshot' mode
(rpki-client, 'routinator vrps', 'fort --mode=standalone') should
immediately try rsync when RRDP fails for one reason or another.

A 'oneshot' implementation should not insert a 'sleep' between a failed
rrdp fetch and an attempt to connect using rsync.

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Fri, 26 Mar 2021 10:39:04 -0700

> clients detect a failure (in CDN or backend) at the same time.

would this not be distributed over the ten minute poll distribition?

randy

---

From: Randy Bush <[email-masked]>
Date: Fri, 26 Mar 2021 10:46:26 -0700

> anycast rsync servers to distribute load!

it is inappropriate to place undue financial and ops burden on the
operator of a publication point because we can not design protocols.

> https://www.ietf.org/rfcdiff?url2=draft-ietf-sidrops-rpki-rov-timing-03

if you mean jittering fallover, credit to ggm.  but it is not clear to
me that it is actualy needed if the rrdp clients are not in convoy.

> the governing cron/systemd process) fetch at that choosen minute of the

yes, copied from drl's.  but those are arbitrary implementations, not
protocol specifications.

> rrdp fetch and an attempt to connect using rsync.

so you argue against ggm's jittered fallover delay?  maybe give us more
of a hint as to the motivation here.

randy

---
[email-masked]
`gpg --locate-external-keys --auto-key-locate wkd [email-masked]`
signatures are back, thanks to dmarc header butchery

---

From: Job Snijders <[email-masked]>
Date: Fri, 26 Mar 2021 22:57:26 +0100

reminder, this is the 'sidr operations group of 2020/2021'.

> copied from drl's. but those are arbitrary implementations, not
> protocol specifications.

I understand a desire to phrase ideas 'in the abstract' and talk about
protocols as a sequence of communication steps between 'autonomous
systems', rather than reference individual software processes by their
brand name. However, some in this working group are implementing without
any perspective on the entire inter-domain system, and they might
benefit from guidance spelled out in terms such as 'oneshot' and
'server' modes.

To complicate things further, in some RPKI ROV deployments the
'fetcher', 'validator', and 'RTR server' are conflated into a single
software process, which can muddy dialogue at times.

> of a hint as to the motivation here.

you hint here https://mailarchive.ietf.org/arch/msg/sidrops/LrNY0v3dy6WQDIrtzazHjhgygh0/

My observation is that unavailability of RRDP service indicates nothing
about the state and availability of a CA Repository's RSYNC service.

The only important thing for me (as Relying Party) is to only spend
bandwidth and CPU on fetching via RSYNC, iff I was NOT able to
successfully fetch from the CA's RRDP service.

Whether the RRDP or RSYNC fetch were successful or not, in the next
fetching event (which is expected to occur between 30 and 60 minutes
later) I should first try fetching from RRDP URIs discovered in the
objects fetched via RSYNC (or fetched via RRDP). If the RRDP fetching
process resulted in some kind of error (HTTP 404, 5xx, connection
refused, DNS problem, etc), I should try to fetch the data via RSYNC
service.

As the world's Relying Parties' software instances were not all booted
at the exact same moment, logically, the collective's fetch events are
not synchronized to any other particular minute of the hour. This is an
excellent characteristic of the existing global RPKI deployment! :)

In retrospect it seems it might have been a mistake to ever even hint
RRDP could be polled every minute.

[ process thought:
    We now have RFC 6481, RFC 8182, draft-ietf-sidrops-prefer-rrdp,
    draft-ietf-sidrops-rov-timing, and now these emails. Confusion might
    be growing instead of being reduced...

    ... would the working group perhaps benefit from a Finite State
    Machine (FSM) specification in which fetches via both RRDP and RSYNC
    protocols are described as events and various timers are also
    described? ]

Is the below the short summary people agree to?

	- Try RRDP first
	- Immediately fall back to RSYNC if RRDP didn't work
	- Don't contact RRDP URIs more than once every 10 minutes
	- Don't contact RSYNC URIs more than once every 30 minutes
	- Don't contact RRDP or RSYNC URIs less than once ever 60 minutes

Kind regards,

Job

---

From: Randy Bush <[email-masked]>
Date: Fri, 26 Mar 2021 20:16:17 -0700

> brand name. However, some in this working group are implementing
> without any perspective on the entire inter-domain system, and they
> might benefit from guidance spelled out in terms such as 'oneshot' and
> 'server' modes.
> 

yup.  indeed that is why we do abstract protocol specs.

>> so you argue against ggm's jittered fallover delay?  maybe give us
>> more of a hint as to the motivation here.
> 
> you hint here
> https://mailarchive.ietf.org/arch/msg/sidrops/LrNY0v3dy6WQDIrtzazHjhgygh0/

i confess to trolling.  as we were taught in first year

    a manhattenite has a lover in brooklyn and one in the bronx (hint:
    they are in opposite directions).  at the manhattan station, both
    trains come every ten minutes.  but our friend finds that, if they
    take the first that comes, they visit the lover in brooklyn far more
    frequently than the one in the bronx.

    it’s the interleave.  the one to brooklyn comes two minutes before
    the one to bronx.  so there is an eight minute window for brooklyn
    and a two minute window for the bronx.

i have not yet seen a case that rrdp exhibits convoy behavior

> My observation is that unavailability of RRDP service indicates
> nothing about the state and availability of a CA Repository's RSYNC
> service.

except in extreme cases such as destination unreachable.  but i take
your point.

> successfully fetch from the CA's RRDP service.

i suspect you could be made to regret "only important."  we would prefer
rrdp; when it works.

> objects fetched via RSYNC (or fetched via RRDP).

yep

> If the RRDP fetching process resulted in some kind of error (HTTP 404,
> 5xx, connection refused, DNS problem, etc), I should try to fetch the
> data via RSYNC service.

yep

> excellent characteristic of the existing global RPKI deployment! :)

yes.  as i said, i have yet to see an argument for convoy phenomena.

> RRDP could be polled every minute.

yep

>     protocols are described as events and various timers are also

in the words of vince perriello, send code :)

> 	- Don't contact RRDP or RSYNC URIs less than once ever 60 minutes

i certainly do not agree to that last line, and doubt you do

as far as the other numbers, see draft-ietf-sidrops-rpki-rov-timing for
my current opinion.  but i am willing to listen.

randy

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 29 Mar 2021 18:33:17 +0200

Hi,

> On 27 Mar 2021, at 04:16, Randy Bush <[email-masked]> wrote:
> 

@Job, can you elaborate why you believe that RRDP should not be contacted more frequently than once every 10 minutes?

The idea behind allowing more frequent checking IFF 'if-modified-since' is used (and quite possibly other relevant headers), is that: A) CDNs would not even blink, B) you have a chance of getting updates out more quickly - e.g. you issued a wrong ROA and want to correct it.

Are you worried about stampeding RPs in case they all check this frequently and then fall back together? I.e. there would be less of a spread? Because if this is the main reason, then I would argue that the fix would be to fall back more carefully, rather than tell people not to check RRDP as often.

Ultimately the timing between the RPs and the repositories are only one step in the chain. To get your updates from your intended changing of BGP to other validating routers at least the following steps are involved.
1- Let your RPKI CA update ROAs
2- Let your RPKI CA publish
3- Wait for RPs to synchronise RRDP/rsync
4- Have other routers pick up the changes RTR or other

Ideally I think you would want the whole chain of 1-4 to typically take an amount of time that is comparable to / acceptable wrt BGP convergence. The numbers above talk about step 3 in this chain. I think it's hard to say how much it can or should be optimized - also considering that the other steps take time. My gut feeling says that the 2 minute range for checking - when everything works - is about right. Don't get me wrong, I am not fundamentally opposed to 10 minutes, but I want to understand the reason.

As an aside:
Ultimately, as others have pointed out before me, the only real way to be sure that routers have all the information they need, when they need it would be to look into shipping relevant information in BGP itself. This deals with timing issues, and is much safer with regards to repo availability - the info is in the mesh and decentralized. Obviously this is not feasible in the short term as it's a great deal of work. We would need to define validation for relevant info possibly TLS like with relevant info, perhaps think RTR like integration so that BGP speakers can offload signing to a local signer, as well as validation (perhaps let them use both central repos for bootstrapping as well as updated info in BGP).. many, many discussion I am sure. Still, I think this would be a good path to consider long-term.

Back to now though.. and the more practical question: why 10 minutes?

Regards,

Tim

---

From: Job Snijders <[email-masked]>
Date: Mon, 29 Mar 2021 21:25:13 +0200

> > my current opinion. but i am willing to listen.

The 24-hour sliding timers we generally observe in certificate issuance,
if combined with fetching RPKI data once a day, can cause needless
brittleness in the face of transient issues between the CA and RP.

With that in mind, I think all of us agree that syncing only once a day
is not good. However, encouraging RP implementers to sync every minute
also appears to result in problematic load.

Perhaps my command of English language is lacking, what I meant to say
with "Don't contact RRDP or RSYNC URIs less than once ever 60 minutes"
is that RPs are encouraged to try to sync with the repositories at least
once an hour, or maybe at least every 3 hours - regardless of the
syncing protocol.

In draft-ietf-sidrops-rpki-rov-timing we appear to be addressing two
groups of problems: those who are polling too fast and those who are
polling too slow. If we can get the global RPKI deployment to a state
where operators see effects in the Default-Free Zone of newly issued (or
revoked) objects within 1 or 2 hours, that would be an excellent
achievement.

Historic perspective:
Changes via 'LOA based' BGP prefix filtering would take days or weeks.
With IRR in many places the cycle was brought down to 24 hours.
With RPKI, hopefully we can achieve 'a few hours'.
The industry is making progress compared to years ago!

> @Job, can you elaborate why you believe that RRDP should not be
> contacted more frequently than once every 10 minutes?

> Are you worried about stampeding RPs in case they all check this
> frequently and then fall back together? 

No, that does not worry me.

As suggested here [0] an RP which previously was able to sync with the
CA via RRDP (or rsync), does not need to do a 'full rsync transfer' as
if the local cache is empty. Coming down 'from RRDP to RSYNC' does not
need to be expensive on the RSYNC server, if the RPs don't pretend they
have zero prior knowledge.

> My gut feeling says that the 2 minute range for checking - when
> everything works - is about right. Don't get me wrong, I am not
> fundamentally opposed to 10 minutes, but I want to understand the
> reason.

Some in the community appear to think that sticking to a two minute
cycle is a silly level of load for benefit. Changing the timer to 10
makes load grow 5x slower. I agree with this position.

It seems to me the collective first went 'too slow' in the RSYNC-only
world, and then RRDP came along and the gas pedal was pressed to the
floor. The timers in the RRDP spec being so different from the RSYNC
timers, and it being somewhat underspecified how exactly an RP
implementer should make the RSYNC and RRDP world co-exist, appear to
have led to implementation challenges in most validators.

> Back to now though.. and the more practical question: why 10 minutes?

Because every two minutes seems too fast. Such fast paced polling
intervals combined with some other (not yet discovered) software defect
can easily lead to significant load. The following github issue
describes an example of short poll timers (combined with another issue)
exacerbating a situation. https://github.com/RIPE-NCC/rpki-validator-3/issues/307

I understand the global CDNs can carry any load, but it would be nice if
small people like myself can also host their RPKI objects and
participate in the Internet from a small device. I think RPs have an
obligation to be as efficient as possible and minimize the load in all
regards.

Obviously a push protocol like BGP would resolve a lot of the issues
arising from the current polling-based mechanisms. That's for another
year.

Kind regards,

Job

[0]: https://mailarchive.ietf.org/arch/msg/sidrops/GJv01_9Nm_hZwoVNLxwbACfrPPk/