Subject: [Sidrops] RPKI fencing limits (was: Tree Hints)

From: Job Snijders <[email-masked]>
Date: Wed, 10 Nov 2021 14:52:34 +0100

Dear all,

(Geoff, NIST - suggested todo items at the bottom of this email :-)

On Wed, Nov 10, 2021 at 10:50:05AM +0100, Tim Bruijnzeels wrote:
> Documenting the decisions as rpki-client did helps. There is also
> merit in a public discussion about limits and motivations. I think
> different PR implementations can benefit from sharing. It may also
> help operators if limits are documented somewhere and RPs give clear
> errors when limits are applied.

To elaborate a bit more on the current limits and behavior of
rpki-client. Some of the current limits (with annotation comment)

    $ grep MAX extern.h
    #define MAX_IP_SIZE             200000	/* RFC 3779 IP entries */
    #define MAX_AS_SIZE             200000	/* RFC 3779 AS entries */
    #define MAX_URI_LENGTH          2048	/* length in bytes */
    #define MAX_FILE_SIZE           4000000	/* length in bytes */
    #define MAX_MANIFEST_ENTRIES    100000	/* largest manifest is ~ 16K */
    #define MAX_CERT_DEPTH          12		/* longest allowed auth chain */
    #define MAX_RSYNC_PROCESSES     16		/* concurrent processes */
    #define MAX_REPO_PER_TAL        1000	/* mitigation 'issue H' */
    #define MAX_REPO_TIMEOUT        (15 * 60)	/* timeout in seconds */

(We are happy to consider additional limits, or change limits.)

Let's zoom in on how exactly MAX_REPO_PER_TAL works, as "issue H"
appears unresolved in some implementations.

rpki-client processes repositories in order of 'closeness to the root'.
After MAX_REPO_PER_TAL (1000) publication points have been discovered,
no new publication points will be added to the processing queue. The
children of a misbehaving CA always will be 'further away' from the root
than the RIR hosted CAs or nominally operating delegated CAs (like
chloe.sobornost.net, rpki-repo.registro.br, rpki-repository.nic.ad.jp).
Consequently, 'good CAs' are stay alive, 'bad CAs' are punished by
virtue of being further removed from the root.

The combination of processing in a specific order, and hard-capping at a
(generous) limit seems to make for a pragmatic and hopefully fairly
effective defense against the "issue 1" Koen brought up here [1].

Looking forward
===============

When 'fencing in the RPKI' (restricting the RPKI) - there are many
dimensions to consider (wall clock time, file sizes, width, depth, etc).
If we operate under the assumption that RPs are upgraded once a year, it
would be very helpful to establish growth projections for the next 1, 2
and 3 years.

Geoff - perhaps you'd be willing to look in your crystal ball, just like
you seem to enjoy predicting internet-scale FIB sizing? :-)

NIST, perhaps your team can also produce projections based on the RPKI
Deployment Monitor data, so that we have two sources to help set course?

The strategy here is to make an inventory of all aspects of the RPKI
that can 'balloon up', and then impose a limit on those things which
permit organic growth (in the next 18 months), but not much more than
that.

This working group has the right people to make this approach work.

Kind regards,

Job

[1]: https://mailarchive.ietf.org/arch/msg/sidrops/FCttiTVor2IpXuqiGljuoN66Xio/

---

From: <[email-masked]>
Date: Thu, 11 Nov 2021 16:09:36 +0100

This is an interesting approach. To me it feels like there might be an
implicit assumption that the deeper down the tree one goes, the less
important the objects become. This seems to me as limiting the freedom that
RPKI provides, which, especially in the advent of IPv6, is a choice that, I
believe, if it were to be made, should be consciously made with wide
consensus, rather than be the consequence of an implementation. Take the
following case: a federal government has an IPv6 prefix, and delegates part
of it to its states, who delegate parts to the districts [1].

>From a practical point of view: imagine I can create ~900 repositories
(which is not unlikely based on current RPKI trees), and I make it so that
they would all finish just before timeout, which is defined at 15 minutes.
Then if I wanted to, I could make them take just under those 15 minutes.
That would mean it takes 900 * 15 * 60 = ~9 days if run sequentially. Now if
we use 16 parallel processes (which is the maximum amount of parallel rsync
processes you define), it still takes *14 hours*. This is without processing
trickery (large ROAs, etc.), rsync start-up times, etc. which would make it
take even longer. Is this really "resolved"? I would argue that everything
over an hour is too long (I believe rpki-client also makes this assumption
with the "killme" timer). Similarly, I am not 100% sure what MAX_FILE_SIZE
means, but if it means the maximum size of each file in the repository, then
I can have a  MAX_MANIFEST_ENTRIES * MAX_FILE_SIZE size repository, which is
400 GB. I can have 900 of them, thus 360 TB. These are exactly the type of
situations I meant when I mentioned that heuristics are difficult.

These considerations are why I believe a dynamic approach might be more
suitable, and more futureproof, than fixed limits based on future
predictions. 

Koen van Hove

[1] https://www.nro.net/wp-content/uploads/2010-burger.pdf#page=4

-----Oorspronkelijk bericht-----
Van: Job Snijders <[email-masked]> 
Verzonden: woensdag 10 november 2021 14:53
Aan: [email-masked]; [email-masked]
Onderwerp: [Sidrops] RPKI fencing limits (was: Tree Hints)

Dear all,

> errors when limits are applied.

To elaborate a bit more on the current limits and behavior of rpki-client.
Some of the current limits (with annotation comment)

    $ grep MAX extern.h
    #define MAX_IP_SIZE             200000	/* RFC 3779 IP entries */
    #define MAX_AS_SIZE             200000	/* RFC 3779 AS entries */
    #define MAX_URI_LENGTH          2048	/* length in bytes */
    #define MAX_FILE_SIZE           4000000	/* length in bytes */
    #define MAX_MANIFEST_ENTRIES    100000	/* largest manifest is ~ 16K
*/
    #define MAX_CERT_DEPTH          12		/* longest allowed auth
chain */
    #define MAX_RSYNC_PROCESSES     16		/* concurrent processes */
    #define MAX_REPO_PER_TAL        1000	/* mitigation 'issue H' */
    #define MAX_REPO_TIMEOUT        (15 * 60)	/* timeout in seconds */

rpki-client processes repositories in order of 'closeness to the root'.
After MAX_REPO_PER_TAL (1000) publication points have been discovered, no
new publication points will be added to the processing queue. The children
of a misbehaving CA always will be 'further away' from the root than the RIR
hosted CAs or nominally operating delegated CAs (like chloe.sobornost.net,
rpki-repo.registro.br, rpki-repository.nic.ad.jp).
Consequently, 'good CAs' are stay alive, 'bad CAs' are punished by virtue of
being further removed from the root.

Looking forward
===============

When 'fencing in the RPKI' (restricting the RPKI) - there are many
dimensions to consider (wall clock time, file sizes, width, depth, etc).
If we operate under the assumption that RPs are upgraded once a year, it
would be very helpful to establish growth projections for the next 1, 2 and
3 years.

Geoff - perhaps you'd be willing to look in your crystal ball, just like you
seem to enjoy predicting internet-scale FIB sizing? :-)

The strategy here is to make an inventory of all aspects of the RPKI that
can 'balloon up', and then impose a limit on those things which permit
organic growth (in the next 18 months), but not much more than that.

Kind regards,

Job

[1]:
https://mailarchive.ietf.org/arch/msg/sidrops/FCttiTVor2IpXuqiGljuoN66Xio/

---

From: 'Job Snijders' <[email-masked]>
Date: Thu, 11 Nov 2021 17:17:02 +0100

Hi,

On Thu, Nov 11, 2021 at 04:09:36PM +0100, [email-masked] wrote:
> I can have 900 of them, thus 360 TB. 

Ah, another limit interested parties should be aware of: rpki-client
disconnects from RRDP servers which try to send more than 4 gigabytes.

error message: "parse failed - content too big"

> These considerations are why I believe a dynamic approach might be
> more suitable, and more futureproof, than fixed limits based on future
> predictions. 

Noted!

Kind regards,

Job

---

From: Jeroen Massar <[email-masked]>
Date: Thu, 11 Nov 2021 17:57:34 +0100

> On 20211111, at 16:09, [email-masked] wrote:
> 
> important the objects become.

The tree will never really be that deep anyway.

If you take IPv6:

IANA -> RIR -> LIR -> End-User

Thus even if there is "PI" involved, much more, even more than double, will unlikely happen, especially as we are talking about routes in the DFZ.

> This seems to me as limiting the freedom that

Please note that these limits have to be added because of your disclosure...

Greets,
 Jeroen

---

From: Job Snijders <[email-masked]>
Date: Thu, 11 Nov 2021 18:06:27 +0100

On Thu, Nov 11, 2021 at 05:57:34PM +0100, Jeroen Massar wrote:
> > This is an interesting approach. To me it feels like there might be
> > an implicit assumption that the deeper down the tree one goes, the
> > less important the objects become.
> 

We cannot differentiate maliciously deep data hierarchies and
non-maliciously deep data hierarchies. This means that people building
non-maliciously deep trees... are being maliciously silly.

We must discourage them from doing so. That discouragement comes in the
form of their ROAs not being generated, because heuristic considers that
part of the subtree too costly.

Kind regards,

Job

---

From: Geoff Huston <[email-masked]>
Date: Fri, 12 Nov 2021 00:24:33 +0000

I hear you Job. It is something I will look at further in the coming weeks - thanks for the suggestion / hint! 

In the meantime, you are probably already aware of https://grafana.wikimedia.org/d/UwUa77GZk/rpki?from=now-30d&orgId=1&to=now&viewPanel=59, but for other WG members who have not seen this URL, I find this a fascinating timeseries on the time spent performing a pass across the RPKI publication point system to check for and gather updates from the perspective of one relying party.

regards,

Geoff

---

From: Ties de Kock <[email-masked]>
Date: Fri, 12 Nov 2021 08:52:03 +0100

> On 11 Nov 2021, at 17:57, Jeroen Massar <jeroen=[email-masked]> wrote:
> 
> 

The tree gets deeper in practice because there are internal levels of CAs at
RIRs (and other locations).

For example, for the RIPE NCC tal:
  1: RIR 'tal' 0/0 offline certificate
	2: 0/0 online certificate
	3: CA containing members
	4: Member CA.

I see multiple reasons for this number of levels to increase. If an RIR were to
add the possibility for a member to have both a hosted + delegated CA, this would
likely add a level (or potentially hit the object size limit with it's manifest
during a key-roll). There are also reasons to structure the "CA containing
members"  as multiple CAs. For easier processing and updating (less objects),
or because the CA is structured based on where the resources originated.

But the critical realisation is that in the tree, for RIRs with NIRs, NIRs
"look like" yet another CA under the "CA containing members". If you perform a
breadth-first traversal, an abusive CA that is a direct member is as important
as a CA under an NIR.

If the order is unlucky (how is the order determined? alphabetical by filename?),
the abuser can prevent the NIR from being processed.

> Please note that these limits have to be added because of your disclosure...

These limits have been added as a stopgap to fix vulnerabilities. I would
welcome work from a researcher. Irregardless of you how you look at the handling
of the CVD by the party running it (NCSC) - this is better than finding about
these issues at Monday morning 3AM.

Kind regards,
Ties

> 
> Greets,
> Jeroen
>> 
> 
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Claudio Jeker <[email-masked]>
Date: Fri, 12 Nov 2021 09:55:52 +0100

> 	2: 0/0 online certificate

These extra CAs are just adding a constant offset to the depth of the
tree. They do not really matter.
 
> or because the CA is structured based on where the resources originated.

Even with this you would end up with 2 CAs per delegation.
So the math is something like 3 + 2*delegations. For rpki-client the max
depth is 12. Which in turn would be 4-5 additional member delegations. This
should be plenty to do NIR, LIR, PI delegation. Especially since using 2 CAs
per delegation is not the common case.
 
> the abuser can prevent the NIR from being processed.
 
The order does not matter in a breadth first scheme all resources of level
n+1 are downloaded before resources of level n+2.
So the bad CA will not stop the NIR from loading.
The only problem is that because of parallel processing the bad CA may run
ahead of slow CAs and hit the limit before the other CAs finished. That is
not good but it only stops the sync of repositories.
So even then, all ROAs directly referenced from those NIR will be validated
and added to the VRP tree. The external CAs form that tree may not have
been able to sync but the system will fallback to its cached data and
validate with that. So in the end most ROAs will be added to the VRP tree.
So no, the abuser can not prevent the NIR from being processed.
Also since this is a limit per TAL all other TALs are not affected.
 
> these issues at Monday morning 3AM.

It was Wednesday 3AM for some people when the CVD was released so how
does that change anything? You run important infrastructure you better
monitor it and keep it up to date.

The problem here is that RP software needs to download untrusted data to
do its work. This is an inherent design issue of RPKI and it opens a large
attack surface to all of RPKI.

-- 
:wq Claudio

---

From: Stephen Kent <[email-masked]>
Date: Thu, 11 Nov 2021 10:28:18 -0500

Koen,
> important the objects become.
"important' is not a meaningful term here. However, it is accurate to 
say that as one descends further down the RPKI tree, the address space 
range available to allocate becomes smaller, relative to the cert path 
in question. That's a security feature.
> consensus, rather than be the consequence of an implementation.
"freedom" also seems like an odd term here. what do you mean?

Steve

---

From: Alex Band <[email-masked]>
Date: Fri, 12 Nov 2021 14:23:02 +0100

Hi Geoff,

> On 12 Nov 2021, at 01:24, Geoff Huston <[email-masked]> wrote:
> 

I look at these time series pretty religiously. I’m happy to see that when everything is humming along normally, even today validation runs take less than 50 seconds to complete, even though there are now 43 repositories to traverse. 

While the graph on Wikimedia is certainly interesting, I recently updated the Grafana template for Routinator to include all the metrics added in v0.9 so you can easily get these views yourself. In addition to the original Trust Anchor metrics, there are now also per-repository metrics to allow an in-depth look and troubleshooting, as well as separate metrics for every RTR client.

https://routinator.docs.nlnetlabs.nl/en/stable/monitoring.html#metrics

https://grafana.com/grafana/dashboards/11922

Cheers,

Alex

---

From: "Koen van Hove" <[email-masked]>
Date: Sat, 13 Nov 2021 14:33:42 +0100

Steve,

Perhaps my wording was not optimal - let me try to show my point by means of example.

Imagine the following structure: A RIR at level 0, my CA directly below it at level 1, and my children at level 2. With the current limits I can easily create 1000 children with different repositories at level 2, thereby exceeding the limit and stopping the further exploring of other repositories within this TAL. With the current traversal order, this means that if your repository is at level 0 or 1, it will be included. If it is at level 2, it may be included, if it is visited before my children. If it is at level 3, you can be sure that it will not be included, as my children at level 2 already exhausted the repository limit.

Currently if you manage a /32 (IPv6), and you host your own repository, you have a choice. You can decide to have all ROAs in one large repository (at level 1), but you can also delegate parts to other organisations who host their own repositories (at level 2+). With these limits, that latter option becomes risky, as with an attack as described above you risk those delegations to other organisations, as they are at the same level as my malicious children. Even worse, if your repository is not directly below the RIR, but instead below, e.g., a NIR, then hosting your own repository already happens at level 2. This is what I meant with "freedom" - the ability to structure your part of the tree in a way that suits your organisational needs.

If we decide that every layer below layer 2 may be ignored when an attack like this happens, then I do believe we can consider those objects to be less "important" (for the lack of a better wording).

Koen van Hove

-----Oorspronkelijk bericht-----
Van: Stephen Kent <[email-masked]> 
Verzonden: donderdag 11 november 2021 16:28
Aan: [email-masked]
Onderwerp: Re: [Sidrops] RPKI fencing limits (was: Tree Hints)

Koen,
> important the objects become.
"important' is not a meaningful term here. However, it is accurate to say that as one descends further down the RPKI tree, the address space range available to allocate becomes smaller, relative to the cert path in question. That's a security feature.
> This seems to me as limiting the freedom that RPKI provides, which, 
> especially in the advent of IPv6, is a choice that, I believe, if it 
> were to be made, should be consciously made with wide consensus, 
> rather than be the consequence of an implementation.

Steve

---

From: Ties de Kock <[email-masked]>
Date: Mon, 15 Nov 2021 09:17:55 +0100

Hi Claudio,

> On 12 Nov 2021, at 09:55, Claudio Jeker <[email-masked]> wrote:
> 

I think we meant the same thing: Dependant on the order of exploration, a
malicious repository at the level _above_ the NIR can prevent delegated
repositories _below_ the NIR from being explored.

Is the approach taken by rpki-client documented somewhere? Details matter here.
For example, I assume we are talking about _repositories_ here (c.f. CAs) when
talking about fetching. But about CAs when discussing depths. And that repositories
are not sorted by in-degree. When that is clear we can discuss scenarios better.

Note that I would assume and recommend that a RIR or NIR uses use (multiple)
_different_ repositories for the children to publish in for security/engineering
reasons (but I would not discuss those details on-list at the moment).

> monitor it and keep it up to date.

Better to wake up to patch then when a vulnerability is being abused!

RPs being less tolerant to untrusted data/oddly behaving/"test" repositories is
a good thing overall. Many of the false positive alerts that I have seen are due
to repositories not being fetched. This was why I added a wrapper for rsync
(calculate time left, run timeout -k [time_left] rsync)  with a "deadline" for
rsync to finish by in my rpki-client wrapper.

In this distributed system one person's test to see how many RPs fetch over IPv6
only may be another person's 24x7 alert.

Kind regards,
Ties

---

From: Claudio Jeker <[email-masked]>
Date: Mon, 15 Nov 2021 13:10:11 +0100

> repositories _below_ the NIR from being explored.

In my understanding above the NIR is the RIR and with that the holder of
the TAL. If we can't trust the RIR then that TAL should be removed from
the set of TRUST anchors.

If the bad repo is on the same level as the NIR, the NIR will sync without
issues. They are on the same level. Repositories hanging of that NIR may
get more into trouble. One way to solve this if for the two RIR that
actually use NIR repositories to put direct 3rd party repos on an extra
repositoriy. Again, the assumption being that on can trust the RIR and NIR
repos.
 
> are not sorted by in-degree. When that is clear we can discuss scenarios better.

Because of the way RPKI works in real live there are two different trees to
consider. The obvious one is what you get following the certificates. This
is the CA depth.

The more complicated on is the hierarchy of repositories. Many
certificates share the same base repository and are synced all together.
Whenever a new repository is detected during tree traversal a new sync
operation starts. Right now thanks to breadth first traversal all
repositories on the same level will be fetched before any repository on
level n+1 is fetched. What we did not implement is a barrier that forbids
repositories of leveln n+2 or higher to be fetched before n+1 is finished.
For now we don't think this is needed. Especially if the RP falls back to
cached data most of the data will be validated just fine. Since the
repositories were synced on the previous run (which was hopefully not too
long ago).

We did not fully document our approach because we will refine it further
in the near future.
 
> reasons (but I would not discuss those details on-list at the moment).

I would generally agree but different repositories also come at a cost.
More servers to contact, more failures that can happen. On the other hand
failures affect smaller bits of the tree. A good balance needs to be
found.
 
-- 
:wq Claudio

---

From: Randy Bush <[email-masked]>
Date: Mon, 15 Nov 2021 06:15:10 -0800

this discussion reminds me of an incident from my compiler years.

in '80 or so, someone on the arpa complers list asked if they should
limit variable identifiers to 8 or 12 characters.  david moon (lisp,
emacs, ...) replied along the lines that there was clearly a bug in the
su-ai mail system, as messages from the '60s were being regenerated but
with current dates.

randy

---

From: Stephen Kent <[email-masked]>
Date: Tue, 16 Nov 2021 14:19:47 -0500

OKoen,
> If we decide that every layer below layer 2 may be ignored when an attack like this happens, then I do believe we can consider those objects to be less "important" (for the lack of a better wording…
The "freedom" to arrange subordinate pub points is what I would call 
flexibility, but that's not the real issue. The fact is that decisions, 
e.g., delegating subordinate pub points, carry risks, which seems pretty 
obvious to me.

You say that "If _we_ decide that every layer below layer 2 may be 
ignored ..." I doubt that it is attractive to establish a uniform layer 
below which all RPs will ignore subordinate pub points. If I were 
developing RP software, and if I worried about  the sort of DoS attacks 
you describe, I might employ heuristics to mitigate such attacks. For 
example, if I seem to be spending too much time retrieving objects from 
a pub point, I might skip that pub point for now, and come back later. 
If the problem persists, I would notify an operator, and let him/her 
decide whether to skip the pub point because it seems to be an attack 
vector (or just very badly provisioned, or ...).

Steve

---

From: Randy Bush <[email-masked]>
Date: Tue, 16 Nov 2021 13:05:36 -0800

> if I seem to be spending too much time retrieving objects from a pub
> point, I might skip that pub point for now, and come back later.

some swengs might set a marker/flag and fork before descending into
*any* pp; protect with a counter; and eventually check for rusty forks.

randy