Subject: [Sidrops] Tree Hints for the Resource Public Key Infrastructure (RPKI)

From: Koen van Hove <[email-masked]>
Date: Tue, 9 Nov 2021 17:48:12 +0100

Dear all,

Let me quickly introduce myself, my name is Koen van Hove from the
University of Twente. Some of you might know me from other channels.

Recently there have been a couple discussions about limits. I found
two issues regarding limits that may be problematic for the
availability of RPKI, and to kill two birds with one stone, I propose
the "RPKI Tree Hints" draft, which can act as a framework for future
limits in RPKI. Here is a direct link:
https://datatracker.ietf.org/doc/draft-kwvanhove-sidrops-rpki-tree-hints/

Issue 1
=======
Every CA in RPKI can have one or more children, who in turn can have
children, etc. Several implementations already impose limits on the
depth of this chain, the most restrictive currently being 12. However,
if a CA goes wide and deep, then due to the exponential growth in the
number of repositories, a shallow depth might suffice to disrupt RPKI.
Imagine the following: a CA has 10 children, who all have 10 children,
ad infinitum. At six layers deep, this already means RPs have to
traverse 111,111 repositories. As I control these repositories, I can
of course make sure that this process is not exactly quick. If I make
every repository take 10 seconds, which is not unreasonable for
current repositories, then this process would already take over 12
days to complete. If instead we use 15 children, then it takes several
years. I believe I do not say anything controversial when I say that
this is too long, and can severely harm the availability of RPKI.

The difficulty in solving this is that having 15 children is in and of
itself nothing special, and nothing that should raise alarm bells.
Limiting this using a global value would severely hinder the
flexibility of the RPKI structure in the future. As the structure is
unknown before walking the tree, discovering this using heuristics is
tricky, and bound to give headaches for many years to come.

Issue 2
=======
The RP retrieves the ROAs, parses them, and then sends them to the
router via RTR. Let us imagine that I own a /48 (which to my knowledge
is not too difficult to obtain). I can split that /48 in 2^80 /128s,
or 2^79 /127s, etc. In total I will have ~2^81 entries. I can also do
that for all 2^32 possible ASNs, creating 2^113 entries in total. This
all has to go over RTR and into the router, and contemporary routers
are to my knowledge not capable of processing such a large amount of
information.

Limiting either the prefixes or ASNs globally is going to be
difficult, as there are legitimate cases where a lot of ASNs are used,
or a lot of different prefixes, but when everyone does that it becomes
a problem.

To solve these two issues, I propose a new object that enables parents
to provide hints about their children (and their descendants). This
way, if a malicious child does something unusual, an RP can detect it
without it impacting the other honest CAs.

I am interested to hear your thoughts on both the issues, as well as
the proposed solution. After all, many roads lead to Rome.

Yours sincerely,
Koen van Hove

---

From: Claudio Jeker <[email-masked]>
Date: Tue, 9 Nov 2021 18:39:27 +0100

> the proposed solution. After all, many roads lead to Rome.

While interesting this can already be solved by the parent CA pulling the
plug on such bad CA repo. This is very simple to implement (removing the
certificate from the parent repository and issue a new manifest).
It would require that the RIRs monitor their part of the tree (which would
be generally a good idea).

Additionally for issue 2 it makes little sense to have VRPs for routes
that are filtered in the DFZ. At least every network admin I know limits
prefix lenght on transit and peering links.

The problem here is that parent publication points need to take
over more responsibility. There is a general weakness in a design that
starts with untrusted input and tries to produce trusted output.
It would be better to have at least some limit on what is fed into the
validator software. At least it seems to be a better plan than
implementing a complex system to impose per CA limits.

Also some of the limits are questionable like minTransferSpeed or
maxTransferSize. The transfer speed of a download can vary a lot and is
not under the control of the publisher of such a limit (all connection
limits have a similar issue). For the transfer size, it depends on the
protocol used and is limited by the maxRepositorySize which gives a upper
bound.

-- 
:wq Claudio

---

From: Ties de Kock <[email-masked]>
Date: Tue, 9 Nov 2021 20:53:38 +0100

> On 9 Nov 2021, at 18:39, Claudio Jeker <[email-masked]> wrote:
> 

While CAs must monitor their part of the tree, they can not reliably detect a
_malicious_ repository. It is trivial to present a split view and decide what
parties should observe what version of the tree.

An attacker can pick what instance they attack. An attacker can choose what
objects to present. Even if there were an audit log (with associated reduced
availability), an attacker could choose what set of objects to combine for a
client, changing their effects.

Furthermore, I think the recent "natural experiment" also shows that many RP
instances are not monitored as acutely as they should be. Many RP instances
do not recover when they crash, and the usage of the affected RP was still
significantly lower on 1-11.

> implementing a complex system to impose per CA limits.

I agree that you can apply limits there and at least one of the implementations
already filters by /24 and /64. However there are other ways to create a large
number of effective objects. I would take this issue as an example of a class of
issues.

Picking thresholds for heuristics is hard. Picking the wrong value for a limit
(for example maximum object size) can cause an RP instance to lose a large part
of the tree for an extended period of time. Picking one value for a limit that
is needed at the top of the tree likely is unsafe when abused by a forest of
sub-CAs.

I think some method of applying heuristics selectively from a point in the
three downward is needed, and this draft proposes one such approach. I also see
why you want to delegate this responsibility instead of letting the RIRs control
it completely.

Kind regards,
Ties
> _______________________________________________
> Sidrops mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/sidrops <https://www.ietf.org/mailman/listinfo/sidrops>

---

From: Job Snijders <[email-masked]>
Date: Tue, 9 Nov 2021 22:39:55 +0100

Hi Koen, group,

> https://datatracker.ietf.org/doc/draft-kwvanhove-sidrops-rpki-tree-hints/

Welcome to the IETF! I appreciate you have joined this community and are
now engaging with the right group of people. Sharing ideas and concepts
by publishing an internet-draft is very helpful for dialogue. Kudos for
taking the effort to write the Tree Hints draft.

> tricky, and bound to give headaches for many years to come.

Good observations, indeed, perhaps there should not be 'global values'
(established through RFCs). An alternative approach is: each implementer
picks their own default limits.

The IETF guidance could be "make sure you set some reasonable limits",
which implementers and operators then take to heart, by ensuring their
software imposes reasonable limits. What constitutes 'reasonable' is
in the eye of the beholder.

Recent versions of OpenBSD rpki-client impose all kinds of limits; these
limits are based on the developers' growth expectations for the next 1
or 2 years, and a bit of highly subjective experience.

Additionally, our expectation is that operators upgrade their RP
instances at least once a year.  (hopefully more frequently as it is
security software!!!! ALWAYS BE UPDATING!!!!) ... Wild idea: as a
security feature, RP implementations could refuse to boot if they
somehow recognize they are older than 2 years.... I digress.

To illustrate the 'hard-coded limits' workflow with an example:

1/ In this announcement we shared some limits: https://mailarchive.ietf.org/arch/msg/sidrops/cFVeb2u1MlYq4PfIsYcoD4a8wks/

2/ One of the SIDROPS participants took note and provided feedback, which resulted in this proposal: https://marc.info/?l=openbsd-tech&m=163645589724336&w=2

3/ rinse repeat

As time goes by we learn, and we'll keep adjusting limits and keep
releasing new versions. This method seems a productive way to keep the
system running. It is better than nothing. :-)

Kind regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 10 Nov 2021 10:50:05 +0100

> On 9 Nov 2021, at 22:39, Job Snijders <job=[email-masked]> wrote:
> 
> Hi Koen, group,
> 

I agree that each RP implementation *will* have to apply limits.

That said.. this will lead to differences in implementations which may
cause objects to be considered invalid (or perhaps not be considered at
all) under implementation A, but appear valid under implementation B.

Documenting the decisions as rpki-client did helps. There is also merit
in a public discussion about limits and motivations. I think different PR
implementations can benefit from sharing. It may also help operators
if limits are documented somewhere and RPs give clear errors when limits
are applied.

The way I see it though, the tree hints proposal is orthogonal to this.
It would allow CAs to give RPs *additional* hints as to what limits may be
applied. RPs should not go beyond their own limits based on these, but they
may then know that they can safely apply even more restrictions.

Of course, the parent CAs have the power to revoke a child in case of
abuse (or hanlon's razor for that matter). But this is by definition reactive.
The tree hints would give RPs more 'pro-active' information. And of course,
the parent CAs could (perhaps should) then still monitor their children. And,
if they would use an RP that honours the defined tree hints, then this could
really help them to detect issues quickly.

As an aside: for this to work well I think we should also consider extending
the RFC 6492 provisioning protocol so that child CAs can be told by their
parents what their limits actually are..

Tim

> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: George Michaelson <[email-masked]>
Date: Wed, 10 Nov 2021 20:08:20 +1000

Well formed mandatory extensions can be useful. A good rule which 3779
brings to the table is "only subsets, no extras" so a rule on defined
limits encoded by parent signing could be made more restrictive but not
less.

I don't like purely arbitrary magic numbers, but it's possible this has to
float for a while between RPs and OPs  before it settles. I personally
think encoded, magic numbers in the spec process here is OK, maybe a
least-worst outcome.

G

---

From: Job Snijders <[email-masked]>
Date: Wed, 10 Nov 2021 11:37:01 +0100

Hi all,

Let's make some todo items:

1/ The tree-hints draft should be amended and define a new X.509 v3
information access extension containing an AccessDescription to the Tree
Hints Distribution Point. Inspiration can be taken from "X509v3 CRL
Distribution Points". I think it is important a robust arc exists from
the CA to the applicable Tree Hints object. Merely listing a Tree Hints
object on a manifest is too lose of a coupling for my taste.

2/ Reduce Section 6 to the bare minimum.  Antoine de Saint-Exupéry once
said "Perfection is achieved, not when there is nothing more to add, but
when there is nothing left to take away." Remove at least:
connectionRetryCount, connectionTimeout, connectionTransferTime, and
minTransferSpeed.

3/ The "NRO Engineering Coordination Group"
(https://www.nro.net/technical-coordination/) should consider and
prepare a position on whether the Trust Anchors are willing to impose
limits on their children. If for one reason or another the RIRs are not
willing to declare limits, this effort is dead in the water anyway. If at
least some of the RIRs have appetite to continuously monitor their tree
and issue appropriate Tree Hints, I'm willing to invest time on the RP
side.

Kind Regards,

Job

---

From: Robert Kisteleki <[email-masked]>
Date: Wed, 10 Nov 2021 11:40:58 +0100

> https://datatracker.ietf.org/doc/draft-kwvanhove-sidrops-rpki-tree-hints/

I'm not a fan of this approach. I believe enforcing such limits in a 
hierarchical fashion is a ticking time bomb. For example restricting the 
size of *all descendants* may work well for a while, but at some point 
in time a (grand)child naturally grows beyond it and will just be 
automatically rejected because of these restrictions.

In addition, what should operators *do* in this situation, particularly 
if the limit was specified some levels above in the hierarchy? Change 
requires the (manual) cooperation of multiple levels - while the 
"suffering" repo owner may not even have a direct relation to the 
"enforcing" CA.

Similarly, prescribing minimum bandwidth requirements between 
essentially random hosts on the network is unlikely to work in practice.

I believe such limits should be specified / enforced in client software, 
where the operator has direct control in case something is amiss. And as 
others mentioned CAs do have the power to deal with deliberately abusive 
clients anyway.

Robert

---

From: Koen van Hove <[email-masked]>
Date: Wed, 10 Nov 2021 15:17:05 +0100

The assumption that is made is that the "view" of RPKI is the same
across the globe. However, I can easily change what data I serve based
on the source IP address. This would make monitoring by TAL operators
difficult, as I can target my attack to a specific organisation. I
would like to have a system where we can trust the input, but I
believe one of the reasons the RPKI exists is because the good faith
approach in BGP has its flaws. For those reasons, I believe being able
to proactively prevent "RPKI abuse" is more valuable than having
monitoring to correct it afterwards, as by then the damage can already
be done.

Regarding the selection of limits: they are based on limits already
imposed by current RP implementations (6.2 - 6.8), as well as new
potential threats (6.1; 6.9). If those are deemed undesirable, then it
might be worth providing some extra guidance for both relying parties
and publication points with uniform limits (or safe values).

Koen van Hove

---

From: Job Snijders <[email-masked]>
Date: Wed, 10 Nov 2021 15:23:29 +0100

On Wed, Nov 10, 2021 at 03:17:05PM +0100, Koen van Hove wrote:
> difficult, as I can target my attack to a specific organisation.

I don't think we make that assumption. I'd flip the argument around: TAL
operators (and others) are able to monitor normal operating levels,
based on that data limits can be established, which help thwart
two-faced attackers.

Kind regards,

Job

---

From: Koen van Hove <[email-masked]>
Date: Wed, 10 Nov 2021 15:58:44 +0100

I agree that it would be good to have a way to inform a CA of their
effective limits, and that an extension to 6492 might be a good
solution for that. If an RP detects a violation of a tree hint, they
can already contact the responsible repository using (one of) its
Ghostbusters Records.

Koen van Hove

---

From: Job Snijders <[email-masked]>
Date: Fri, 12 Nov 2021 00:16:18 +0100

Dear all,

I share Robert's perspective: the "Tree Hints" approach is fatally flawed.

Anyone who disagrees: try implementing it in a RP yourself. :-)

Kind regards,

Job

On Wed, Nov 10, 2021 at 11:40:58AM +0100, Robert Kisteleki wrote:
> size of *all descendants* may work well for a while, but at some point in
> time a (grand)child naturally grows beyond it and will just be automatically
> rejected because of these restrictions.
> 
> In addition, what should operators *do* in this situation, particularly if
> the limit was specified some levels above in the hierarchy? Change requires
> the (manual) cooperation of multiple levels - while the "suffering" repo
> owner may not even have a direct relation to the "enforcing" CA.
> 
> Similarly, prescribing minimum bandwidth requirements between essentially
> random hosts on the network is unlikely to work in practice.
>

---

From: Ties de Kock <[email-masked]>
Date: Fri, 12 Nov 2021 10:32:19 +0100

Hi Job,

> On 12 Nov 2021, at 00:16, Job Snijders <job=[email-masked]> wrote:
> 
> Dear all,
> 

Could you elaborate more on why you think it is fatally flawed?

Robert names specific limits he has concerns with, as well as the need for
manual cooperation to raise limits.

I think the latter is a _good_ property. A CA wants to use an unsafe level of
resources. You need person-to-person interaction (c.f., machine-to-machine) to
ensure that it is legitimate instead of a malicious repository trying to
break the whole tree.

> Anyone who disagrees: try implementing it in a RP yourself. :-)

I agree that it should be shown that the type of limits proposed in this draft
can be implemented.

At an abstract level, this draft proposes to evaluate these limits during tree
traversal. I think that showing that (some of) these limits can be evaluated
during tree -traversal and -processing is enough. Do you agree? 

If you can show that this is possible during tree traversal it shows that you
can apply it in an RP.

Kind regards,
Ties

---

From: "Koen van Hove" <[email-masked]>
Date: Sat, 13 Nov 2021 15:09:56 +0100

Dear Job, all,

Before submitting the draft, I consulted several RP software developers
whether they considered it feasible to implement - they do not think it will
be a walk in the park, but do not consider infeasible; I agree that a
standard that cannot be implemented has little use. To show how the
maxDescendants works in practice, as per Ties' suggestion, I created a
little model in Python, which can be found here:
https://gist.github.com/Koenvh1/49c7fdaeb042fdcc9dd189f03348fd19 It creates
DOT graphs for each step, so you can see how an RP could process tree hints
in the case of maxDescendants - other limits can be implemented in similar
fashion. The code can be optimised, but I mainly want to show how the
process could work. I do not see any major problems for RPs to implement it
in a similar fashion.

There are some open questions, mainly: how does a parent inform a child of
what their limits are? And what if those limits change (especially: become
stricter)? How and when do these limits change?

More generally, I want to point out that these are Tree *Hints*, not Tree
Rules. They give hints to operators (both of RP software and of CAs) what
they may reasonably expect, hence the scope states: "Following the
information in the Tree Hint object is RECOMMENDED. However, local policy
may prevail over the data in a Tree Hint object.". In the bandwidth example,
if you are using FORT Validator then any repository that does not manage
200KB/s is already excluded using the default configuration, thus that
requirement is already in place in practice. The same applies to all limits
mentioned in 6.2-6.8. The main difference is that if included in Tree Hints,
both parties know what "safe" values they can use, rather than having to
keep up with all limits individual implementations impose.

Koen van Hove

-----Oorspronkelijk bericht-----
Van: Job Snijders <[email-masked]> 
Verzonden: vrijdag 12 november 2021 00:16
Aan: Robert Kisteleki <[email-masked]>
CC: [email-masked]
Onderwerp: Re: [Sidrops] Tree Hints for the Resource Public Key
Infrastructure (RPKI)

Dear all,

Kind regards,

Job

On Wed, Nov 10, 2021 at 11:40:58AM +0100, Robert Kisteleki wrote:
> > availability of RPKI, and to kill two birds with one stone, I 
> > propose the "RPKI Tree Hints" draft, which can act as a framework 
> > for future limits in RPKI. Here is a direct link:
> hierarchical fashion is a ticking time bomb. For example restricting 
> the size of *all descendants* may work well for a while, but at some 
> point in time a (grand)child naturally grows beyond it and will just 
> be automatically rejected because of these restrictions.
> 
> In addition, what should operators *do* in this situation, 
> particularly if the limit was specified some levels above in the 
> hierarchy? Change requires the (manual) cooperation of multiple levels 
> - while the "suffering" repo owner may not even have a direct relation to
the "enforcing" CA.
> I believe such limits should be specified / enforced in client 
> software, where the operator has direct control in case something is 
> amiss. And as others mentioned CAs do have the power to deal with 
> deliberately abusive clients anyway.

---

From: 'Job Snijders' <[email-masked]>
Date: Mon, 15 Nov 2021 16:38:59 +0100

On Sat, Nov 13, 2021 at 03:09:56PM +0100, Koen van Hove wrote:
> Before submitting the draft, I consulted several RP software
> developers whether they considered it feasible to implement

I question the soundness of your consultation, knowing that "Several RP
software developers" also failed to understand the purpose of RPKI
manifests [1], needed a year and lots of hand-holding to recognize the
true situation ([2] -> [3]); or happily violated CA intent by ignoring
missing CRLs and CRL staleness [4], [5]; or failed to see an issue with
certificate malleability [6], [7], [8].

How can this be explained? Groupthink is a psychological phenomenon that
can occur within groups of people in which the desire for conformity
results in an irrational or dysfunctional decision-making outcome. Or
perhaps it is lack of ever experiencing consequences, who knows! :-)

For everyone's amusement: https://sobornost.net/~job/should_vs_must.jpg

> To show how the maxDescendants works in practice, as per Ties'
> suggestion, I created a little model in Python, which can be found
> here: https://gist.github.com/Koenvh1/49c7fdaeb042fdcc9dd189f03348fd19
> It creates DOT graphs for each step, so you can see how an RP could
> process tree hints in the case of maxDescendants - other limits can be
> implemented in similar fashion. 

It seems you implemented something that takes a Python dictionary and
outputs a DOT graph. This is not the same as ingesting potentially
hostile data from an untrusted network input, parsing the data according
to the X.509 and RPKI specifications, building validation paths, and
generating a set of validated ROA payloads.

It also should be recognize that "maxDescendants set to zero" is the
same as "pathLenConstraint set to zero" (RFC 5280 Section 4.2.1.9).
This begs the question, why is pathLenConstraint not used in the RPKI?
Was it an oversight? Or was it recognized that pathLenConstraint is not
a security feature, but rather a "product differentiator"? I consider
maxDescendants to be in the latter category.

> There are some open questions, mainly: how does a parent inform a
> child of what their limits are? And what if those limits change
> (especially: become stricter)? How and when do these limits change?

Great questions indeed. Perhaps the parent can inform the child by
sending an invoice, and only adjust limits after having received payment?

> More generally, I want to point out that these are Tree *Hints*, not
> Tree Rules.

Interesting how a set of 'hints' can be an effective mitigation against
Issue H, J, and K. It was my understanding that you propose - only when
these hints are applied as if they are rules - safety can be achieved.
But you now appear water down the applicability? Which is it?

More importantly: already two different RP implementers recognized that
"each RP implementation *will* have to apply limits" [9]. Unfortunately
only one implementer took action so far. The problems you documented
have to be solved irregardless of the existence of "Tree Hints", because
an RP can't rely on Tree Hints being reasonable, or even existing.

My suggestion would be everyone gets a move on and focuses on
immediately resolving their issues, because at present H, J & K are not
mitigated in multiple implemenations. Hint: accepting chains up to 32
long is ludicrous. After having implemented sane limits, implementers
are likely to come to understand there is no need for "Tree Hints". 

> In the bandwidth example, if you are using FORT Validator then any
> repository that does not manage 200KB/s is already excluded using the
> default configuration, thus that requirement is already in place in
> practice. The same applies to all limits mentioned in 6.2-6.8. The
> main difference is that if included in Tree Hints, both parties know
> what "safe" values they can use, rather than having to keep up with
> all limits individual implementations impose.

You correctly place "safe" between quotation marks. Merely labeling
something "safe" does not intrincically make it "safe". Fun to read that
FORT now doubles as a general purpose network speedtest. However, from
there it doesn't follow that "the requirement is already in place in
practise".

I remain skeptical about your proposal, as I find its merit lacking.

Kind regards,

Job

[1]: https://blog.apnic.net/2020/11/10/rpki-manifests-securely-declare-contents/
[2]: https://lists.nlnetlabs.nl/pipermail/rpki/2019-December/000109.html
[3]: https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000245.html
[4]: https://github.com/NLnetLabs/routinator/issues/274
[5]: https://github.com/RIPE-NCC/rpki-validator-3/issues/162
[6]: https://github.com/RIPE-NCC/rpki-validator-3/issues/255
[7]: https://github.com/NLnetLabs/routinator/issues/365
[8]: https://github.com/cloudflare/cfrpki/issues/66
[9]: https://mailarchive.ietf.org/arch/msg/sidrops/aeM_IAyZ0XZ2luho_FEewScAvjQ/

---

From: Ties de Kock <[email-masked]>
Date: Tue, 16 Nov 2021 08:19:27 +0100

Hi Job,

> On 15 Nov 2021, at 16:38, Job Snijders <job=[email-masked]> wrote:
> 

A constructive point that I can take from this is that we should strive for a
testbed with various constraints used to evaluate the implementation of the
constraints. A BBN conformance but with repositories, not objects.

> generating a set of validated ROA payloads.

For me, as a software engineer, proving that you can apply this in any tree
generalises to the RPKI tree. And this meets the burden of proof I proposed.

> sending an invoice, and only adjust limits after having received payment?

That is already possible—no change by this proposal.

A parent can (already, hypothetically) enforce additional commercial conditions
depending on how RPKI is being used (e.g. change a charging scheme). Remember
that the objects published by a legitimate repository are already visible to all
participants (and a parent can thus monitor and revoke if commercial conditions
are not met).

Yet, the objects of a malicious repository are not always visible.

Furthermore, the PoCs for the “Possibility to DOS the whole ecosystem” exist.
Without additional evidence, your scenario is a hypothetical that appeals to the
fears of network operators. Can you provide additional evidence for this intent?

> are likely to come to understand there is no need for "Tree Hints". 

The effective branching factor we see in the RPKI is ~15K. With this lower bound
on the branching factor, just three levels can build a tree that is orders of
magnitude larger than what can reasonably be processed. Both 10 and 32 are
thresholds that offer no protection.

I do believe that cutting off exploration could improve the situation. I hope we
can have a productive and civil discussion in the WG about the short-term
bandaids and structural fixes for these issues.

Kind regards,
Ties

---

From: "Koen van Hove" <[email-masked]>
Date: Tue, 16 Nov 2021 19:00:10 +0100

Dear Job, all,

I am afraid I do not quite follow you. Could you clarify what you meant by
"fatally flawed"? What merit is lacking?

For those unaware, I have been looking into the question "Can I disrupt RPKI
RP software by introducing a malicious CA/repository into the tree?". A
summary of what I tested can be found here:
https://labs.ripe.net/author/koen-van-hove/improving-the-resiliency-of-rpki-
relying-party-software/, but the long story short is that CAs and
repositories can attempt to use malicious behaviour (e.g., creating many
descendants) to make RP software take unreasonably long to process. To back
this up, I created proof of concepts, which have the letters A to O. Using
heuristics to exclude the malicious nodes without collateral damage (ergo
also excluding non-malicious nodes) is difficult, if not impossible. The
Tree Hints draft is a proposal to solve this issue, by providing the RP
software the information it needs to determine when a node is malicious.

> My suggestion would be everyone gets a move on and focuses on immediately
resolving their issues, because at present H, J & K are not mitigated in
multiple implemenations. Hint: accepting chains up to 32 long is ludicrous.
After having implemented sane limits, implementers are likely to come to
understand there is no need for "Tree Hints".

Sane limits would be great - I am just not sure whether there are static
limits that exclude malicious nodes without also excluding non-malicious
nodes, and that can ensure that the RP software can continue working as it
should within a reasonable timeframe. Feel free to propose sane limits, then
I will run the numbers to see if that effectively solves the problem. I have
already done so for the limits posted in the RPKI fencing thread and shown
that they do not suffice to effectively stop the attack, as well as possibly
causing collateral damage, but I am happy to run the numbers again for any
other proposals you may have.

The problem still exists and is in need for a solution. Tree Hints can be a
solution, but I invite you to come up with a different solution to solve
this problem. Then I will try to find out if an attacker can find a way
around those limits.

Koen van Hove

-----Oorspronkelijk bericht-----
Van: 'Job Snijders' <[email-masked]> 
Verzonden: maandag 15 november 2021 16:39
Aan: [email-masked]
Onderwerp: Re: [Sidrops] Tree Hints for the Resource Public Key
Infrastructure (RPKI)

I question the soundness of your consultation, knowing that "Several RP
software developers" also failed to understand the purpose of RPKI manifests
[1], needed a year and lots of hand-holding to recognize the true situation
([2] -> [3]); or happily violated CA intent by ignoring missing CRLs and CRL
staleness [4], [5]; or failed to see an issue with certificate malleability
[6], [7], [8].

How can this be explained? Groupthink is a psychological phenomenon that can
occur within groups of people in which the desire for conformity results in
an irrational or dysfunctional decision-making outcome. Or perhaps it is
lack of ever experiencing consequences, who knows! :-)

It seems you implemented something that takes a Python dictionary and
outputs a DOT graph. This is not the same as ingesting potentially hostile
data from an untrusted network input, parsing the data according to the
X.509 and RPKI specifications, building validation paths, and generating a
set of validated ROA payloads.

It also should be recognize that "maxDescendants set to zero" is the same as
"pathLenConstraint set to zero" (RFC 5280 Section 4.2.1.9).
This begs the question, why is pathLenConstraint not used in the RPKI?
Was it an oversight? Or was it recognized that pathLenConstraint is not a
security feature, but rather a "product differentiator"? I consider
maxDescendants to be in the latter category.

> (especially: become stricter)? How and when do these limits change?

Great questions indeed. Perhaps the parent can inform the child by sending
an invoice, and only adjust limits after having received payment?

More importantly: already two different RP implementers recognized that
"each RP implementation *will* have to apply limits" [9]. Unfortunately only
one implementer took action so far. The problems you documented have to be
solved irregardless of the existence of "Tree Hints", because an RP can't
rely on Tree Hints being reasonable, or even existing.

My suggestion would be everyone gets a move on and focuses on immediately
resolving their issues, because at present H, J & K are not mitigated in
multiple implemenations. Hint: accepting chains up to 32 long is ludicrous.
After having implemented sane limits, implementers are likely to come to
understand there is no need for "Tree Hints". 

You correctly place "safe" between quotation marks. Merely labeling
something "safe" does not intrincically make it "safe". Fun to read that
FORT now doubles as a general purpose network speedtest. However, from there
it doesn't follow that "the requirement is already in place in practise".

Kind regards,

Job

[1]:
https://blog.apnic.net/2020/11/10/rpki-manifests-securely-declare-contents/
[2]: https://lists.nlnetlabs.nl/pipermail/rpki/2019-December/000109.html
[3]: https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000245.html
[4]: https://github.com/NLnetLabs/routinator/issues/274
[5]: https://github.com/RIPE-NCC/rpki-validator-3/issues/162
[6]: https://github.com/RIPE-NCC/rpki-validator-3/issues/255
[7]: https://github.com/NLnetLabs/routinator/issues/365
[8]: https://github.com/cloudflare/cfrpki/issues/66
[9]:
https://mailarchive.ietf.org/arch/msg/sidrops/aeM_IAyZ0XZ2luho_FEewScAvjQ/

---

From: Job Snijders <[email-masked]>
Date: Tue, 16 Nov 2021 19:18:26 +0100

Hi Koen,

On Tue, 16 Nov 2021 at 19:00, Koen van Hove <[email-masked]>
wrote:

> other proposals you may have.

When you say “run the numbers”, do you mean you input the suggested limits
in something like a spreadsheet, then derive a pass / fail based on some
calculation?

Kind regards,

Job

---

From: Mikhail Puzanov <[email-masked]>
Date: Wed, 17 Nov 2021 13:41:50 +0100

Hi Koen, all

I would like to add my 2 cents about the draft.

1) Some sort of limits will have to be forced in RPs regardless of 
"Tree Hints” implemented or not since an RP can not rely on these hints
always present. I would imagine that a reasonable RP implementation uses 
some default limits and policies and adjusts them according with the hint 
objects if it finds them on a manifest. From that viewpoint the draft 
would benefit from  describing these default limits and, probably, 
the approaches to enforcing them.

Something along the lines of "RP SHOULD traverse the CA tree in such a way 
that a malicious CA/repository does not stop well-behaving CA to be processed” 
or “RP MUST set a timeout on TA tree processing and SHOULD process all the CAs 
that fit into this timeout even if the whole tree does not finish in time” 
would make it more solid.

It may be, of course, that it is better to mention these RP limits in RFC 6486 
and 6487 and refer from this draft.

Setting such hard limits would automatically impose a lot of constraints on RP 
implementation such as using timeouts pretty much everywhere, limiting CPU time 
and memory allocations on some per-subtree basis, running concurrent processes 
instead of sequential processing, etc. 

I can imagine that it probably was not your intention when writing this draft, 
just want to point out that every RP-implementor has to answer the question
“what do I do if there are no hint objects?” so some solid ground needs be set 
here first before starting to use these hints.

2) I have a feeling that it is not very easy to implement the backend 
part of it, for non-technical reasons. How does, for example, RIPE NCC set these 
limits, based on what? Some reasonable default for everybody? But then it can still 
be too low or too high for a particular CA/NIR. Does it imply some sort of 
application procedure when an LIR says “I am planning to have a child CA and 
not more than 100 ROAs”? I have a hunch that there are legal implications to 
this, so RIRs need to change their disclaimers, CPS, even contracts(?) in order to 
incorporate something like “By setting this checkbox you agree to get you CA revoked 
if you (or any of your child CAs) publish more than 1GB of objects”. 

Legal issues, of course, should not be your concern when proposing a technical 
draft, but if the draft goes on, this topic will pop up a lot and maybe it would 
make sense to add something about realistically setting these hints to the draft
or at least have a more detailed discussion about it.

--
Misha

> On 16 Nov 2021, at 19:00, Koen van Hove <[email-masked]> wrote:
> 
> Dear Job, all,
>

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 17 Nov 2021 15:31:52 +0100

Hi Koen et al,

I went through https://www.ietf.org/archive/id/draft-kwvanhove-sidrops-rpki-tree-hints-00.html and the major thing I am wondering is: which CA will even bother with this?

If you take a RIR, do you think they will define for each LIR how much a LIR can do, as it becomes rather hard to define who is 'big' and others who are 'small', especially as that can change on an hourly basis.

Just that classification will be tricky and very error-prone and quick to time out.

If we are talking NIRs in the APNIC region, do you think CNNIC (the Chinese NIR) wants to be regulated/limited that way at that level?
They definitely hold enough power to let APNIC configure anything, and so do many large corporations in the US.

What about CERNET, who run all the universities in China, their limit would have to be set very very large, APNIC won't know and even CNNIC would have a hard time guessing what CERNET will do.

Similar take a Google, Facebook, Comcast or even a one of many transit providers and their customers.

What if a LIR did not deploy much RPKI yet, then the number of objects will be low, while moments later they might automatically do it for all their prefixes it will explode going over the limits the CA set.

Thus a CAs limit's will either be too low and thus hurtful, or too high and useless.

Your proposal would also imply that every CA that implements limits would have to run an analysis of the complete tree to see if any limits might be tripped at one point or another. But, that would mean they have to do it from a variety of unknown vantage points, as a malicious CA would just return different data. But even if such a mechanism would exist, due to the rapidly changing of the internet, the limit might already be tripped before you realize, and if these are automated limit changes without verification, then what is the value: malicious CA slowly just adds objects so the CA automatically raises the limits.

The CA would thus also have to spend significant resources into verifying this.
Thus why would they bother being the shepard of that?

Thus your proposal means that very few will apply them, which will likely be in the wrong places.

Note also that in the largest PKI system we have in the world, the one from SSL/TLS, there are no such limits in place in the CA tree, even though there are billions of objects (https://ct.cloudflare.com, and no you don't easily run that checker yourself, requires a bit of decent hardware).
(RPKI is akin to Cert Transparency as one has to fetch all objects from a variety of untrusted locations and then parse them).

In the end, having the limits imposed by the software, means you are trusting the information of the vendor that you chose.

Fortunately you agree with that in the end of section 3:

"By using reasonable values at the top level, it can be assumed that the tree can be traversed in reasonable time, whilst keeping the flexibility of the tree structure as it is currently."

The complete top level is not the TAL, but the software, which you trust to have implemented code correctly (as they could circumvent any limit a TAL or CA would set).

This does imply, like many other systems, that the software has to be maintained. monitored and updated regularly (both vendor releases and people upgrading their systems), but hopefully for a system as important as RPKI people take care of it.

TLDR:
 - Letting the CAs define limits will be error prone or just won't happen, while it would require extra code with more possible failures.
 - Letting RP software that you control, configure, trust and update define these limits is a much more resilient and better approach.

Greets,
 Jeroen

---

From: Randy Bush <[email-masked]>
Date: Wed, 17 Nov 2021 15:06:43 -0800

continuing my resistance to being dragged down this rathole ...  and my
penchant for metaphor

    "I think there is a world market for maybe five computers."
    Thomas Watson, president of IBM, 1943

along with simplicity, a major skill of protocol design is scale, both
horizontal and vertical.

    “The only real problem in the Internet is scaling”
    Mike O’Dell (CTO UUNET)

and

    Ettore Bugatti, maker of the finest cars of his day, was once asked
    why his cars had less than perfect brakes.  He replied, "I make my
    cars to go, not to stop."

randy

---

From: "Koen van Hove" <[email-masked]>
Date: Thu, 18 Nov 2021 18:16:13 +0100

Dear Misha, all,

Regarding point 1: I believe that is a good idea, although I am not sure whether this draft is necessarily the right place for it. RFC 6486 and 6487 may be more suitable for it, or as its own RFC, especially as it might be a (fast) moving target. 

Regarding point 2: Regarding the limits themselves: I would expect that a general limit would work for 99% in practice. For there where it does not work currently, an exception can be made, by opening a ticket, making a call, sending a facsimile, or whatever the conventional way is to reach your CA. The legal aspect is an interesting one. Perhaps someone else could provide more insight into this? I would assume they already have the means (from a policy perspective) to revoke the certificate for abuse, and that this would be an extension to that policy. 

Koen van Hove

-----Oorspronkelijk bericht-----
Van: Mikhail Puzanov <[email-masked]> 
Verzonden: woensdag 17 november 2021 13:42
Aan: Koen van Hove <[email-masked]>
CC: [email-masked]
Onderwerp: Re: [Sidrops] Tree Hints for the Resource Public Key Infrastructure (RPKI)

Hi Koen, all

1) Some sort of limits will have to be forced in RPs regardless of "Tree Hints” implemented or not since an RP can not rely on these hints always present. I would imagine that a reasonable RP implementation uses some default limits and policies and adjusts them according with the hint objects if it finds them on a manifest. From that viewpoint the draft would benefit from  describing these default limits and, probably, the approaches to enforcing them.

Something along the lines of "RP SHOULD traverse the CA tree in such a way that a malicious CA/repository does not stop well-behaving CA to be processed” 
or “RP MUST set a timeout on TA tree processing and SHOULD process all the CAs that fit into this timeout even if the whole tree does not finish in time” 
would make it more solid.

It may be, of course, that it is better to mention these RP limits in RFC 6486 and 6487 and refer from this draft.

Setting such hard limits would automatically impose a lot of constraints on RP implementation such as using timeouts pretty much everywhere, limiting CPU time and memory allocations on some per-subtree basis, running concurrent processes instead of sequential processing, etc. 

I can imagine that it probably was not your intention when writing this draft, just want to point out that every RP-implementor has to answer the question “what do I do if there are no hint objects?” so some solid ground needs be set here first before starting to use these hints.

2) I have a feeling that it is not very easy to implement the backend part of it, for non-technical reasons. How does, for example, RIPE NCC set these limits, based on what? Some reasonable default for everybody? But then it can still be too low or too high for a particular CA/NIR. Does it imply some sort of application procedure when an LIR says “I am planning to have a child CA and not more than 100 ROAs”? I have a hunch that there are legal implications to this, so RIRs need to change their disclaimers, CPS, even contracts(?) in order to incorporate something like “By setting this checkbox you agree to get you CA revoked if you (or any of your child CAs) publish more than 1GB of objects”. 

Legal issues, of course, should not be your concern when proposing a technical draft, but if the draft goes on, this topic will pop up a lot and maybe it would make sense to add something about realistically setting these hints to the draft or at least have a more detailed discussion about it.

--
Misha

> I am afraid I do not quite follow you. Could you clarify what you 
> meant by "fatally flawed"? What merit is lacking?
> 
> For those unaware, I have been looking into the question "Can I 
> disrupt RPKI RP software by introducing a malicious CA/repository into 
> the tree?". A summary of what I tested can be found here:
> https://labs.ripe.net/author/koen-van-hove/improving-the-resiliency-of
> -rpki- relying-party-software/, but the long story short is that CAs 
> and repositories can attempt to use malicious behaviour (e.g., 
> creating many
> descendants) to make RP software take unreasonably long to process. To 
> back this up, I created proof of concepts, which have the letters A to 
> O. Using heuristics to exclude the malicious nodes without collateral 
> damage (ergo also excluding non-malicious nodes) is difficult, if not 
> impossible. The Tree Hints draft is a proposal to solve this issue, by 
> providing the RP software the information it needs to determine when a node is malicious.
> 
>> immediately
> resolving their issues, because at present H, J & K are not mitigated 
> in multiple implemenations. Hint: accepting chains up to 32 long is ludicrous.
> After having implemented sane limits, implementers are likely to come 
> to understand there is no need for "Tree Hints".
> 
> Sane limits would be great - I am just not sure whether there are 
> static limits that exclude malicious nodes without also excluding 
> non-malicious nodes, and that can ensure that the RP software can 
> continue working as it should within a reasonable timeframe. Feel free 
> to propose sane limits, then I will run the numbers to see if that 
> effectively solves the problem. I have already done so for the limits 
> posted in the RPKI fencing thread and shown that they do not suffice 
> to effectively stop the attack, as well as possibly causing collateral 
> damage, but I am happy to run the numbers again for any other proposals you may have.
> 
> The problem still exists and is in need for a solution. Tree Hints can 
> be a solution, but I invite you to come up with a different solution 
> to solve this problem. Then I will try to find out if an attacker can 
> find a way around those limits.
> 
> Koen van Hove
> 
> I question the soundness of your consultation, knowing that "Several 
> RP software developers" also failed to understand the purpose of RPKI 
> missing CRLs and CRL staleness [4], [5]; or failed to see an issue 
> with certificate malleability [6], [7], [8].
> 
> How can this be explained? Groupthink is a psychological phenomenon 
> that can occur within groups of people in which the desire for 
> conformity results in an irrational or dysfunctional decision-making 
> outcome. Or perhaps it is lack of ever experiencing consequences, who 
> knows! :-)
> 
> For everyone's amusement: 
> https://sobornost.net/~job/should_vs_must.jpg
> 
>> here: 
>> https://gist.github.com/Koenvh1/49c7fdaeb042fdcc9dd189f03348fd19
>> process tree hints in the case of maxDescendants - other limits can 
>> be implemented in similar fashion.
> 
> hostile data from an untrusted network input, parsing the data 
> according to the
> X.509 and RPKI specifications, building validation paths, and 
> Was it an oversight? Or was it recognized that pathLenConstraint is 
> not a security feature, but rather a "product differentiator"? I 
> consider maxDescendants to be in the latter category.
> 
> Interesting how a set of 'hints' can be an effective mitigation 
> against Issue H, J, and K. It was my understanding that you propose - 
> only when these hints are applied as if they are rules - safety can be achieved.
> More importantly: already two different RP implementers recognized 
> that "each RP implementation *will* have to apply limits" [9]. 
> Unfortunately only one implementer took action so far. The problems 
> you documented have to be solved irregardless of the existence of 
> "Tree Hints", because an RP can't rely on Tree Hints being reasonable, or even existing.
> 
> immediately resolving their issues, because at present H, J & K are 
> not mitigated in multiple implemenations. Hint: accepting chains up to 32 long is ludicrous.
> After having implemented sane limits, implementers are likely to come 
> to understand there is no need for "Tree Hints".
> 
> something "safe" does not intrincically make it "safe". Fun to read 
> that FORT now doubles as a general purpose network speedtest. However, 
> from there it doesn't follow that "the requirement is already in place in practise".
> 
> [2]: 
> https://lists.nlnetlabs.nl/pipermail/rpki/2019-December/000109.html
> [3]: 
> https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000245.html

---

From: "Koen van Hove" <[email-masked]>
Date: Thu, 18 Nov 2021 18:18:11 +0100

Dear Jeroen, all,

The cases you name are a great example of why I believe heuristics are
difficult. I am not claiming the politics will be easy. To take your example
of APNIC and CNNIC: there is already some relation between the two (APNIC
signed the certificate of CNNIC), thus they can also create an exception for
CNNIC in the Tree Hints to allow them to use a "very large number" (I do not
believe ASN.1 integers support infinity, but operationally it would be
equivalent) to bypass them entirely. From a security perspective this may
not be the wisest, but it may be a political compromise.

Regarding having to ask your parent for more resources: I do not think this
is an issue, rather, I think this is a feature. The "a CAs limit's will
either be too low and thus hurtful, or too high and useless." is the same
problem RP software faces, or at least their operators face. How this
process of requesting more resources is shaped is outside the scope of Tree
Hints.

The alternative seems to be that the RP software decides these limits
instead, and if then a small LIR suddenly creates a lot of objects, it must
wait for everyone to update their RP software for it to work. I am not sure
whether that is a better compromise. I do as well hope that people regularly
update their RPKI software, but that unfortunately does not seem to be the
case.

Koen van Hove

-----Oorspronkelijk bericht-----
Van: Jeroen Massar <[email-masked]> 
Verzonden: woensdag 17 november 2021 15:32
Aan: Koen van Hove <[email-masked]>
CC: [email-masked]
Onderwerp: Re: [Sidrops] Tree Hints for the Resource Public Key
Infrastructure (RPKI)

Hi Koen et al,

I went through
https://www.ietf.org/archive/id/draft-kwvanhove-sidrops-rpki-tree-hints-00.h
tml and the major thing I am wondering is: which CA will even bother with
this?

If you take a RIR, do you think they will define for each LIR how much a LIR
can do, as it becomes rather hard to define who is 'big' and others who are
'small', especially as that can change on an hourly basis.

Just that classification will be tricky and very error-prone and quick to
time out.

If we are talking NIRs in the APNIC region, do you think CNNIC (the Chinese
NIR) wants to be regulated/limited that way at that level?
They definitely hold enough power to let APNIC configure anything, and so do
many large corporations in the US.

What about CERNET, who run all the universities in China, their limit would
have to be set very very large, APNIC won't know and even CNNIC would have a
hard time guessing what CERNET will do.

Similar take a Google, Facebook, Comcast or even a one of many transit
providers and their customers.

What if a LIR did not deploy much RPKI yet, then the number of objects will
be low, while moments later they might automatically do it for all their
prefixes it will explode going over the limits the CA set.

Thus a CAs limit's will either be too low and thus hurtful, or too high and
useless.

Your proposal would also imply that every CA that implements limits would
have to run an analysis of the complete tree to see if any limits might be
tripped at one point or another. But, that would mean they have to do it
from a variety of unknown vantage points, as a malicious CA would just
return different data. But even if such a mechanism would exist, due to the
rapidly changing of the internet, the limit might already be tripped before
you realize, and if these are automated limit changes without verification,
then what is the value: malicious CA slowly just adds objects so the CA
automatically raises the limits.

The CA would thus also have to spend significant resources into verifying
this.
Thus why would they bother being the shepard of that?

Thus your proposal means that very few will apply them, which will likely be
in the wrong places.

Note also that in the largest PKI system we have in the world, the one from
SSL/TLS, there are no such limits in place in the CA tree, even though there
are billions of objects (https://ct.cloudflare.com, and no you don't easily
run that checker yourself, requires a bit of decent hardware).
(RPKI is akin to Cert Transparency as one has to fetch all objects from a
variety of untrusted locations and then parse them).

In the end, having the limits imposed by the software, means you are
trusting the information of the vendor that you chose.

"By using reasonable values at the top level, it can be assumed that the
tree can be traversed in reasonable time, whilst keeping the flexibility of
the tree structure as it is currently."

The complete top level is not the TAL, but the software, which you trust to
have implemented code correctly (as they could circumvent any limit a TAL or
CA would set).

This does imply, like many other systems, that the software has to be
maintained. monitored and updated regularly (both vendor releases and people
upgrading their systems), but hopefully for a system as important as RPKI
people take care of it.

TLDR:
 - Letting the CAs define limits will be error prone or just won't happen,
while it would require extra code with more possible failures.
 - Letting RP software that you control, configure, trust and update define
these limits is a much more resilient and better approach.

Greets,
 Jeroen

---

From: Alex Band <[email-masked]>
Date: Thu, 18 Nov 2021 18:49:07 +0100

Hi Koen,

> On 18 Nov 2021, at 18:16, Koen van Hove <[email-masked]> wrote:
> 
> Dear Misha, all,
> 

Taking the RIPE NCC as an example, I’m sure there are some interesting things to be found in the two sets of Terms and Conditions they have for this.

The RPKI service itself:
https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/legal/ripe-ncc-certification-service-terms-and-conditions

The repository:
https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/legal/ripe-ncc-certification-repository-terms-and-conditions

-Alex

---

From: Job Snijders <[email-masked]>
Date: Thu, 18 Nov 2021 18:55:23 +0100

On Thu, 18 Nov 2021 at 18:49, Alex Band <[email-masked]> wrote:

> > Regarding point 1: I believe that is a good idea, although I am not sure
> whether this draft is necessarily the right place for it. RFC 6486 and 6487
> may be more suitable for it, or as its own RFC, especially as it might be a
> (fast) moving target.
> >
> > Regarding point 2: Regarding the limits themselves: I would expect that
> a general limit would work for 99% in practice. For there where it does not
> work currently, an exception can be made, by opening a ticket, making a
> call, sending a facsimile, or whatever the conventional way is to reach
> your CA. The legal aspect is an interesting one. Perhaps someone else could
> provide more insight into this? I would assume they already have the means
> (from a policy perspective) to revoke the certificate for abuse, and that
> this would be an extension to that policy.
>
> Taking the RIPE NCC as an example, I’m sure there are some interesting
> things to be found in the two sets of Terms and Conditions they have for
> this.
>
> The RPKI service itself:
>

RIPE NCC’s Certification Practise Statement (CPS) also bears relevance in
this context:

https://www.ripe.net/publications/docs/ripe-751

The above statement is modelled after RFC 7382 / BCP 173

Kind regards,

Job

---

From: Robert Kisteleki <[email-masked]>
Date: Fri, 19 Nov 2021 09:56:08 +0100

Hi,

> Regarding point 2: Regarding the limits themselves: I would expect
> that a general limit would work for 99% in practice. For there where
> it does not work currently, an exception can be made, by opening a
> ticket, making a call, sending a facsimile, or whatever the
> conventional way is to reach your CA. The legal aspect is an

Except, as I said before, when it's not your parent who prescribes that, 
but your grandparent or above. Then you have no business relation and 
likely no contact either. So from one day to the next your repo can just 
disappear in all RP implementations and the only thing you can do is ask 
you parent to ask their parent to ... make an exception. It doesn't 
sound like a fun exercise.

To reiterate: I don't think the specification of these limits and 
controls belong inside the protocol. Making a BCP or such, if really 
needed, is much preferred.

Cheers,
Robert

---

From: Job Snijders <[email-masked]>
Date: Fri, 19 Nov 2021 10:16:14 +0100

On Thu, Nov 18, 2021 at 06:16:13PM +0100, Koen van Hove wrote:
> Regarding point 1: I believe that is a good idea, although I am not
> sure whether this draft is necessarily the right place for it. RFC
> 6486 and 6487 may be more suitable for it, or as its own RFC,
> especially as it might be a (fast) moving target. 

A simple wiki page is sufficient to keep track of such numbers.

https://trac.ietf.org/trac/sidrops/wiki/FencingLimits

Kind regards,

Job

---

From: Ties de Kock <[email-masked]>
Date: Fri, 19 Nov 2021 10:37:59 +0100

Hi Job,

> On 19 Nov 2021, at 10:16, Job Snijders <job=[email-masked]> wrote:
> 

Quick clarifying questions:

Is there a global transfer timeout, or is the 15 minutes per repository? Which
could take 9*15 minutes in the worst case?
(My need for quick rpki-client runs made me implement a global rsync timeout 
using a deadline and coreutils timeout in [0])

Does rpki-client apply the same limit at all levels of the tree (both for the
repository and ca-manifest tree)?

So the 100K children is possible at any level of the tree:
  * RIPE NCC member CA can have >50K objects on it: good, needed.
  * A delegated CA below a delegated CA can have 100K objects on it that refer
    to CAs on it (and nest that 4+ times, >2^66 objects): bad?

Note that a dangerously low limit of 20K objects, that would be hit in practice
during a keyroll, would would still allow >2^57 objects.

Are there any limits on the number of files retrieved per repository (for rsync
and rrdp)? I have created a rsync repository in a testing setup which needed
multiple orders of magnitude more inodes than the 300000 rpki-client recommends.

Kind regards,
Ties

[0]: https://github.com/ties/rpki-client-web

---

From: Claudio Jeker <[email-masked]>
Date: Fri, 19 Nov 2021 13:50:46 +0100

> could take 9*15 minutes in the worst case?

The worst case does not matter because rpki-client stops after 60min of
runtime. We had to use 15min because some real life repositories take a
long time to sync. The limit is selected based on real life data and not
on some worst case scenario. It makes no sense to be overly aggressive
here.

> using a deadline and coreutils timeout in [0])

In other words you accept bad output in preference of faster execution
time.
 
> repository and ca-manifest tree)?

It applies the timeout to all sync actions. So 15min for RRDP and 15min
for rsync. So worst case a repository may only be validated after 30min
but that is fine.
 
>     to CAs on it (and nest that 4+ times, >2^66 objects): bad?

You have this the wrong way around. We need to have this stupid 100k limit
because of RIPE NCC and AS0 tal from APNIC. If these repositories would
split their mft into multiple ones with e.g. 1000 entries max that limit
could be reduced and everyone would win.

Also this deep and wide issue is far less of an issue then it is pushed by
Koen van Hove. The CA repos are all run by well known institutions
(enforced by the RIRs and the way IP delegation works). So it is rather
clear who needs to be prosecuted in the case of an attack. This is not a
traffic DDoS case where the source is unknown.
 
> during a keyroll, would would still allow >2^57 objects.

Why are you rolling all keys at once? Also why is the repository not setup
in a way that does not require to have everything in a single MFT?
What is the reason for this and why can't it be changed?
 
> multiple orders of magnitude more inodes than the 300000 rpki-client recommends.

This is hard to implement. Neither rsync or rrdp allow this to implement
in an easy way. It requires that the number of files is tracked and in the
rsync case this can only done post factum.

-- 
:wq Claudio

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 19 Nov 2021 15:03:39 +0100

> On 19 Nov 2021, at 13:50, Claudio Jeker <[email-masked]> wrote:
> 

slightly on a tangent, but triggered by the inodes..

It may, or may not - just a thought I wanted to share - be useful to store objects
in a different way - e.g. some key value store where you do not need a file per object.

This is somewhat hard / impossible when using rsync as an external tool because it will
expect things on disk - but may be an option if you have native rsync client code. It is
pretty easy to use this storage strategy when doing rrdp.

Of course this is only one small piece of the puzzle, and I will leave it to RP
implementers to judge what works for them.

Tim

---

From: Ties de Kock <[email-masked]>
Date: Fri, 19 Nov 2021 15:46:02 +0100

Hi Claudio,

> on some worst case scenario. It makes no sense to be overly aggressive

Earlier empirical data from Job showed that 90 minutes is needed for slow RPs.
If I recall correctly some rpki-client instantices on NLNOG ring nodes in China
hit this. Which is why repositories support rsync clients that take up to 120
minutes.

Completely agree that it does not make sense to be overly agressive here.

> In other words you accept bad output in preference of faster execution

Indeed. It's a bias towards liveliness and provides a bounded time for the
sync to terminate. I disabled it by default but it made sense for my use case.

I have experimented with very low execution timeouts. For rsync I observed that
an aborted rsync run syncs part of files. In practice, after multiple aborted
rsync runs, the RP can fetch the complete repository.

> for rsync. So worst case a repository may only be validated after 30min

What happens if you have chained slow repositories? Is the total duration
(depth)*(15+15 minutes)?

> could be reduced and everyone would win.

There are two limits at play:
  * large number of VRPs per ROA: needed for AS0
  * large number of objects per CA: Used by many of the larger CAs (or if some
    resource holders follow the "one VRP per ROA advice")

Since it is exponential growth any realistic threshold that you apply is bound
to become intractable when used by a malicious repo. 1000^6 = large as well.

> traffic DDoS case where the source is unknown.

There are > 25K CAs that can create a delegated CA at a moments notice. I would
not claim these are well known institutions. This is many more than when the EFF
ran their SSL observatory experiments. Luckily, unlike for the web, these CAs
have limited resources they can sign for.

web ca:
  * High barrier to entry.
  * High stakes.
  * Compliance/auditing schemes.
  * Can sign for anything.
rpki:
  * Low barrier to entry.
  * Low stakes.
  * No compliance required.
  * Can sign for their own resources.

In my opinion, the main risk is that if a malicious (or incompetent) repository
operator breaks many relying party instances, many will not recover.

> What is the reason for this and why can't it be changed?

The implementation follows https://datatracker.ietf.org/doc/html/rfc6489

I do not recall there being questions about restructure the CA hierarchy to have
less objects per manifest before. Can you explain how changing this would
mitigate "wide and deep" and what branching factor you would propose?

> rsync case this can only done post factum.

In the RIPE validator this would have been easy to implement for RRDP, hard for
rsync. The latter likely needs a custom binary.

Kind regards,
Ties

---

From: Randy Bush <[email-masked]>
Date: Fri, 19 Nov 2021 10:08:48 -0800

> Earlier empirical data from Job showed that 90 minutes is needed for
> slow RPs.  If I recall correctly some rpki-client instantices on NLNOG
> ring nodes in China hit this. Which is why repositories support rsync
> clients that take up to 120 minutes.

folk, you are coming up with magic numbers which might work today on odd
numbered fridays and might look embarrassing and break things in five
years.

640k syndrome

and, btw, are folk taking into consideration that CAs may have multiple
parents; and those parents might be at quite different depths in
disjoint (thank you RIR egos) trees?

randy