Subject: [Sidrops] [WG ADOPTION] draft-yan-sidrops-roa-considerations - 04/16/2021

From: Chris Morrow <[email-masked]>
Date: Fri, 26 Mar 2021 17:38:28 +0000

Howdy WG Folken,
The authors of:
  https://datatracker.ietf.org/doc/html/draft-yan-sidrops-roa-considerations

had asked a while ago (probably 5 revisions ago :( ) for WG Adoption
of their draft. The draft abstract is:

  "The address space holder needs to issue an ROA object when it
   authorizes one or more ASes to originate routes to multiple prefixes.
   During the process of ROA issuance, the address space holder needs to
   specify an origin AS for a list of IP prefixes.  Besides, the address
   space holder has a free choice to put multiple prefixes into a single
   ROA or issue separate ROAs for each prefix based on the current
   specification.  This memo analyzes and presents some operational
   problems which may be caused by the misconfigurations of ROAs
   containing multiple IP prefixes.  Some suggestions and considerations
   also have been proposed."

a) Please consider this a WG Call for Adoption of this draft.

2) Please take a few moments to read the ~5 page draft and send along comments/
questions/complaints(in the form of fixed text) to the authors.

iii) this call expires 04/16/2021 - April 16 2021

Thanks!
-chris
wg-chair-persona

---

From: George Michaelson <[email-masked]>
Date: Mon, 29 Mar 2021 09:20:28 +1000

I support adoption.

-G

On Sat, Mar 27, 2021 at 3:38 AM Chris Morrow <[email-masked]> wrote:
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 29 Mar 2021 16:06:56 +0200

Hi,

I am not sure yet whether this should be a separate RFC in the end, or that this should be considered with other BCP work. But I support adoption so this can be discussed more thoroughly.

I have a number of comments / requests for clarity, most importantly:

= revocation?

   A large number of experiments for the process of ROA issuance have
   been made on our RPKI testbed, it is found that the misconfigurations
   during the issuance may cause the ROAs which have been issued to be
   revoked.

I don't see how the ROAs would be revoked in this context. I think the problem is that ROAs which contain multiple prefixes can become invalid if one of the prefixes is lost by the issuing CA and the ROA is seen to 'over-claim'.

I.e. the problem that I see is fate-sharing. This has been discussed in the past and the advice - in general - was to issue ROAs on a per prefix basis if possible. Unless of course, as the authors also say, this would result in 100s of ROAs. There is a tradeoff to be considered.

= analysis

It looks like an analysis was included in version -02 but has since been removed.

For the analysis I would like to raise that multi-prefix ROAs are not always a problem. The issue (that I see) arises when a parent shrinks a CA certificate before the child CA realizes this. This is an issue for remote children using RFC 6492 - they may find that the parent has already shrunk their certificate.

However, for hosted CAs where the parent and children live in the same system it is achievable to make sure that child CAs re-issue ROAs in advance of the parent re-issuing their CA certificate. So, the problem may never come up there. Meaning that just the occurrence of multi-prefix ROAs in such system is not necessarily an issue.

= redundant transmission?

From the document:

   That is to say, the update of the ROA containing multiple
   IP address prefixes will lead to redundant transmission between RP
   and BGP routers

I am not sure what the authors mean here. The RTR protocol is based on validated ROA prefixes (VRPs) (section 5.6/5.7 of RFC 8210). If there are ROA objects with many prefixes, then replacing those ROAs does not necessarily result in a lot of churn to the routers. Meaning if a ROA is replaced then only the difference in VRPs is sent to the router.

Kind regards,
Tim

> On 26 Mar 2021, at 18:38, Chris Morrow <[email-masked]> wrote:
> 
> 
> Howdy WG Folken,
> The authors of:

---

From: Job Snijders <[email-masked]>
Date: Tue, 30 Mar 2021 12:00:23 +0200

Hi Tim, working group,

first things first: I support adoption of this draft.

On Mon, Mar 29, 2021 at 04:06:56PM +0200, Tim Bruijnzeels wrote:
> I don't see how the ROAs would be revoked in this context.

Perhaps 'revoked' is the wrong word, maybe 'withdrawn' or 'unpublished'?

I took the third suggestion in 'Section 3' to mean that if the ROA
issuer does not confirm the End-Entity certificate signing the ROA
eContent checks that it is allowed to sign the resources listed in the
eContent, it might end up submitting an invalid object to the
Repository.

The Repository deferences any previous ROA with the same filename,
performs validation on the newly submitted ROA, concludes it is invalid
(because the resources in the eContent overclaim compared to the
resources listed as subordinate on the EE cert) and ends up publishing a
CA Repository without the intended ROA.

I would suggest:

"""
3) A safeguard scheme is essential to protect the process of ROA
   issuance. Before committing to the issuance of a ROA object, the
   signer software should confirm the ipAddrBlocks in the To-Be-Signed
   eContent are fully contained by the RFC 3779 extensions listed on the
   End-Entity certificate used to generate the signature. 
"""

In other words, a signing pipeline would help the operator by performing
RFC 6482 Section 4 validation *before* constructing the ROA, to prevent
constructing ROAs the Repository operator would consider invalid.

Kind regards,

Job

---

From: Tom Harrison <[email-masked]>
Date: Tue, 30 Mar 2021 21:19:36 +1000

On Fri, Mar 26, 2021 at 05:38:28PM +0000, Chris Morrow wrote:
> of their draft.

I support adoption.

-Tom

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 30 Mar 2021 13:24:32 +0200

Hi Job, all,

> On 30 Mar 2021, at 12:00, Job Snijders <job=[email-masked]> wrote:
> 
> Hi Tim, working group,
> 

No, the publication server does not validate content. See RFC 8181:

5.  Security Considerations

   The RPKI publication protocol and the data it publishes use entirely
   separate PKIs for authentication.  The published data is
   authenticated within the RPKI, and this protocol has nothing to do
   with that authentication, nor does it require that the published
   objects be valid in the RPKI.

That being said.. as yet another aside..

I have been thinking of proposing extensions to the publication protocol (RFC 8181) to make Publication Servers somewhat more responsible. For example they could enforce quota both in terms of number of files, and size. They could also verify objects, e.g. only allow known types, and only allow objects which at least parse. But there is a potential issue then with bugs in verification code. Also, if this is done for security then it may not matter because malicious CAs could 'just' run their own publication server, until they might be revoked by an angry parent CA. Still... I think this is worth a (separate) discussion. Raising the barrier to abuse and mistakes seems like a good idea.

> constructing ROAs the Repository operator would consider invalid.

All RPKI CA implementations that I know of already take care to issue objects which do not overclaim. The problem is that a delegated CA may not find out that a parent has shrunk their resources until whenever it happens to send them a "Resource Class List Request" as defined in RFC 6492.

It would be much nicer practice if parent CAs can hold off shrinking resources until a child has had a chance to clean up. I will suggest some text to the ROV timing draft.. not sure that this belongs there, but it may be a start.

Tim

---

From: Job Snijders <[email-masked]>
Date: Tue, 30 Mar 2021 14:02:03 +0200

>    objects be valid in the RPKI.

Interesting! Today I learned something :-)

I thought the *Repository* server is expected to validate objects
submitted by its repository clients, at least that is how I interpret
the implementation of some RPKI CA software.

The text in the Security Considerations seems at odds with the existence
of the 'bad_cms_signature: Bad CMS signature' error code. How can a bad
CMS signature be detected if the receiving Repository is not validating?

I'd still suggest to expand on the 'safeguard' language to avoid ROA
creation when resources listed in eContent are not contained by
resources listed as subordinate on the EE cert. Perhaps the authors of
the draft at hand can elaborate on what was intended with 'safeguard'?

> I have been thinking of proposing extensions to the publication
> protocol (RFC 8181) to make Publication Servers somewhat more
> responsible. For example they could enforce quota both in terms of
> number of files, and size. They could also verify objects, e.g. only
> allow known types, and only allow objects which at least parse. But
> there is a potential issue then with bugs in verification code. Also,
> if this is done for security then it may not matter because malicious
> CAs could 'just' run their own publication server, until they might be
> revoked by an angry parent CA. Still... I think this is worth a
> (separate) discussion. Raising the barrier to abuse and mistakes seems
> like a good idea.

These are excellent suggestions, I agree an 'Object too large' error
code (and others) would be helpful.

> All RPKI CA implementations that I know of already take care to issue
> objects which do not overclaim. The problem is that a delegated CA may
> not find out that a parent has shrunk their resources until whenever
> it happens to send them a "Resource Class List Request" as defined in
> RFC 6492.
> 
> It would be much nicer practice if parent CAs can hold off shrinking
> resources until a child has had a chance to clean up. I will suggest
> some text to the ROV timing draft.. not sure that this belongs there,
> but it may be a start.

The trouble is that by the time the client issues a "Resource Class List
Query", the resources may already have been shrunken and 'its too late'
for the child CA.

Frequently polling the superior CA can help reduce the time window in
which the overclaiming exists, but ultimately I think the default
validation algorithm should be changed to avoid this brittleness.

I hope this group can jointly work on https://datatracker.ietf.org/doc/draft-spaghetti-sidrops-rpki-validation-update/

Kind regards,

Job

---

From: Ties de Kock <[email-masked]>
Date: Tue, 30 Mar 2021 14:21:29 +0200

Hi Tim, all,

> On 30 Mar 2021, at 13:24, Tim Bruijnzeels <[email-masked]> wrote:
> 
> Hi Job, all,
> 

I would be hesitant to add too much validation to the publication protocol. In
my opinion the BPKI relationship ensures that the content of the messages is
what the client intends to publish.

However, I would welcome the removal of some foot cannons. For example,
updating an object [and thus changing the hash of] on a manifest that currently
has entries with matching names and hashes, without also updating the manifest,
likely is a sign of a bug - not of the intention of the client.

Ties

>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/sidrops <https://www.ietf.org/mailman/listinfo/sidrops>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/sidrops <https://www.ietf.org/mailman/listinfo/sidrops>

---

From: Job Snijders <[email-masked]>
Date: Tue, 30 Mar 2021 15:01:46 +0200

On Tue, Mar 30, 2021 at 02:21:29PM +0200, Ties de Kock wrote:
> I would be hesitant to add too much validation to the publication
> protocol. In my opinion the BPKI relationship ensures that the content
> of the messages is what the client intends to publish.
> 
> However, I would welcome the removal of some foot cannons. For
> example, updating an object [and thus changing the hash of] on a
> manifest that currently has entries with matching names and hashes,
> without also updating the manifest, likely is a sign of a bug - not of
> the intention of the client.

Good points!

Another case where X509-based validation of an to-be-published object
could be detrimental for the object creator is when they want to 'stage'
ROAs that are not yet valid, and are expected to become valid at a later
moment.

Thanks for sharing this insight.

Kind regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 30 Mar 2021 15:12:10 +0200

[trimming recipient list for this side-discussion]

> On 30 Mar 2021, at 14:21, Ties de Kock <[email-masked]> wrote:
> 
> Hi Tim, all,
> 

First off, I don't think there is a huge and immediate operational issue with RFC 8181 right now, but it's good to keep an open mind to improvements.

My first take on this would be quota. And when it comes to objects I would not validate them, but just sanity check that they are of a known type and maybe that they can be parsed. I.e. add error codes as Job suggested.

We may want to extend the protocol so that quota information can be communicated. Not sure.. protocol changes are not trivial, even error codes might break things. There may be a way to do this without breaking existing relations if we include capability communication on HTTP headers for example - to help version negotiation. But it's not easy, and we should see if it's really worth it.

For consistency checks I would leave it to the CA to make sure they send all their updates as a single RFC 8181 delta.

On the other end of the possible spectrum a Publication Server could actively validate everything, and even remove things. I think that goes way too far and is dangerous, error prone and harmful. Just saying for the sake of discussion.

But.. perhaps it would be good to draft some text and have structured discussion? And yes, if we are going to discuss extending the publication protocol, then there is of course the risk, or feature, that people will want other changes too.

Tim

---

From: Randy Bush <[email-masked]>
Date: Tue, 30 Mar 2021 12:52:16 -0700

> allow known types, and only allow objects which at least parse.

why?  we need more complexity to give misconfigurations, bugs, weaseling
on specifications, etc. more places to hide?

> The problem is that a delegated CA may not find out that a parent has
> shrunk their resources until whenever it happens to send them a
> "Resource Class List Request" as defined in RFC 6492

my memory is that the child is expected to Resource Class Query before
doing anything with resources it assumes it still has.

randy

---
[email-masked]
`gpg --locate-external-keys --auto-key-locate wkd [email-masked]`
signatures are back, thanks to dmarc header butchery

---

From: George Michaelson <[email-masked]>
Date: Wed, 31 Mar 2021 09:47:49 +1000

I should have talked a bit more about the draft beyond Adoption.

Two things stood out to me. Both in Section 4. They are important to
me, because I work on the "producer" side and help operate CA systems
which would be affected by this draft if it had recommendations.

Section 4: recommendations

This reads more as a problem statement than a recommendation. I think
we need some work to discuss what is the acceptable prefix count, and
when you should exceed it or when you should go to a single prefix per
ROA. I can think of times all-or-nothing is acceptable, and might even
be what people want.

I would want to see operations and field experience here, but I do
think the CA side needs recommendations of specific numbers. Things
which can be configured into the production systems, and guidance on
how to expose this to the portal/GUI

I'm not convinced yet the single-ROA problem *is* a problem. I think
its more conjecture than reality that one ROA per announcement is a
problem. So maybe this needs more work in the draft to discuss whats
been observed as the memory effect and fetch-delay effect here?

The second thing is also in Section 4,  3) safeguard scheme. I would
like to understand what kind(s) of safeguards are being thought about.
More examples, maybe a table of problems and potential mitigations?

So, I support Adoption and I think there are things to be done here. I
can't suggest words yet because I still think I need to understand.

cheers

-George

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 31 Mar 2021 09:23:36 +0200

Hi Randy, all,

> On 30 Mar 2021, at 21:52, Randy Bush <[email-masked]> wrote:
> 

Because of third party publication servers.

Malicious/misconfigured Publication Servers and/or malicious/misconfigured CAs publishing in those, can lead huge broken repositories which can be abused to DoS RP software, as well as to distribute all kinds of less desired content.

Ultimately the response to this is reactive: parent CAs can revoke, or RPs can blacklist a repository (e.g. for using 1 TB of data).

When I publish with my CA at a third party publication server, then I would much rather do so at a server that applies some very basic sanity checks to prevent that my content becomes collateral damage of someone else's actions.

> doing anything with resources it assumes it still has.

The problem is much more likely to arise at another moment. The objects that were created were just fine when they were created. For what it's worth Krill checks with the parent every 10 minutes. But this still leaves a gap where previously issued objects can be seen to be invalid.

On the remote chance that something changed in entitlements just when operators are making a change. I do not think it's wise to block network operations on every change to ROAs with a call to a parent who may not even be available right then and there. Doing so would increase the operational co-dependency a lot. Child CAs would not just require that parent repository is available, but also that their RFC 6492 functions at all times. Furthermore the issue may be higher up in the chain and the parent may be unaware.

Also, if the CA gets additional resources on a certificate, and naively uses those in ROAs or anything *before* RPs have found the newly issued certificate, then again the child CA is seen to create invalids.

Tim