Subject: [sidr] Comments on draft-ymbk-rpki-rtr-protocol

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 15:33:22 +0000

Could the authors comment on the expected behavior of a single router with a
pair of sessions to cache servers operated by the same provider which
implement the same RPKI policy?

The use-case I have in mind is that a given provider may have a number of
cache servers deployed within their network.  A given router may wish to
have a session with two servers for redundancy purposes.

Given the current protocol, the best a router can do is know that it has all
of the records from a given cache server.  There is no way to tell that
the caches are in sync with each other or not.

By somewhat poor analogy, there's no way for me to check for the equivalent
of a DNS SOA record to see which cache may be more up to date and to "do
something" about that.

-- Jeff

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Aug 2010 18:49:16 +0200

> implement the same RPKI policy?

yes.  do not do it.

> have a session with two servers for redundancy purposes.

do not do it

> the caches are in sync with each other or not.

you can safely assume that they are not

randy

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 16:55:50 +0000

> > have a session with two servers for redundancy purposes.

It seems odd for you of all people, Randy, to want a feature that can so
severely impact traffic to have a single point of failure.  Thanks for
clearing things up though. 

I'm interested in the opinion of the rest of the WG on this operational
matter.

-- Jeff

---

From: Rob Austein <[email-masked]>
Date: Wed, 04 Aug 2010 13:08:08 -0400

> have a session with two servers for redundancy purposes.

Yes, the protocol is designed to support that.

> the caches are in sync with each other or not.

If it really cares whether the caches are exactly in sync, it could
compare the results, but I doubt that doing so would be particularly
useful.  This protocol is presenting a live view of a loosely
consistent distributed database, there is no way to force two caches
into lock step that does not involve fate sharing, which would defeat
the point of using redundant caches.

> of a DNS SOA record to see which cache may be more up to date and to "do

There is no concept equivalent to a DNS zone in the RPKI system.

Given that the state of each RPKI cache is a composite of the states
of each of the however many objects it might be tracking (best guess
for fully deployed system is tens or hundreds of thousands of objects,
collected from an arbitrary number of publication points worldwide),
the answer to your question is probably "mu", even before factoring in
fat fingers problems such as clock or trust anchor skew.

About the only useful thing i can see the router doing here is to
compare the cache update times (as perceived by the router).  If one
cache is reporting frequent updates and the other is not, odds are
that the first one is healthy and the second one is not, but even
that's just guessing, the first might just have horrible intermittent
connectivity to the outside or a whacked clock, so you might also want
to check size of the respective databases.

To a first approximation, a router using this protocol just has to
trust the cache.  There's nothing (other than CPU and storage hits) to
prevent a router that doesn't trust caches from maintaining its own,
but the point of this protocol is to get something we can deploy
cheaply on current generation hardware.  Tracking multiple caches in
case one of them dies is probably wise; attempting to get clever about
hopping from one to another in an attempt to be smarter than any of
your caches is probably beyond the intended scope of this protocol.

---

From: Rob Austein <[email-masked]>
Date: Wed, 04 Aug 2010 13:22:42 -0400

To remove any confusion caused by the apparent contradiction between
the answers that Randy and I gave:

- It's ok to have connections open to more than one cache server at a
  time, but if you're going to do that, you have to buffer the results
  for each cache separately.

- It's -not- ok to -use- the data from more than one cache server at a
  time, attempt to combine the data from multiple cache servers in any
  way, or otherwise get clever about comparing data from multiple
  cache servers.

- The only point of allowing multiple connections is to allow fast
  transfer to a new cache if the cache one has been using dies.  That
  is: we're trying to avoid the situation where you realize that the
  cache you're using has died, and it takes several minutes to get a
  full load from an alternate cache.

- The main way of detecting that the cache you're using has died is
  the (required) periodic poll; if other mechanisms happen to give you
  a hint sooner, fine, but that's the real check.

- This is all covered in the draft, albeit somewhat tersely.  If you
  think it's unclear, please send text.

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 17:44:24 +0000

Rob,

Thanks for a clearer answer than Randy's. 

> the point of using redundant caches.

The issue is less a matter of being in lock-step between a pair of caches.
My concern is that if a cache is out of step with the policy server that is
used to populate that cache.  In this particular case, the protocol will
move along happily and you'll see updates from one server, but not the
other.

What I had in mind was a simple message from the cache server noting what
its database version number was from the policy server and the last update
time.  This would permit a router to note when a cache has become radically
desynchronized from its redundantly configured cache.

> fat fingers problems such as clock or trust anchor skew.

I think I see my confusion.  I wouldn't have expected a provider to operate
multiple pieces of infrastructure to gather RPKI objects and generate policy
from them.  Since consistency of policy is important, this scenario didn't
occur to me.

Given the assumption above of more than one RPKI object collector and policy
generator, I can see how a domain-wide policy serial number would be
problematic.  Thanks for the clarification.

> cheaply on current generation hardware.

I agree that the protocol accomplishes this nicely.

> Tracking multiple caches in

I'd say I'm still concerned about redudannt cache integrity and that the WG
should spend a few cycles considering reasonable and inexpensive to
implement solutions.  As I pointed out to Randy, a single point of failure
for such a stateful mechanism is problematic.

Random example since I'm here:

We learn 10/8 le 24 from cache servers A and B.  The refcount is thus 2.
A becomes desynchronized for some long period of time.
B withdraws 10/8 le 24.

The remaining refcount is 1 for as long as it stays desynchronized.  Thus a
previously authorized prefix which should be unauthorized remains in the
system.

Bounding the time that such a thing stays in the system would be my goal.

-- Jeff

---

From: Rob Austein <[email-masked]>
Date: Wed, 04 Aug 2010 14:06:53 -0400

At Wed, 4 Aug 2010 17:44:24 +0000, Jeffrey Haas wrote:
> 
> My concern is that if a cache is out of step with the policy server
> that is used to populate that cache.

This description makes no sense in the context of the current protocol
and implementations.  You've introduced an independent element between
the RPKI and the rpki-router protocol server (your "policy server"),
then postulated a failure of that element.  How you handle failures in
elements you introduce would be between you and your rabbi. :)

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Aug 2010 20:37:07 +0200

> The issue is less a matter of being in lock-step between a pair of
> caches.  My concern is that if a cache is out of step with the policy
> server that is used to populate that cache.

i have no idea what a policy server is.

> What I had in mind was a simple message from the cache server noting
> what its database version number was from the policy server and the
> last update time.  This would permit a router to note when a cache has
> become radically desynchronized from its redundantly configured cache.

there are no redundantly configured caches.  there are no database
version numbers.  read the draft.

> I think I see my confusion.  I wouldn't have expected a provider to
> operate multiple pieces of infrastructure to gather RPKI objects and
> generate policy from them.  Since consistency of policy is important,
> this scenario didn't occur to me.

and a globaly consistent view of the dns is important.  it just happens
not to exist.  and global bgp never really converges.

from section 5

   As caches can not be rigorously synchronous, a client which changes
   servers can not combine data from different parent caches.
   Therefore, when a lower preference cache becomes available, if
   resources allow, it would be prudent for the client to start a new
   buffer for that cache's data, and only switch to those data when that
   buffer is fully up to date.

i could add a lot of words, but i don't think it would make things much
clearer.  i am thinking of including the roll slide from the maastricht
preso in the next version.  but some folk have pointed out that there
are other recipies that are as good.

> Given the assumption above of more than one RPKI object collector and
> policy generator, I can see how a domain-wide policy serial number
> would be problematic.  

how about the assumption of multiple object collectors and no policy
generator(s)?

> I'd say I'm still concerned about redudannt cache integrity and that
> the WG should spend a few cycles considering reasonable and
> inexpensive to implement solutions.

when you actually have a proposal, do make it.

> As I pointed out to Randy, a single point of failure for such a
> stateful mechanism is problematic.

what single point of failure?  and please refer to the draft and do not
invent components which are not there.

randy

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Aug 2010 20:42:08 +0200

> severely impact traffic to have a single point of failure.

sans ad homina, what is the spof?

i suspect you have not read the, admittedly and intentionally terse,
draft carefully.

randy

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 19:02:09 +0000

> elements you introduce would be between you and your rabbi. :)

Let me attempt to correct my incorrect verbiage.  It's been a while since
I've read the -arch document.

-arch notes that a provider must maintain a Local Cache.

The Local Cache may contain configuration - policy - that directs things
like which TAs to use.  After processing the RPKI objects, the output is
effectively a database of prefixes (and max-lengths) and their origin ASes.

A provider may wish to override validated RPKI data for their own purposes.
While not explicitly a SIDR-driven requirement, this was discussed multiple
times as a requirement during the original RPSEC work.

The resulting output (override or no), is a set of AS:prefix/maxlen tuples
that reflects the provider's routing policy with regard to origin
validation.

This policy must now be distributed to routers for implementation.  The
mechanism for distributing the policy is currently
draft-ymbk-rpki-rtr-protocol.

Implementation/Deployment models:
1. The Local Cache with resulting routing origin policy and the rpki-router
protocol server are in the same box.

In this model, the provider will likely have to deploy more than a single
instance of this model throughout their network.  This will result in each
box hitting the RPKI repositories in order to keep their state fresh.
Presuming the lack of a mechanism to keep each box in lock-step with their
state, this provides a scenario where these servers may become out of synch
with each other.  This may yield inconsistent routing origin policy
within a given network.

2. The Local Cache with resulting routing origin policy is implemented
within one box.  The rpki-router protocol servers are located in multiple
boxes.

In this model, the single Local Cache minimizes the impact on the global
RPKI repository infrastructure.  A mechanism is required to distribute the
resulting origin routing policy to the rpki-router servers (let's call it
scp for simplicity's sake).  The only way for this infrastructure to provide
inconsistent routing origin policy within the network would be for the Local
Cache to be topologically isolated from the rpki-router servers.  This is a
detectable condition at the Local Cache presuming a push model where it is
noted that the push fails.

In both models there is a possibility that the rpki-router server session is
up but the server session is out of sync with the RPKI within a given
network.  This yields inconsistent policy which is usually considered a 
Bad Thing.  If you don't have this perspective, you must be talking to
different providers than I am.

I believe it is possible to minimally update the rpki-router protocol to
allow routers to detect policy skew.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 19:28:06 +0000

To amend my last post:

> with each other.  This may yield inconsistent routing origin policy

The draft also covers a Large End Sites which have a hierarchy of servers.
Scenario 1 still roughly covers this hierarchy where intra-provider cache
servers may not be consistent.

-- Jeff

---

From: Rob Austein <[email-masked]>
Date: Wed, 04 Aug 2010 15:41:17 -0400

Last message for a bit, as I do have other things I need to do today.

> -arch notes that a provider must maintain a Local Cache.

Yes.

> effectively a database of prefixes (and max-lengths) and their origin ASes.

And other data which you appear to be ignoring, but let that pass for now.

> times as a requirement during the original RPSEC work.

What RPSEC did or did not say is not my problem, but I agree that an
operator has policy settings somewhere.  At present, they're mostly in
the router itself, thanks to Doctor Knob; the only policy setting in
the RPKI part of the picture at the moment is choice of trust anchor,
which is a basic configuration issue for the validator and something
we almost certainly do not want to extrude into the rpki-rtr protocol.

> that reflects the provider's routing policy with regard to origin

You're jumping to the conclusion that policy is applied before
distribution.

> draft-ymbk-rpki-rtr-protocol.

No.  That protocol is specified as distributing distilled RPKI data
(or distilled IRR data, if you're trusting enough to use that).  Show
me where that draft says -anything- about distributing policy.  The
word never even appears in the draft.

The rest of your message is a detailed derivation of how you get into
trouble by attempting to feed policy down a protocol pipe that was not
designed to carry it.

I will note that, of your two options, ignoring your injection of
policy into the protocol, the first one (co-location of RPKi validator
with rpki-rtr server) is the one we envisioned, although one can
imagine more complex topologies where validators fetch from
pre-aggregated caches generated by other validators, to reduce
external load and promote a weak form of coherence.  Note that in a
complex topology the object timestamps will tend to do what you want
automatically, given that we're using rsync as the RPKi data gathering
tool.  Also note that all of the fancy topology stuff would happen in
front of RPKI validation, by the time one has gotten past that it's
just the rpki-rtr protocol feeding results to the routers.

Your second option, on the other hand, creates a nasty single point of
failure.  Insert Randy's standard comment about encouraging all of my
competitors to do that.

But I really think your problem here is confusion of the rpki-rtr
protocol with a policy distribution service.  It's not intended to be
that, thus your call for a scheme to detect skew in something it was
never intended to carry looks out of scope to me.

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 19:49:30 +0000

On Wed, Aug 04, 2010 at 08:37:07PM +0200, Randy Bush wrote:
> there are no redundantly configured caches.

: 7. Router-Cache Set-Up
: [...]
:    A router may be configured to peer with a selection of caches

> there are no database version numbers.

That was my proposal and not my assertion that your protocol included this.

>   read the draft.

Aside from my sloppy verbiage, you're mistaken if you think I haven't.  It's
why I'm asking questions.

>    buffer is fully up to date.

And the crux of my question is, why can't they be synchronous at least
within a given provider domain?  Or at least synchronous enough?

> how about the assumption of multiple object collectors and no policy

The default policy is your choice of trust anchors.

> desynchronized from its redundantly configured cache.

At this point, the deployment model that appears to be documented (Section
8) seems to presume that it's okay to have inconsistent RPKI policy within a
provider.  If everyone is cool with that, we're done.

If not, then we have more work to do, even if it's not a simple database
version number. 

> invent components which are not there.

This was in response to your original comment:
> > have a session with two servers for redundancy purposes.

If you can't have a session with more than one server, you have a single
point of failure.

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Aug 2010 21:56:28 +0200

> At this point, the deployment model that appears to be documented
> (Section 8) seems to presume that it's okay to have inconsistent RPKI
> policy within a provider.  If everyone is cool with that, we're done.

there is no rpki policy in this document.  hence such policies can not
be inconsistent.

Abstract

   In order to formally validate the origin ASes of BGP announcements,
   routers need a simple but reliable mechanism to receive RPKI
   [I-D.ietf-sidr-arch] or analogous prefix origin data from a trusted
   cache.  This document describes a protocol to deliver validated
   prefix origin data to routers over ssh.

i encourage you to describe and document a protocol which maintains
synchronized and skew-free trusted caches.  sorry, i did not know how,
and did not think it was actually needed or appropriate in this context.

randy

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 20:21:27 +0000

> we almost certainly do not want to extrude into the rpki-rtr protocol.

I think you have chosen to take the word policy and apply it in the
route filtering sense rather than my intended RPKI sense.  Please insert
whatever word you want to use that directs my use of which objects from the
RPKI I choose to be permitted for validation purposes.  The simplest example
of this would be "I choose not to trust Foo-NIC as a trust anchor".  Another
example would be "I choose to ignore AS64512's objects".  My intention was
never "drop 10/8 le 24 from BGP" in the style of RPSL.

> just the rpki-rtr protocol feeding results to the routers.

Let's stick with this example since it's sufficient for my point as long as
we're talking about a set of caches in a single domain/AS/administrative
region/etc. and have consistent configuration.

Ignoring rpki-rtr, there should be a mechanism to determine whether all of
the local caches within the domain are consistent with each other.
Pathologically, this is each cache rsync'ing with every other cache and
finding no changes.  More realistically, it'd be agreement that all objects
of a certain age are present on all caches.

Passing that age into the rpki-rtr protocol is sufficient to establish a
measure of confidence in the consistency of caches in the same domain for
differing sessions.

-- Jeff

---

From: Randy Bush <[email-masked]>
Date: Wed, 04 Aug 2010 22:29:40 +0200

> never "drop 10/8 le 24 from BGP" in the style of RPSL.

i need to sleep, long day tomorrow, but one more try.

this document is not about how, why, and under what policies you get
whatever you get into whatever cache(s) you trust.

please read the above sentence a few times slowly.

this document is utterly independent of what trust anchors, chicken
entrails, or borg mind controls caused whatever data to be put in the
cache(s).

this document is about how we get those data UNEDITED AND WITH NO POLICY
APPLIED from one or more of these caches to a router.

randy

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 4 Aug 2010 20:34:16 +0000

> whatever you get into whatever cache(s) you trust.

Ignore policy.  Answer the second paragraph in the context of *unedited
cache consistency* from the  same mail you just quoted and we're done.

-- Jeff

---

From: Randy Bush <[email-masked]>
Date: Thu, 05 Aug 2010 12:28:12 +0200

>  Answer the second paragraph in the context of *unedited cache
> consistency*

givem the rpki operational structure, it would be deliciously wonderful
if you could describe how multiple gathered caches thereof can be
consistent.

anxiously awaiting your design.

randy

---

From: [email-masked]
Date: Thu, 5 Aug 2010 10:50:42 +0000

> if you could describe how multiple gathered caches thereof can be

	consistent with what exactly?
	i would hope/expect that the caches w/in a given
	AS should be consistent within the AS but have no
	illusions about external consistancy with anyone
	else (other AS or RIR)

--bill

---

From: Randy Bush <[email-masked]>
Date: Thu, 05 Aug 2010 12:53:49 +0200

> consistent with what exactly?

as no other entities were mentioned, it would be safe to assume
eachother.

> i would hope/expect that the caches w/in a given AS should be
> consistent within the AS but have no illusions about external
> consistancy with anyone else (other AS or RIR)

while you are hoping and expecting, send design for a scheme to assure
this.

until then, i have real work to do.

randy

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 5 Aug 2010 11:40:01 +0000

Bill,

> 	illusions about external consistancy with anyone

The model described is likely to lead to inconsistent route origin policy
for eBGP speakers within a given AS for short periods of time.  How short,
and whether the behavior covered by a given ROA will converge to
consistency within the AS will depend on the deployment model.

-- Jeff

---

From: Stephen Kent <[email-masked]>
Date: Thu, 5 Aug 2010 10:23:29 -0400

>...

A specific proposal for how to locally override the RPKI structure as 
retrieved form the repository system is contained in a document being 
considered as a SIDR WG item: 
http://tools.ietf.org/html/draft-reynolds-rpki-ltamgmt-00

>that reflects the provider's routing policy with regard to origin

In the case of the doc cited above, it is not in that form.

>draft-ymbk-rpki-rtr-protocol.

I think the ymbk protocol distributes the results of validating the 
RPKI data, with whatever policy has been applied to that validation 
process.

Steve