Subject: Re: [Sidrops] WGLC - draft-ietf-sidrops-validating-bgp-speaker - ENDS 09/07/2018 - Sept 7th 2018

From: Christopher Morrow <[email-masked]>
Date: Tue, 4 Sep 2018 11:57:29 -0400

I think restating an earlier conversation about this draft (which I think
was the rpki-light draft name) is useful:
  "The draft aims to provide the ability for a third-party (IXP) to
validate routes for me, because I can't do that myself"

The reasons why 'I' may not be able to do the validation are perhaps
shifting in time though, which I think should be acknowledged in this
conversation that's progressed since the original draft was proposed.

On Fri, Aug 24, 2018 at 10:33 AM Daniel Kopp <[email-masked]> wrote:

>
> The initial idea for the draft came from network operators at IXPs.
>

It's worth noting that -ops groups in the IETF are supposed to gather
requirements from actual operators and coalesce these into change proposals
for ietf protocol groups as part of their charters. particularly SIDROPS
should be taking requirements for SIDR protocol deployments from the bgp
operating community(ies). So, it's great to get feedback from IXP operators
and network operators for requirements/improvements/deployment-guidance.

> The idea of the draft is to help with the adoption of RPKI in the domain
> of IXPs.
> The idea draft is not a replacement for RPKI validation or a global
> distribution of RPKI validation results.
> The draft is there to help at IXPs till we all drop invalids by default :)
>

I think ~2yrs back there were more 'problems' with the idea of deployment
of route origin validation and taking action(s) upon that decision process.
It's possible that in today's review of the problem space the problems have
changed or resolved, right? It sounds like, from the discussion, that some
operators (of IXP and ISP networks) are saying that:
  1) "route origin validation is not as hard to see deploying today"
  2) "just introducing an IXP lan/RS that simply implements the validation
and takes action(s) is the right course of action"
  3) "Using the IXP's RS feed in the case of: 'please drop routes based on
validation state' can get data/experience to the IXP participants such that
they can then deploy route origin validation in their networks as well"

Maybe  we can re-frame the discussion along those lines?

-chris

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 5 Sep 2018 00:03:25 +0100

Christopher Morrow wrote on 04/09/2018 16:57:> I think restating an 
earlier conversation about this draft (which I
> think was the rpki-light draft name) is useful:
rather than having a discussion about theoreticals, I set up an RPKI 
cache from scratch on a VM earlier this evening and tied it into an XR 
router connected to an IXP.  The RPKI cache ran on vanilla centos7 with 
the RIPE NCC RPKI validator v3 software.  Here are the installation 
instructions for the validator software:

> https://github.com/RIPE-NCC/rpki-validator-3/wiki/RIPE-NCC-RPKI-Validator-3-Production

Two minor additions were made to the 8 commands required to install the 
software and get it running: one to get the rpki-rtr process to bind to 
the public IP address of the server in question, and the second was to 
install the ARIN TAL.

The router was configured to tag and accept invalids.  It took 6 lines 
of copypasta config (3 policy and 3 bgp/rpki) pulled from a couple of 
quick internet searches to do this.

The result of this was a fully operational RPKI PoC which validated 
prefixes on the router immediately.  Embarrassingly, it took longer to 
beat the firewall into allowing access to the rpki-rtr server than it 
did either to configure the router or to set up the rpki validator 
service (this was due to a typo: udp/8323 instead of tcp/8323, oops).

In terms of router stack support, rpki validation has been there for 
years on Junos, IOS-XE, XR, SR-OS, vanilla IOS on some platforms and 
various other commonly-used router stacks.  It's not there on some kit, 
including, for example, vanilla IOS on older non-core-oriented legacy 
kit and Mikrotik RouterOS, which gets a mention mostly because it's 
widely used at IXPs, despite its shortcomings.

Obviously a PoC is not a production configuration, but the point of the 
exercise was to show that the bar for a basic rpki implementation has 
been lowered to the point of it requiring no more than a skeleton linux 
deployment and 8 lines of cut-n-paste from a wiki.  It isn't rocket 
science, and it is widely supported on the sorts of routers that people 
use in production environments, particularly IXPs, i.e. the pool of 
situations where people can't deploy rpki is continuing to shrink and 
there is no real operational need to create standardised hacks for 
situations where it can't be deployed.

Nick

---

From: Randy Bush <[email-masked]>
Date: Tue, 04 Sep 2018 23:07:29 -0700

>   1) "route origin validation is not as hard to see deploying today"

see draft-ietf-sidrops-ov-clarify-05.txt.  essentially vendors'
implementations are funky and so it is hard for me to do it myself.

and it is not only ov-clarify.  test whether your fave implementation
re-evaluates a bgp prefix when a roa change comes in over rpki-rtr.  the
messy story goes on.

> and takes action(s) is the right course of action"

s/the right course/one right course/

what is nice is that the ixp-provided filter does not have the same
problems as above.  so it is a leapfrog while hardware vendors catch up.
it is driving origin validation deployment.

randy

---

From: Job Snijders <[email-masked]>
Date: Wed, 5 Sep 2018 09:34:54 +0200

> re-evaluates a bgp prefix when a roa change comes in over rpki-rtr.  the

the IXP route server software (that is used in practise) is provided by
a single vendor. Perhaps this single vendor got it right, but from a
conceptual point of view they wouldn't be exempted and may still need
'ov-clarify'.

> problems as above.

because of the single vendor? or because of some other reason? how
exactly is their bgp more magic than others?

> so it is a leapfrog while hardware vendors catch up.  it is driving
> origin validation deployment.

except that we cannot point at a single instance where the approach has
driven origin validation. on the other side, i can point at many
instances where IXP route servers have negatively impacted businesses
because they propagated/amplified incorrect routing information.

What /actually/ drives origin validation is customers asking their
providers (be it ISPs or IXPs) to implement origin validation. This
approach has already been succesfully tested at FranceIX, AMS-IX, and
later this year will be yield positive results at DE-CIX and LINX.

I have to ask, (given the author's affiliations) - if this draft is
published as an RFC, will you turn back the clock and start propagating
invalid route announcements to your customers (marked with an extended
community)? 

Kind regards,

Job

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 5 Sep 2018 15:51:09 +0100

> re-evaluates a bgp prefix when a roa change comes in over rpki-rtr.  the
[...]
> what is nice is that the ixp-provided filter does not have the same

really it does.  As Job suggested, the majority of ixp route servers run 
BIRD, and taking the example you mention, one of BIRD's known 
limitations is that it does not handle revalidation.  Another limitation 
would be that it doesn't handle aggregators as the last element in the 
as path.

In general, if the IXP uses any particular bgp stack (whether ios-xe, 
junos, bird, etc), it will be stuck with the bugs and shortcomings of 
that particular implementation, and under the proposals of the 
validating-bgp-speaker draft, everyone at the ixp will be subject to 
those particular bugs and shortcomings.  It's not valid to say that the 
IXP rpki implementation will be any better than a "hardware" router 
because - as Job pointed out already - it's just software under the hood.

Nick

---

From: Randy Bush <[email-masked]>
Date: Wed, 05 Sep 2018 07:56:11 -0700

> As Job suggested, the majority of ixp route servers run BIRD, and
> taking the example you mention, one of BIRD's known limitations is
> that it does not handle revalidation.

then it is pretty much useless.  

> Another limitation would be that it doesn't handle aggregators as the
> last element in the as path.

don't care.  they're irrelevant.

randy

---

From: Job Snijders <[email-masked]>
Date: Wed, 5 Sep 2018 17:06:38 +0200

Hi,

As an experiment, can FranceIX and DE-CIX propagate a more-specific of the
AMS-IX peering LAN prefix via their route servers (but clearly mark it with
an extended community). This way, all the authors’ organizations, can make
observations on customer feedback.

In the hours that follow we can perhaps learn of the consequences of
willingly propagating RPKI invalid route announcements.

This would be a repeat of the incident that happenend at DE-CIX on December
5th, 2017. I hope people still remember.

Kind regards,

Job

On Wed, 5 Sep 2018 at 16:56, Randy Bush <[email-masked]> wrote:

> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops
>

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 5 Sep 2018 12:39:49 -0400

> then it is pretty much useless.
"is pretty much useless" - Needs bug(s) filed and development work in order
to rectify this problem.
I'd caution against: "is useless, never talk to it again" because ...
ideally we want all the bgp software folk to do the right thing here, right?
 so ideally ov-clarify already whacks "popular hardware vendors" on the
nose, let's make sure the 'popular software vendors' also have bugs filed
against them?

> don't care.  they're irrelevant.
"they are irrelevant" - because of the total DFZ there are X% and of that
set Y% have a supernet with the same path so the subnet 'does not matter'
for routing's behaviour.

right?

---

From: Daniel Kopp <[email-masked]>
Date: Thu, 6 Sep 2018 13:21:49 +0000

> On 5. Sep 2018, at 09:34, Job Snijders <[email-masked]> wrote:
> 

It’s not turning back the clocks. There is no origin validation at DE-CIX yet and as far as I know, DE-CIX wanted the draft to drive the implementation.
Moreover the draft defines the different modes of operations, so as far as I know also AMS-IX wants to add the tagging to their current implementation.

Kind regards,
Daniel

---

From: Job Snijders <[email-masked]>
Date: Thu, 6 Sep 2018 15:40:26 +0200

On Thu, Sep 06, 2018 at 01:21:49PM +0000, Daniel Kopp wrote:
> > published as an RFC, will you turn back the clock and start
> > propagating invalid route announcements to your customers (marked
> > with an extended community)? 
> 
> It’s not turning back the clocks. There is no origin validation at
> DE-CIX yet and as far as I know, DE-CIX wanted the draft to drive the
> implementation.

In that case, it appears the draft has been taken over by current
events?  From what I understood from your organisation's communications
related to deploying RPKI origin validation there has been a commitment
to deploy this fall.

> Moreover the draft defines the different modes of operations, so as
> far as I know also AMS-IX wants to add the tagging to their current
> implementation.

This puzzles me - AMS-IX already implemented this, some documentation
can be found here: https://ams-ix.net/technical/specifications-descriptions/ams-ix-route-servers/route-server-filtering
(note the 6777:65012, 6777:65022, and 6777:65023) communities.

Kind regards,

Job

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 7 Sep 2018 08:17:35 +0100

Christopher Morrow wrote on 05/09/2018 17:39:
> On Wed, Sep 5, 2018 at 10:56 AM Randy Bush <[email-masked] <mailto:[email-masked]>> wrote:
> "is pretty much useless" - Needs bug(s) filed and development work in 
> order to rectify this problem.
> nose, let's make sure the 'popular software vendors' also have bugs 
> filed against them?

from the bigger picture point of view, buggy rpki implementations are 
irritating but implementation problems like this are not sufficient 
reason for the ietf to create workarounds, particularly when those 
workarounds require new code on router stacks, i.e. have a substantial 
material cost and require debugging.  As an aside, router vendors have 
shown historically that they don't much like supporting cli glue for 
extended communities, so it's difficult to see how they'd write code for 
this.

The ov-clarify draft will probably help deployment in the longer term, 
as it creates a tickbox which will end up on RFPs and compliance testing 
suites.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 7 Sep 2018 08:23:08 +0100

> don't care.  they're irrelevant.

they're irrelevant until the point that they're not.  I mentioned them 
because they crop up from time to time at IXPs, which is where you 
notice that your traffic routing is being handled by covering supernets 
rather than your more-specifics.  The last ticket we had on this was 
earlier this summer.

Nick

---

From: Daniel Kopp <[email-masked]>
Date: Fri, 7 Sep 2018 14:03:33 +0000

> On 6. Sep 2018, at 15:40, Job Snijders <[email-masked]> wrote:
> 

No, the draft hasn’t been taken over by other events. 
The idea was to adjust the implementation according to the draft.
So I don’t see that this draft is outdated in that sense.

> (note the 6777:65012, 6777:65022, and 6777:65023) communities.

Yes, AMS-IX has already some flavour of implementation for that… and one goal of the draft is to have a common and well defined way how to implement this.

Kind regards
Daniel

---

From: Ruediger Volk <[email-masked]>
Date: Sat, 08 Sep 2018 04:19:47 +0200

> Howdy WG folk!
  > The authors of: draft-ietf-sidrops-validating-bgp-speaker
  > are thinking their draft is ready to move forward, there hasn't been
  > significant conversation about this document since the last meeting in
  > Montreal... so:

I believe this looked much more innocent (just the document name?)
at the adoption call (and slipped under my radar - like too many things).

This draft is not acceptable for a number (>1) of reasons.
The primary being: it introduces security problems by
carrying security information over uncontrolled paths
without securing authenticity.
By choosing extended community as the container for the unauthenticated information the problem is made worse,
because for a long time we cannot assume that operator will
have methods operational for controlling propagation of that unauthenticated security information; and the draft does 
not seem to propose or consider any methods and practices 
for controlling and limiting propagation - except .

To illustrate the class of problem I give an example:
I'm operating AS64510. Anybody connected to the global BGP 
can inject routes with the proposed extended community
or (using special hacked BGP) the extended community
to routes it propagates indicating whatever validation state
suits their purpose and indicate that high reputation AS64510
is source of the validation state.
Now AS64510 does not even has a way diagnosing presence or removing
of that fake information; note: for handling extended communities
vendors actually have to spin code - even to support generic
policy primitives such as delete or match (as a decision criterium).

The security considerations are certainly seriously incomplete.
Explicit and prominent note of transitive propagation of
unauthenticated security information MUST be added
(even a reference to general BGP security analysis that
warns about missing authencation and authenticity of BGP routes
and attributes is missing!).

Some consideration of the trust relations involved with signaling
the extended community or using it as a relying party in 
local policy would be needed.
>From that point of view it looks strange that no mention of
the relying party controlling which source over which path
it trusts.

While searching through the document for the security handling
I came across quite some spots that looked like somewhat fuzzy
in terminology or otherwise lacking - not ready for publication.

In some protocol proposals it is a good idea to clearly identify
the functions of implementations that need configuration,
and whether the default should be ON or OFF.
Here it quite definitely ought to be default OFF!

I believe the draft should be declared dead;
perhaps after a final spin that inserts a security warning..
Discussion of requirements and reasonable approaches 
to reduce operational burden of origin validation
may be needed.  However this draft not help for that!

The basic signal flow idea of this draft could be handled easily
in already available/deployed router software with user 
configured policy with much better control;
someone desperate could be using it in a months time ./.
waiting more than a year for new router software version.
+ we should help the vendors to focus on fixing and completing
the standard RPKI OV functionality (think clarify-ov)
and avoid diverting energy and attention to another new feature.

Different approaches to outsourcing of the security function
may be more secure and much easier to configure.
Documenting a solution obviously would be helpful to some
and could help to avoid some bad security designs. 

Ruediger Volk