Subject: [GROW] Playing with origin validation

From: Robert Raszuk <[email-masked]>
Date: Sun, 12 Apr 2020 20:39:58 +0200

Hi,

Would anyone be able to explain the below phenomenon?

RPKI Origin validation marks net 45.227.254.0/24 as INVALID as it expects
it to be originated by ASN: 395978

c1001-08-10#sh ip bgp 45.227.254.0/24
BGP routing table entry for 45.227.254.0/24, version 221816327
Paths: (1 available, no best path)
  Not advertised to any peer
  Refresh Epoch 1
  6461 3257 42624 *51852*
    128.177.133.177 from 128.177.133.177 (64.125.0.193)
      Origin IGP, metric 100, localpref 90, valid, external
      Community: 423434093
      path 7F76F542CF58 *RPKI State invalid*
      rx pathid: 0, tx pathid: 0

c1001-08-10#show ip bgp rpki table | inclu 45.227.254.0
45.227.254.0/24      24      395978     0       10.250.80.18/8082
45.227.254.0/24      24      395978     0       10.250.80.18/8323
c1001-08-10#

But it comes from  51852 which according to ipinfo or bgpview is legitimate
ASN:

https://ipinfo.io/AS51852/45.227.254.0/24
https://bgpview.io/prefix/45.227.254.0/24

As I see similar discrepancies in many global networks I would like to ask
who to trust ? If RPKI data is not valid then I think we have a real
problem.

The particular net is a bit interesting ...

cleantalk.org reports it is coming from Swiss:
https://cleantalk.org/whois/45.227.254.0 but in the same time when searched
.240/32 is suddenly reported coming from germany and is marked as spam:
https://cleantalk.org/whois/45.227.254.240

Many thx,
Robert.

---

From: Job Snijders <[email-masked]>
Date: Sun, 12 Apr 2020 19:51:19 +0000

> it to be originated by ASN: 395978

If you look at https://stat.ripe.net/45.227.254.0%2F24#tabId=routing you
can see that the prefix is only seen by 72% of the RIPE RIS collectors,
this is a very low score, I'd consider this a problematic network
outage. If you'd have internet access users serviced out of the block it
probably would mean many websites don't work, or don't work well.

In the 'Routing History' widget we can see:

    May 2018 - Jul 2018 - AS 395978 
               Aug 2018 - AS 62088
    Oct 2018 - Mar 2019 - AS 42260
    Feb 2019 - Mar 2020 - AS 51852

I guess early someone deemed AS 395978 deemed to be the right origin ASN
and create RPKI ROAs, but subsequently didn't update these RPKI ROAs to
the new ASNs as the space moved from lessee to lessee. I suspect IP
address leasing is in play because the announcement periods don't seem
to overlap, suggesting there might have been coordination between
previous and next Origin ASN.

Since the ROA still exists, whoever created the ROA (to authorize
395978) still is in authority, so from an operational perspective it is
incumbent on that entity to correct the RPKI information. If the space
had been transferred from one LIR to another LIR the 'offending' ROA
would've been deleted in that transfer process.

> But it comes from  51852 which according to ipinfo or bgpview is
> legitimate ASN:
> 

I am not sure in what way you are reading the data, the information
displayed here doesn't weigh in on legitimacy. Both websites are
frontends to public whois data, they shows the prefix is suballocated to
'Xwin universal ltd', but the originating ASN is 'Private Layer INC'.

> As I see similar discrepancies in many global networks I would like to
> ask who to trust ? If RPKI data is not valid then I think we have a
> real problem.

I am not sure it is about trust. I trust the system works as designed,
which means there is potential for human error in the ROA creation
process. In this sense IRR, DNSSEC, and RPKI have some similarities -
they all potentially set a user up for failure.

Operators deploying OV have to the balance of inconvenience for entities
who misconfigured their ROA against the consequences of accepting BGP
misconfigurations or hijacks of prefixes which could've been prevented
had ROAs been honored.

An operator should notify its customers who are announcing RPKI invalids
before deploying Origin Validation with 'invalid == reject' policies on
the EBGP edge. This way the alert notification about the ROA
misconfiguration follows contractually established inter-organisation
communication channels. Sometimes that mechanism works well!

Another theory is that some (a lot?) of the RPKI Invalids that exist in
the default-free zone in a steady state are not really in use, just
'parked'. Folk wisdom suggests if you don't announce all your prefixes
in the DFZ, malicious actors tend to notice and start using the space in
your stead. Because of this (and other reasons) we can't really know
what IP address space is actually in use or not.

Traffic studies done by some network operators in the months prior to
deploying RPKI OV commonly show very little or no traffic destined for
RPKI Invalids. In these studies it is important to separate RPKI
invalids that become 'unreachable', and traffic for IPs covered by RPKI
invalids which are covered by a less specific not-found/valid
announcement. https://mailman.nanog.org/pipermail/nanog/2019-February/099522.html

Back to the prefix at hand, I believe this to be the party in charge of
the RPKI ROA:

    $ whois 45.227.252.0/22 | grep @ | sort -u
    e-mail:      [email-masked]

One could reach out to the operator and ask them?

Kind regards,

Job

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 12 Apr 2020 21:20:34 +0100

Robert Raszuk wrote on 12/04/2020 19:39:
> ask who to trust ? If RPKI data is not valid then I think we have a real 
> problem.

generally, rpki-invalid is dropped fairly early on in policy evaluation 
chains.  This means that rpki invalid will mostly cause a network to be 
unroutable from a practical point of view, which will put pressure on to 
ensure that that more attention is paid to keeping rpki info up-to-date.

Nick

---

From: Randy Bush <[email-masked]>
Date: Sun, 12 Apr 2020 14:12:56 -0700

i'll repeat my comment on sidrops

    >> how often do RPs fetch?
    > As often as necessary that the RP can synchronize with a repository.

    how often is 'necessary?'  in units of time, please.

    i suspect that we are judging 'success' by the number of ASs doing
    'something,' and not the quality of what they do in terms of, for
    example, publishing ROAs in a 'reasonable' time (imiho < 1hr), actual
    routers responsiveness to changing ROA registration, BGP announcements,
    etc.

i suspect we will want to clean this up before we start to hit real
problems.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 12 Apr 2020 23:21:29 +0200

Thank you Job & Nick for your comments. They pretty much do confirm what I
was suspecting that the ROA may be incorrect.

I am looking at this topic from the perspective of stub enterprise AS not a
transit operator. So with that being said in such cases of INVALID ASN I
could potentially drop it from entering my DMZ routing. Currently just
playing with lowering a bit local pref. And I am getting those prefixes
like this one from multiple peers.

In fact I found this one 45.227.254.0/24 and many others which drew my
attention by adding an automated correlation of invalid ROAs to my TCP
analyzer watching 100% of my Internet in/out traffic.

Most prefixes marked as INVALID are actually turning up as covering sources
of TCP attacks !  But not all. So I digged a bit more and found that those
which are legitimate INVALID are INVALID LENGTH not INVALID ASN.

Pretty bad that routers have no options to distinguish one vs the other
state when performing origin validation.

Of course as a workaround I could feed them with only INVALID ASN ROAs
validated on the server, but I do not recall anyone so far recommended such
practice. Those who drop INVALID would by default drop both ASN or LENGTH
isn't it ?

See another point here is that if I am at the DMZ and if I am seeing quite
a nice correlation of attacks with INVALID ASN it is tempted to apply those
prefixes as src drop ACLs shielding my servers or other DMZ infra from the
attacks. Of course cluster of inline IPS are supposed to filter those just
by learning traffic patterns on a /32 or /128 basis, but with Nx10G of
traffic they get pretty busy anyway.

Last on the part of INVALID LENGTH  ... Imagine I owe /19 of IPv4 and I am
allocating /24s in many of my global DMZs. If I do not sign my /19
aggregate I have no problem as it will be globally not found. But the
moment I sign it I must also sign all /24s I advertise as otherwise they
will become INVALID - right ? So now each time any engineering group
globally injects new /24 they also must also sign it ... Can you imagine
the operational burden here in a global company when the RPKI operation is
usually centralized ? No wonder that RIPE Validator shows 1700+ INVALID
LENGTH prefixes.

And at least in IOS XE as mentioned above I do not see a way to only
validate against INVALID ASN.

To the point Randy shared - I am afraid if prefix is leased from some
operator who is RPKI aware to the one which is not which could likely be
the root cause on the example I provided I am not sure there is process in
place to make sure the obsolete data is erased. If this is just up to
humans it is pretty much guaranteed it is going to be wrong.

Many thx all,
R.

On Sun, Apr 12, 2020 at 9:51 PM Job Snijders <[email-masked]> wrote:

> announcement.
> https://mailman.nanog.org/pipermail/nanog/2019-February/099522.html
>

---

From: Randy Bush <[email-masked]>
Date: Sun, 12 Apr 2020 14:30:44 -0700

> Imagine I owe /19 of IPv4 and I am allocating /24s in many of my
> global DMZs. If I do not sign my /19 aggregate I have no problem as it
> will be globally not found. But the moment I sign it I must also sign
> all /24s I advertise as otherwise they will become INVALID - right ?

we have been here.  the alternative is that an attacker can hole punch
you.

to steal an anology from geoff (crappy ipv6 peering and transit due to
tunnels), as more money rides on correct roa registration, distribution,
and use in routers, this will get cleaned up, both at the protocol and
ops levels.  at the moment, we're on the ugly part of the curve, with
half-assed ops, half-assed vendor code, etc.

things will get better; but keep measuring.

randy

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 12 Apr 2020 22:47:35 +0100

> state when performing origin validation.

from a practical point of view, does it matter?

There are too many prefixes out there to want to bother micromanaging 
things.  The guidelines are straightforward though: create a roa for any 
prefix you intend to announce, with the correct origin ASN. If you mess 
this up, there will be lots of incentive for you to fix it.

Nick

---

From: "Guyunan (Yunan Gu, IP Technology Research Dept. NW)" <[email-masked]>
Date: Mon, 13 Apr 2020 08:25:46 +0000

draft-ietf-sidrops-ov-egress-02 could be a good solution to the case that Robert raised. The egress OV provides the export filtering for routes with invalid ROA validation result. With this capability supported, the origin AS/AS operator is in fact able to identify that its own ROA entry is out of date and needs to be updated. However, currently there's no further action/suggestion specified other than filtering in the draft-ietf-sidrops-ov-egress-02. So, a small suggestion to the authors that maybe some specifications can be added to the draft regarding how the origin AS could/should react upon this case. 

BR,

Yunan

-----邮件原件-----
发件人: GROW [mailto:[email-masked]] 代表 Job Snijders
发送时间: 2020年4月13日 3:51
收件人: Robert Raszuk <[email-masked]>
抄送: [email-masked] [email-masked] <[email-masked]>
主题: Re: [GROW] Playing with origin validation

On Sun, Apr 12, 2020 at 08:39:58PM +0200, Robert Raszuk wrote:
> RPKI Origin validation marks net 45.227.254.0/24 as INVALID as it 
> expects it to be originated by ASN: 395978

If you look at https://stat.ripe.net/45.227.254.0%2F24#tabId=routing you can see that the prefix is only seen by 72% of the RIPE RIS collectors, this is a very low score, I'd consider this a problematic network outage. If you'd have internet access users serviced out of the block it probably would mean many websites don't work, or don't work well.

I guess early someone deemed AS 395978 deemed to be the right origin ASN and create RPKI ROAs, but subsequently didn't update these RPKI ROAs to the new ASNs as the space moved from lessee to lessee. I suspect IP address leasing is in play because the announcement periods don't seem to overlap, suggesting there might have been coordination between previous and next Origin ASN.

Since the ROA still exists, whoever created the ROA (to authorize
395978) still is in authority, so from an operational perspective it is incumbent on that entity to correct the RPKI information. If the space had been transferred from one LIR to another LIR the 'offending' ROA would've been deleted in that transfer process.

> https://bgpview.io/prefix/45.227.254.0/24

I am not sure in what way you are reading the data, the information displayed here doesn't weigh in on legitimacy. Both websites are frontends to public whois data, they shows the prefix is suballocated to 'Xwin universal ltd', but the originating ASN is 'Private Layer INC'.

> ask who to trust ? If RPKI data is not valid then I think we have a 

I am not sure it is about trust. I trust the system works as designed, which means there is potential for human error in the ROA creation process. In this sense IRR, DNSSEC, and RPKI have some similarities - they all potentially set a user up for failure.

Operators deploying OV have to the balance of inconvenience for entities who misconfigured their ROA against the consequences of accepting BGP misconfigurations or hijacks of prefixes which could've been prevented had ROAs been honored.

An operator should notify its customers who are announcing RPKI invalids before deploying Origin Validation with 'invalid == reject' policies on the EBGP edge. This way the alert notification about the ROA misconfiguration follows contractually established inter-organisation communication channels. Sometimes that mechanism works well!

Another theory is that some (a lot?) of the RPKI Invalids that exist in the default-free zone in a steady state are not really in use, just 'parked'. Folk wisdom suggests if you don't announce all your prefixes in the DFZ, malicious actors tend to notice and start using the space in your stead. Because of this (and other reasons) we can't really know what IP address space is actually in use or not.

Traffic studies done by some network operators in the months prior to deploying RPKI OV commonly show very little or no traffic destined for RPKI Invalids. In these studies it is important to separate RPKI invalids that become 'unreachable', and traffic for IPs covered by RPKI invalids which are covered by a less specific not-found/valid announcement. https://mailman.nanog.org/pipermail/nanog/2019-February/099522.html

Back to the prefix at hand, I believe this to be the party in charge of the RPKI ROA:

Kind regards,

Job

_______________________________________________
GROW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/grow