Subject: Re: [Sidrops] [WGLC] draft-ietf-sidrops-rpkimaxlen-08 - ends 25/October/2021

From: Alexander Azimov <[email-masked]>
Date: Tue, 7 Dec 2021 20:58:00 +0300

Dear colleagues,

I mixed up the 'reply all' and 'reply' buttons, and only Job was able to
see my reply and keep on the discussion.
I'm sending the original message to the mailing list so we can proceed from
this point.

ср, 24 нояб. 2021 г. в 13:42, Job Snijders <[email-masked]>:

> On Tue, Nov 23, 2021 at 04:43:26PM +0300, Alexander Azimov wrote:
> > In the world with high adoption rate of ROV I would have definitely
> > supported this document for publication.
> > Unfortunately, according to latest research by Geoff Huston - we are
> > living in a very different world.
>
> Can you provide a reference for this 'latest research'?
>
Oops, forgot to add it:
https://www.potaroo.net/presentations/2021-10-20-isoc-rpki.pdf

> > We have steady progress in signing ROAs but far less progress in
> > adopting ROV.  And you can't easily distinguish is it safe enough to
> > start using maximal ROA length or not. For the network that I'm
> > responsible at the moment - it is not safe.
>
> Can you elaborate? This seems a rather subjective statement. In any
> regard, you are free to configure your own ROAs as you deem fit for
> purpose. :-)

> Another complication may happen when ROV is done at egress (rfc8893).
> > In case of fast de-aggreagation you may need to turn it off, though
> > such issue is much easier to resolve.
> > To sum it up: I still have a feeling that the document is ahead of its
> > time, and in the imperfect world may itroduce new gaps between
> > routing-security-in-theory and operations.
>
> I fail to see how this document introduces new gaps. It merely warns
> about the potential pitfalls of using the maxlen macro functionality.
>
Yes, I'm free to configure whatever I like, and the authors of the document
are free to do it in a different way.
The difference is that authors of these documents suggest their operational
practice as BCP, and I can't agree that this practice is safe at the moment.

Let's elaborate what kind of events we are facing at the moment.
The majority of hijacks and route leaks are mistakes. We can assume that
the difference between accidental and malicious hijacks is absence of
forged as path in case of accidental events.
So, ROA with ANY prefix length is good enough to limit the blast radius of
accidental hijacks.

Unfortunately ROV deployment isn't at the level when we are no longer
affected by accidental hijacks. And the best option is instant prefix
deaggregation, if it is possible.
And I hope you will agree - the time is precious.

Now let's discuss how the recommendations from this document may complicate
things that already got nasty due to the fact that your address space was
hijacked.

*Scenario 1:*
Being a proactive and BCP aware autonomous system holder, one may add
egress ROV with strict policy to drop Invalid routes with an exception to
blackhole routes.
In case of a hijack this good citizen will have to relax it's routing
policy/turn off ROV on egress. It is still not that bad, but in huge
companies with complicated access rules to devices, it may still take time.

*Scenario 2:*
Previously I described this scenario as two islands of good and bad actors,
where good actors perform ROV and sign minimal ROA, bad actors that perform
hijacks and 'ocean' of Internet in the middle. In this scenario good actors
lose opportunity for deaggregation without updating ROA records. Let's give
it a more realistic and simple shape.

Let we have an ASX that peers with two 'important ISPs' in selected
regions. These two ISPs (ASY and ASZ) have significant intersections in
their customer cones.
ASX creates ROA, ASZ does perform ROV, ASY doesn't. And a customer of ASY
makes an accidental more specific hijack, and due to lack of filtering it
is accepted by ASY.
Since it is customer's route it gets higher local preference, and prefix
deaggregation toward ASY doesn't help.
But deaggregation toward ASZ may be helpful, since we will have two routes
with same length competing in the intersection of customer cones of ASY and
ASZ.

If ASX has ROA with maximum prefix length, this simple technique will
reroute back to ASX at least portion of the traffic.
If ASX has ROA with minimal prefix length - effective deaggregation will be
possible only after ROA update.

So IMO minimal prefix length will be safe if all 'important' peers of
selected AS support ROV. For Yandex this statement is not true.
I wonder how many networks can say that they fulfill this requirement
today.

ср, 24 нояб. 2021 г. в 13:42, Job Snijders <[email-masked]>:

> On Tue, Nov 23, 2021 at 04:43:26PM +0300, Alexander Azimov wrote:
> > In the world with high adoption rate of ROV I would have definitely
> > supported this document for publication.
> > Unfortunately, according to latest research by Geoff Huston - we are
> > living in a very different world.
>
> Can you provide a reference for this 'latest research'?
>
> > We have steady progress in signing ROAs but far less progress in
> > adopting ROV.  And you can't easily distinguish is it safe enough to
> > start using maximal ROA length or not. For the network that I'm
> > responsible at the moment - it is not safe.
>
> Can you elaborate? This seems a rather subjective statement. In any
> regard, you are free to configure your own ROAs as you deem fit for
> purpose. :-)
>
> > Another complication may happen when ROV is done at egress (rfc8893).
> > In case of fast de-aggreagation you may need to turn it off, though
> > such issue is much easier to resolve.
> > To sum it up: I still have a feeling that the document is ahead of its
> > time, and in the imperfect world may itroduce new gaps between
> > routing-security-in-theory and operations.
>
> I fail to see how this document introduces new gaps. It merely warns
> about the potential pitfalls of using the maxlen macro functionality.
>
> Kind regards,
>
> Job
>

-- 
Best regards,
Alexander Azimov

---

From: Job Snijders <[email-masked]>
Date: Wed, 8 Dec 2021 11:46:18 +0100

Hi Alexander, group,

What puzzles me is why the state of global RPKI ROV deployment bears any
relevance to the proposed BCP at hand.

For networks which do NOT deploy RPKI ROV 'invalid == reject' policies,
it doesn't matter what the maxlength macro value is set to; it doesn't
even matter whether the ROA exists at all! For those networks RPKI
doesn't exist :-)

On the other hand, for networks which DO honor RPKI ROAs by having
deployed RPKI ROV, a minimal maxlength value is advantageous (because of
the reasons outlined in the draft).

Resource holders wishing to deaggregate at a moment's notice (like
Alexander's employer?) - apparently their best course of action is to
either create most-specific ROAs, or set maxlength to /24. Why? Because
of the widespread RPKI ROV deployment :-)

On the topic of egress ROV: this is what SLURM can be used for, or a
clause in your routing policy similar to what you mentioned about
blackholes. If instant deaggregation is a requirement, I think it is
safe to assume operators will ahead-of-time prepare for such
eventualities. Indeed, counter-hijacking is complicated. Minimal ROAs
does not preclude one from makin a best effort to counter-hijack.

I checked this morning, for only 51,858 out of 297,901 VRPs maxlength is
set, so it seems to me a majority of the industry already adheres to
using minimal ROAs.

Kind regards,

Job

---

From: Ben Maddison <[email-masked]>
Date: Wed, 8 Dec 2021 14:09:44 +0200

Hi Alexander,

> I'm sending the original message to the mailing list so we can proceed from
Thanks!
> [...]

The draft contains substantial discussion of the tension that exists
between minimising attack surface and retaining the ability to make
ad-hoc route origination changes.

The whole of sections 5.1 and 5.2 are dedicated to the topic, and
essentially say "some operators may choose to publish non-minimal ROAs
in order to retain flexibility, but should think carefully about the
trade-offs before doing so".

> So, ROA with ANY prefix length is good enough to limit the blast radius of

Only if the origin AS of the hijack route is that of the hijacker. That
is not the case for all mistakes (Noction-leaks, for example).

Minimal ROAs are the best current initial line of defense to such
incidents.

> companies with complicated access rules to devices, it may still take time.
If your neighbors drop invalids, turning off egress ROV doesn't make any
difference: the update gets dropped on the other end of the wire.

If your neighbors don't drop invalids then this can be done in local
policy using SLURM, in close to real time.

In either case, this is a red-herring.

> possible only after ROA update.
I think this analysis is flawed.

If the customer of ASY (call them ASW) doesn't forge the origin, then
ASZ will filter the path. ROV works, and no de-aggreagation is
necessary.

If ASW does forge the origin, but the leaked route has length greater
than maxLength in the published ROA, then again ASZ will filter the
path, and no de-aggreagation is necessary.

De-aggreagation is only necessary to influence ASZ's bestpath selection
if the ROA is non-minimal to start with.

In general, with minimal ROAs, de-aggreagation is only a useful response
to influence bestpath selection in third-party networks that:
a) do not filter ROV-invalids;
b) are at least as topologically close to you as your attacker;
c) assign at least as high a local-pref to routes from you as your
attacker; and
d) you actually care about exchanging traffic with.

For many (most?) operators, a-d define a small and shrinking list of
ASs.

Because of a), the existence of a ROA is irrelevant, other than in the
corner case that you previously set out, and which is discussed in
section 5.2.

> selected AS support ROV. For Yandex this statement is not true.

I disagree that this is the correct test, for the reasons above.
The contents of a ROA are (by definition) not relevant when discussing
path selection in a network that doesn't drop ROV-invalids[*].

Cheers,

Ben

[*]: Some operators might (foolishly) de-pref invalids. They appear
simply not to perform ROV at all for the purposes of this discussion.

---

From: Alexander Azimov <[email-masked]>
Date: Fri, 10 Dec 2021 19:38:58 +0300

Hi Ben,

> Minimal ROAs are the best current initial line of defense to such
I agree with your point about Noction. At the very moment, we don't have
any other technique if forged disaggregated routes are leaked.
Though it is mixing up route leaks and hijack problems, and there might be
more suitable solutions for route leaks in the long term.

My point and my main concern grow from the fact that Noction-like incidents
don't present all the scope.
The incidents that I witnessed were results of redistribution of static
routes with Invalid origin AS. I hope we can agree that for this kind of
incident the length of ROA doesn't matter.

> In either case, this is a red-herring.
In this scenario, I was supposing that ROV is done on egress and is absent
at the ingress of the neighbor.
And, yes, it can be done using local policy, but it may take time if this
scenario is not thought out in advance.
I wonder if you want to cite it in your document. I don't see any
references to rfc8893 at the moment.

> if the ROA is non-minimal to start with.
I think you missed my point in this scenario.

ASW doesn't forge the ASPATH, but creates a more specific hijack. ASW is a
customer of ASY, so disaggregation from ASX to ASY doesn't help, and ASY
will advertise hijacked routes to its peers, providers, and, especially
important, to its customers. If ASX advertises disaggregated routes to ASZ
it may effectively get some of AS from the intersection of customer cones
of ASY and ASZ. Minimal ROA forbids this action. And IMO opportunity to
instantly disaggregate routes toward ASZ is much better than nothing.

And this is where the coverage of ROV makes the difference. Half-done ROV
validation among your peers makes you half-exposed to this scenario.

> For many (most?) operators, a-d define a small and shrinking list of

I can imagine that for access providers the list of 'important' targets may
be quite small these days.
But this view can be very different from the content provider sude. Last
time I made this math I had 1000AS to cover 80% of egress traffic and
another 1000 AS to cover 95%.
I hope you admit that makes 'some' networks care about the customer cones
of their peers. And as far as I know, these networks are primary targets
for Noction-like systems.

> path selection in a network that doesn't drop ROV-invalids[*].
I hope you reconsider. Let's try to find the golden mean between our
positions.
We are discussing two scenarios that have very different (conflicting?)
actions.

   - Leaked routes from Noction-like egress control systems, that look like
   more specific leaks. For this scenario, minimal ROA can limit the
   propagation of such anomaly, though the disaggregation option may be also
   limited.
   - Redistributed static routes that result in 'simple hijack' without
   forged path, in this scenario disaggregation is the best answer, though
   minimal ROA may again limit propagation of your counter route.

Which scenario is more likely to happen? To face Noction-like route leak
that would be filtered by surrounding ISPs with ROV or widespread simple
hijack (or leak) that can be addressed by disaggregation but will be
limited by ROV at your neighbors? I don't have the right answer.

Cheers,

-- 
Best regards,
Alexander Azimov

---

From: Job Snijders <[email-masked]>
Date: Fri, 10 Dec 2021 18:05:13 +0100

> at the ingress of the neighbor.

Sasha, I fail to see what you want to achieve with this discussion -
egress ROV is a novelty that almost nobody uses today. All of today's 
educational outreach efforts focus on ingress ROV.

draft-ietf-sidrops-rpkimaxlen can be summarized in just a few sentences:
"Configure your ROAs such that they align exactly with your BGP
announcements. Adversarial actors can and will use any 'holes' you leave
open: if you permit more-specifics but don't announce those
more-speciics, someone else might!"

If this group really cannot come to consensus on the draft at hand, I
suppose the next step simply is to propose to remove the ROA MaxLength
functionality entirely from the ecosystem. Eradicate the misfeature.

The 'MaxLength' feature seems to cause nothing but confusion, as a
result it all to easily leads to a reduction of security posture.

Kind regards,

Job

---

From: Ben Maddison <[email-masked]>
Date: Sat, 11 Dec 2021 15:37:26 +0200

Hi Alexander,

> incident the length of ROA doesn't matter.
Yes. We can certainly agree there.
However a substantial proportion of the wide scale incidents that I have
seen in recent months have been such events.
We really don't have a good defense to these other than the ROV
length-check.

> references to rfc8893 at the moment.
I'm hesitant to add any more length to an already lengthy document.
I kinda think this should be fairly obvious to an educated reader.
What do others think here?

> I think you missed my point in this scenario.
Yes, Sriram pointed out to me that you were considering competition for
path selection inside the intersection of Y and Z's customer cones. I
missed that previously. Apologies for making you repeat yourself.

Viewed from this perspective, this scenario is simply a variation on the
one that lead to us adding section 5.2.

In this case the defensive routes are hidden because of differences
between local-pref in Y, but the effect is the same as if there is no
adjacency between X and Y.

I stand by my view that, in the general case, solving for this creates
more risk than it removes. But I also appreciate that this general
guidance is "general", and isn't necessarily true for your employer, or
those with similar topologies...

> another 1000 AS to cover 95%.

Yes, the same distinction occurred to me.
This phenomenon only occurs when the networks that one cares about are
not directly adjacent to the hijack victim.
Most content networks with which I am familiar have very high external
interconnection density, and are usually only one AS away from the other
end of a TCP connection.
This would make the above much less of a problem, and perhaps explains
why other CDN ops people here seem less concerned about it?

I wonder if you have looked at how many of those ~2k networks you are
directly adjacent to?

> I hope you reconsider. Let's try to find the golden mean between our

I think that we can resolve this by explicitly noting that the "general"
recommendation of section 5.2 will not apply to everyone.

Something like (section 5.2):

    447,451c447,452
    <    victim and hijacker filters ROV invalid prefixes, it may be the case
    <    that the existence of a minimal ROA issued by the victim prevents the
    <    defensive more- specific prefixes being propagated to the networks
    <    topologically close to the attacker, thus hampering the effectiveness
    <    of this response.
    ---
    >    victim and hijacker filters ROV invalid prefixes (or where path
    >    selection policy within intermediate ASs prefers paths learned from
    >    the attack source) it may be the case that the existence of a minimal
    >    ROA issued by the victim prevents the defensive more-specific
    >    prefixes being propagated to the networks topologically close to the
    >    attacker, thus hampering the effectiveness of this response.
    453,455c454,456
    <    Nevertheless, this document recommends that where possible, network
    <    operators publish minimal ROAs even in the face of this risk.  This
    <    is because:
    ---
    >    Nevertheless, this document recommends that in general, network
    >    operators SHOULD publish minimal ROAs even in the face of this risk.
    >    This is because:
    468a470,480
    > 
    >    Some operators, however, especially those that are topologically
    >    distant from traffic sources that they consider "important", MAY
    >    legitimately prioritise the ability to defensively de-aggregate above
    >    the protection that minimal ROAs afford.  Operators making such a
    >    decision SHOULD carefully evaluate (insofar as they are able) the
    >    filtering and path selection behaviour of ASs in their vicinity to
    >    determine the extent to which the aforementioned risks apply to their
    >    situation.  Furthermore, operators SHOULD continue to re-evaluate
    >    this decision periodically in the face of continuing deployment of
    >    route origin validation by other ASs.

Let me know if this works for you?

Cheers,

Ben