Subject: [Idr] Open issues with Comments on draft-vandevelde-idr-remote-next-hop-08

From: "Susan Hares" <[email-masked]>
Date: Wed, 15 Oct 2014 20:32:20 -0400

Many of you have read of my concern regarding the proliferation of nexthop
setting in the L2VPN and L3VPN.  I personally am interested to find out if
there is a clean way to distribute tunnel-endpoints/remote next-hops.  This
arises from my reading of several next-hop related drafts, and specifically
draft-sajassi-l2vpn-evpn-inter-subnet-forwarding which stuff next-hops
information into a series of communities. While I credit the Ali and his
co-authors with a delightful hack, it brings up that IDR has not really
provided a good general mechanism for tunnels/remote next hops. 

As we have discussed the next-hop issue over the last 10 years, we have
found the more general solution for next-hops is hard.  We have ended up
adopting more and more point solutions.  Our current potential solutions
such as RFC 5512 and draft-vandevelde-idr-remote-next-hop-08. Are there
other solutions? 

These solutions must at least consider how this will interact with SIDR
origin validation and bgp-sec. Please let me know if you have interesting
work we should discuss.  If we run out of time at IETF 91, we will hold
these discussions in interim meetings after IETF 91. 

In this light, I offer 11 summary discussion points on the
draft-vandevelde-idr-remote-08 draft. I appreciate the hard work that the
authors of this draft have done in originating and refining the draft.  If
you wish to discuss one of these points, I recommend you repost the point
and begin the discussion. 

Sue Hares

------------

Open issues from my reading: 

1) RFC 5512 related issues

Comment: Section 3.1 indicates that SAFI includes tunnel, but the
remote-next-hop attribute allows this to be separated from the SAFI.

Question to WG:  Are there any RFC5512 implementations deployed? 

Question to Rosen: Does this section 3.1 in version -08 address your
comments on:  SAFI with different NLRI, multiple tunnel endpoints , RFC5512
avoidance of per NLRI encapsulation. 

2. IPv6 multi-homing use case and other use cases 

Concern: Bruno and Yakov have suggested IPv6 multi-homing use case is not
proved.  I agree with their assessment. 

Proposal: Pull Use Case 11.1, 11.2, 11.3, and 11.4 out of the draft to
another draft.  We will use this draft to determine the use cases that
remote can handle. Put one use case (11.2 or 11.3) that is fully fleshed out
with: a) why this attribute is needed, b) how this mechanism is better than
existing mechanisms (RFC5512 at least), c) why it is better than other
alternatives, and d) how extensible the alternative is. 

One use cases is the 5+ nexthops in various places. The following are
examples:

•                    draft-sajassi-l2vpn-evpn-inter-subnet-forwarding-05 –
puts it in a series of communities,

•                    RFC5512 put it in SAFI

•                    RFC6514 – provides it as part of MPLS for L3VPN

•             The use case draft should survey existing uses and then
propose a set of requirements. 

3. Discuss why it is necessary to have both a next-hop and remote next hop? 

        The following things need to be improved in this draft:

•                    Clarify that  many nlris to 1 remote nexthop attribute
in a update packet (Example: 30 NLRIs to one Remote Nexthop attribute).
Expanding section 3 might help. 

•                    Why both a next hop (mandatory) and a remote nexthop
(optional transitive)?  This could expand section 10. 

•                    What underlay routes are, and how they are used to do
recursive lookup validation. This is mentioned in the security section, but
it clearly needs to also be in the operational section.

•                    How next-hop and the remote nexthop operate in bestpath
selection. Keyur indicated it followed this logic:

o   policy validates the use of remote next hop 

o   If the remote next-hop valid to policy, the remote next hop is used in
preference to nexthop. 

o   If the remote next-hop is valid but does not have a valid recursive
nexthop, then the next hop is selected.

•                    Tie breaking needs to be defined within the above logic
for:

o   tie-breaking between two valid remote-nexthops for the cases:

§   two routes have the same nexthop hop and different valid
remote-nexthops,

§  two routes have different nexthops and the same valid remote nexthop. 

o   Check that the validation of this work solves the following situations
mentioned by Eric Rosen: 

§  Situation 1:  “Can we get into a situation where two routes have the same
next hop but different remote next hops?  If we reach the decision process
tie-breaker step that chooses the route with the closest next hop, we might
end up  choosing the route that does not have the closest remote next hop.
Since the remote next hop is where we want to send the  data, does this make
sense?

§  Situation 2: “Suppose a given route does not have a remote next hop, but
when we do recursive resolution of the next hop we encounter a route that
does have a remote next hop.  Are we supposed to use that remote next hop?

§  Situation 3:  When a remote next hop attribute is attached to a labeled
address,  do we  assume that the label has been assigned by the remote next
hop, or by the  next hop?  If a BGP speaker changes the next hop but not
the remote next  hop, does it have to assign a new label?

o   Changing of next hops and remote next-hops:

§  Is it even valid to change the next hop without changing or removing the
remote next hop attribute?

4. Discuss impact of leakage of this attribute:

Concern: Since the remote next hop attribute is a transitive attribute, it
can easily be leaked outside of the administrative domain to which it
applies.  This could result in all kinds of strange effects on the routing
upstream.  

Discuss: One might point out that in the RFC 5512 model it is less likely
that information leaks out, since the encapsulation-SAFI would have to be
explicitly enabled. Is this true or false? 

5. Discuss impact of the remote next hop attribute to different address
families

Proposal: A summary of this impact to address families should be included in
the use case drafts, and a specific consideration should be given for the 1
or 2 use cases in the document. 

What to discuss: Walk through all NLRI families passed through BGP and
discuss how next hop impacts this work. 

7.  AS linkage

Concern: The attribute as defined contains an AS number, but there are no
procedures specified that make use of the AS number.

Question for WG: does section 7.1 and 7.2 provide enough procedures? 

8.  Tunnel endpoint 

Concern: Address length implies AFI  Is it better to specific tis directly
in section 4.0  

•Discussion:  It is better and extensible to specify the TLV (afi, length,
value).  Currently, the address family of the tunnel endpoint needs to be
inferred from  its length; it might be better to explicitly encode the
address family.

9. Bestpath considerations:

Concern: Basically, this says "use the worst tunnel endpoint cost".
However, in a network that may contain a mix of speakers that understand
this attribute and ones that do not and may use the BGP (MP)Nexthop, route
selection may be inconsistent.  

Discussion:  Please indicate how you will prevent fluctuation in the path
selection. 

10. Multiple remote nexthops:

Concern: This was Jeff’s original concern when it wasn't apparent that this
feature wasn't being used for chaining and instead is simply an odd form of
"add-path for nexthops" by loose analogy.  Section 8 seems to imply that the
desire is that the RNH is effectively bound to the same device, but there's
no guarantee of that.  (Hence the man-in-the-middle concerns.)  While
there's a comment about bestpath calculation, there's no related commentary
about which tunnel should be used or whether there are ECMP considerations
if you can use more than one tunnel endpoint.

11. Origination/modification:

Concern: The sane behavior of this feature appears to be that the originator
of the attribute is the only party that is adding to it.  Intermediaries may
remove remote nexthops for various reasons, including error handling.  

Counter-point: Keyur indicated that intermediaries may add this feature if
they become the new originator of this attribute.  

What needs to be discussed:   A add/remove scenario should be examined with
four AS scenario (AS-1—AS2—AS3—AS4)

---

From: Randy Bush <[email-masked]>
Date: Fri, 17 Oct 2014 09:44:31 -0700

> origin validation and bgp-sec.

could you descirbe in what way they will interact?  at first blush,
draft-vandevelde is about data plane forwarding, and both origin
validation and path validation are about the control plane and admit
that the world has already complicated the data plane beyond saving;
see mpls for example. :)

though some folk are dabbling in constraining the data plane to follow
the control plane [0], it's still pretty researchy.

thanks.

randy

---

[0] - hot sdn 2014 poster
Josh Bailey, Dean Pemberton, Andy Linton, Cristel Pelsser, Randy
Bush. Enforcing RPKI-Based Routing Policy on the Data Plane at an
Internet Exchange.  http://archive.psg.com/140620.rpki-hot-sdn.pdf

---

From: "Susan Hares" <[email-masked]>
Date: Fri, 17 Oct 2014 15:45:31 -0400

Randy: 

Section 7.2 states: 
   
(1) BGP Remote-Next-Hops Tunnel Endpoint and AS number association
   using BGP Origin Validation

This is validation validates Tunnel-endpoint is own by AS.  As to how the
creation of these tunnels impacts plans on bgp-sec, you know better than I
do.  I am in the middle of my "due-diligence" reading.   I agree that once
tunnels are up and going - it is data plane.  More Qs? 

Sue 
-----Original Message-----
From: Randy Bush [mailto:[email-masked]] 
Sent: Friday, October 17, 2014 12:45 PM
To: Susan Hares
Cc: idr wg
Subject: Re: [Idr] Open issues with Comments on
draft-vandevelde-idr-remote-next-hop-08

> These solutions must at least consider how this will interact with 
> SIDR origin validation and bgp-sec.

could you descirbe in what way they will interact?  at first blush,
draft-vandevelde is about data plane forwarding, and both origin validation
and path validation are about the control plane and admit that the world has
already complicated the data plane beyond saving; see mpls for example. :)

though some folk are dabbling in constraining the data plane to follow the
control plane [0], it's still pretty researchy.

thanks.

randy

---

[0] - hot sdn 2014 poster
Josh Bailey, Dean Pemberton, Andy Linton, Cristel Pelsser, Randy Bush.
Enforcing RPKI-Based Routing Policy on the Data Plane at an Internet
Exchange.  http://archive.psg.com/140620.rpki-hot-sdn.pdf

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Oct 2014 21:53:20 +0200

Hi Sue,

Origin validation is a local (to the router) CPU cheap check operation
already supported by number of vendors.

This has not much in common with bgp-sec proposals of SIDR. The only
part which is used by origin validation is an extract of RPKI database
again already present on edge routers. Perhaps draft should just
clearly spell out to perform such check if already supported on a
given SAFI even without remote-next-hop feature.

Best,
R.

On Fri, Oct 17, 2014 at 9:45 PM, Susan Hares <[email-masked]> wrote:
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: Randy Bush <[email-masked]>
Date: Fri, 17 Oct 2014 13:02:21 -0700

hi sue

>    using BGP Origin Validation

isn't that just a check to see if the endpoint ip address has roa which
associated the address of the end point with the as?  this is a use of
rpki-based origin validation, not an attack on it.

and it has naught to do with bgpsec path validation.

randy

---

From: "Susan Hares" <[email-masked]>
Date: Fri, 17 Oct 2014 16:08:15 -0400

Randy - agreed on roa and bgp-path-validation IMHO.  Due-diligence Q - was
tunnel end-pt + roa sufficient.  IMHO-yes.  Sue 

-----Original Message-----
From: Randy Bush [mailto:[email-masked]] 
Sent: Friday, October 17, 2014 4:02 PM
To: Susan Hares
Cc: idr wg
Subject: Re: [Idr] Open issues with Comments on

hi sue

randy

---

From: "Susan Hares" <[email-masked]>
Date: Fri, 17 Oct 2014 16:11:12 -0400

Robert: Agreed on origin-validation.  IMHO - I do not see this impact path-validation.  Some have suggested if this tunnel-end-point + roa is not secure enough use bgp-sec.  I'm not sure how the 2 interact so I could use input if you or other see link.  Sue 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, October 17, 2014 3:53 PM
To: Susan Hares
Cc: Randy Bush; idr wg
Subject: Re: [Idr] Open issues with Comments on draft-vandevelde-idr-remote-next-hop-08

Hi Sue,

Origin validation is a local (to the router) CPU cheap check operation already supported by number of vendors.

This has not much in common with bgp-sec proposals of SIDR. The only part which is used by origin validation is an extract of RPKI database again already present on edge routers. Perhaps draft should just clearly spell out to perform such check if already supported on a given SAFI even without remote-next-hop feature.

Best,
R.

On Fri, Oct 17, 2014 at 9:45 PM, Susan Hares <[email-masked]> wrote:
> This is validation validates Tunnel-endpoint is own by AS.  As to how 
> the creation of these tunnels impacts plans on bgp-sec, you know better than I

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 17 Oct 2014 22:17:50 +0200

I do not see any link.

There are other good ways to protect bgp speaker from using received
remote next hop information. Say ACL may be a good start ...

Best,
r.

---

From: "Russ White" <[email-masked]>
Date: Fri, 17 Oct 2014 16:47:14 -0400

> tunnel end-pt + roa sufficient.  IMHO-yes.  Sue

The roa is the only part that matters, at least until someone comes up with
a realistic solution for path validation. What's being pushed forward is
undeployable; no point in trying to work around something that can't be used
in the real world.

:-)

Russ

---

From: Randy Bush <[email-masked]>
Date: Fri, 17 Oct 2014 14:31:51 -0700

> was tunnel end-pt + roa sufficient.

for what?  

the originating AS in origin validation is not cryptograaphically
signed.  so the end point could be lying.  you need path validation
to get crypto assurance.

but it sure avoids a lot of accidents we seem to have.

randy

---

From: "Susan Hares" <[email-masked]>
Date: Fri, 17 Oct 2014 17:54:19 -0400

Randy -  agree -tunnel end pt  + roa is sufficient to stop stupid errors.
IMHO this is important for this draft. Is path-validation solid enough to
specify on this draft?   Some say "not quite yet".  I'm in due-diligence on
topic. 

Sue 

-----Original Message-----
From: Randy Bush [mailto:[email-masked]] 
Sent: Friday, October 17, 2014 5:32 PM
To: Susan Hares
Cc: 'idr wg'
Subject: Re: [Idr] Open issues with Comments on

for what?  

the originating AS in origin validation is not cryptograaphically signed.
so the end point could be lying.  you need path validation to get crypto
assurance.

randy

---

From: Randy Bush <[email-masked]>
Date: Fri, 17 Oct 2014 15:31:40 -0700

> Is path-validation solid enough to specify on this draft?

patience

---

From: Russ White <[email-masked]>
Date: Fri, 17 Oct 2014 19:47:51 -0400

Since sidr is running in the wrong direction, it's going to take a lot of patience... 

Roa is fine for this particular draft.

:-)

Russ

---

From: Eric Rosen <[email-masked]>
Date: Thu, 23 Oct 2014 14:09:23 +0000

I think Sue has done a good job summarizing the problems with this draft; I
think it still needs a lot of work before it can be accepted.

The number one question is whether the approach can be made architecturally
sound.  The draft significantly alters the role played by the "next hop" in
BGP.  In doing so, in doing so leaves a lot of unanswered questions that
will lead to lots of future interoperability problems.  I understand that
there are already deployments of some of these mechanisms in particular
environments.  The question though is how the mechanisms will work outside
the environments that the authors happen to have in mind today.

The authors compare their approach to that taken by RFC5512, but I don't
think they focus on the essential differences.  To my mind, the essential
feature of the RFC5512 approach is that there is no change at all to the
next hop semantics.  A BGP Update associates an NLRI with a next hop, and
information can then be provided to tell an ingress node how to forward
traffic through a tunnel to that next hop.  If there are multiple tunnel
types that can be used to reach the next hop, information can be specified
for each of them.  Particular NLRIs can also be bound to particular next
hops.

It is true that RFC5512 proposes to use a new AFI/SAFI to distribute he
encapsulation information, but this is really a secondary issue.  One could
certainly put the encapsulation SAFI attribute on NLRIs of AFI/SAFI 1/1 or
1/2.  The real difference between "remote next hop" and RFC5512 is that
"remote next hop" does not require the tunnel endpoint to be the same as the
next hop.  The primary issue is whether the use of the "next hop" concept in
BGP is being changed in a well-defined way.

In the remote-next-hop draft, every NLRI can be associated with not one next
hop, but with a whole set of them.  This is the part I find problematic.
The BGP specifications have many rules that involve next hops, and we now
have to find them all, look at each one, and figure out how to change it so
that it works when there are multiple next hops.  (And it does not appear to
me that this exercise has been done.)

For instance, suppose an update associated NLRI N with Next Hop H and
with Remote Next Hops R1, R2, and R3.

We could decide that this update is equivalent to the following three
conventional BGP routes:

           <NLRI=N, NH=R1>
           <NLRI=N, NH=R2>
           <NLRI=N, NH=R3>

distributed using add-path.  Then the semantics would at least be
well-defined.  But this doesn't seem to be at all what the authors
intend.   In fact, I can't tell from the draft just how the authors
really do intend such an update to be interpreted.

The draft does say that when choosing the best path for N, if the "IGP
metric" tie breaker is used, the IGP metric should be largest one from among
the set of remote next hops.  Well, at least that's well-defined.  But is it
a sensible strategy?  Will it result in the proper route being chosen?

If some or all of the remote next hops are unreachable, but the
"ordinary" next hop is reachable, is the route discarded, or do we just
ignore the unreachable next hops?  What if the ordinary next hop is
unreachable, but some or all of the remote next hops are reachable?  Is
the route discarded altogether, even though the tunnels it binds to the NLRI
could actually be used?

What if a route is distributed to another AS, and the next hop changes, but
the remote next hops do not change?  Now if the BGP decision process reaches
the "IGP metric" tie breaker, only the IGP metric to the next hop itself can
be considered.  (The remote next hops are in a different AS, so the IGP
metric to them is not known.)  But the next hop might not even be on the
path to the remote next hop.

If there are multiple remote next hop endpoints, are they all equally
preferable?  Or is each one going to have its own LOCAL_PREF and its
own MED?

If the NLRI is a labeled route, and there are multiple remote next hops,
does it need to carry a label for each of the remote next hops and a label
for the ordinary next hop as well?  (Presumably all the remote next hops are
not using the same label space.)

Various specifications say that labels must be changed when the next
hop is changed.  What happens if the next hop is changed, but some or
all of the remote next hops are not changed?  Or is it supposed to be
illegal to use remote next hop on labeled routes?

If a route carries an AIGP attribute, does the AIGP value apply to some
or all of the remote next hops?  When the AIGP attribute is modified,
are you supposed to add in the IGP metric to the furthest of the remote
next hops?

I know that there are deployments of this feature in specific
environments where these questions don't necessarily come up.  That
doesn't mean that the feature is well-defined, or that it is the best
solution.  Section 6 is nowhere near satisfactory.

I appreciate that the problematic "multi-homed IPv6" use case has been
removed, but none of the other use cases is described in enough detail to be
evaluated.  And no attempt is made to characterize those cases where the
mechanism can be used without problems, and to distinguish those from cases
where the mechanism may give unintended results.

As currently specified, this just doesn't seem to me to be an
architecturally sound solution.

It's also worth noting that, by default, the remote next hops leak
across administrative domain boundaries, which doesn't seem like the
right thing.

A few other more minor points:

   "If a device does not support this attribute, and receives this
   attribute, then it follows the normal NLRI processing and BGP best
   path selection, and the resulting forwarding decision is used, as the
   attribute is optional."

Won't that lead to the packets going to the wrong place?

   "While [RFC5512] allows multiple tunnel endpoints and multiple tunnel
   types to be carried within a BGP Encaps SAFI, the correlation of
   Tunnel information with other SAFIs is done using the color extended
   community which is also non-trivial."

I suppose one could complain that adding a level of indirection is
"non-trivial".  But indirection is not exactly unheard of in computer
science, and could also be useful.  By using the color EC, the originator of
the route could say "always use a red tunnel for packets to this NRLI".  The
red tunnel would not need to be the same tunnel in each AS through which the
packet passes.  Without this level of indirection, the tunnel encapsulation
attribute might have to be "swapped" at AS boundaries.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 23 Oct 2014 16:27:39 +0200

Hi Eric,

Thank you a lot for the comments made here (and before).

But honestly speaking root of your questions lays in the model of flat
routing. You very correctly pointing out issues in that respect.

However I look at this proposal not within the flat routing domain. I look
at this as overlay reachability plane without necessity to construct
separate mapping plane (say LISP-ALT).

If your point is that we must never try to add overlays to SAFI 1/1 or 2/1
and if everyone agrees with this point then indeed the draft can be
shelved.

But I think you as well as number of other IDR/BESS/XYZ WG folks see value
in automation of creation more control in the IP reachability and if so
choices are:

* use out of band static configuration (ie call yr peer)

* use out of band alternative signalling layer (LISP or some sort of
controller)

* augment BGP to be able to carry effectively opaque info having in mind
that where it is used for packets is under operators control.

Frankly I am myself not convince there is one winner of all of the above.
At least I am not able to pick one. But that may come from lack of
operational experience especially in large scale with all of those.

So till we try how do we learn ?

Maybe the draft should be moved to experimental and progressed for few
years ? Maybe we could add much more strict use rules.

What do you think ?

Best,
Robert.