Subject: [Idr] Link Local Next Hop Capability for BGP

From: Donatas Abraitis <[email-masked]>
Date: Tue, 28 Dec 2021 11:12:14 +0200

Hi, folks!

More and more issues come dealing with inter-operability handling IPv6
link-local next-hops. Especially it's tricky how to handle 32-bytes length
next-hops and every implementation handles this independently which makes
things even worse.

Some open-source daemons set NH length to 32-bytes, but the global address
(GA) is set to `::`. How this should be handled correctly? In such a case,
a DUT device can be in trouble while setting next-hop at receiving side (if
global is preferred).

If IPv6 GA does not exist, sending LL(LL) with 32-bytes is the correct way
or not again?

Controversially, is it OKay sending GA(::) (I didn't see and even expect
this to happen, but...).

Proposing a new BGP capability:
https://datatracker.ietf.org/doc/html/draft-white-linklocal-capability

It covers some cases where 32-bytes length next-hops and/or 16-bytes
link-local ONLY next-hops are sent/received.

Waiting for comments!

-- 
Donatas

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 28 Dec 2021 19:27:57 -0500

Hi Donatas

By default most vendors generate the LL via EUI64 MAC address with FFFE bit
stuff.

If it were desirable by operators to configure hardcoded intuitive LL
address would there be any issue in the capabilities advertisement should
pick up the configured LL?

Also in BGP only DC fabrics where no IGP exists for recursion, however in
NVO3 multi tiered fabrics where the leaf, spine and BL/BS nodes sit in the
same AS with spine acting as RR, how would recursive routing work to the LL
next hop attribute as the LL is only locally reachable?

I think the standard next-hop-self rewrite to loopback routable would solve
the recursive issue.  However maybe cases where the external peering link
must have reachability for NMS telemetry or maybe other reasons.

Other that that the draft looks sound and I support progressing.

Kind Regards

Gyan

On Tue, Dec 28, 2021 at 4:12 AM Donatas Abraitis <[email-masked]>
wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Donatas Abraitis <[email-masked]>
Date: Wed, 29 Dec 2021 16:45:01 +0200

Not at all, it is not related to the form of the LLA.

Mostly this is just by design. LLAs are more designed for point-to-point
networks, like in L3 only (Clos topology). And yes, iBGP, RR should keep
using GUA with rewritten next-hops.

On Wed, Dec 29, 2021 at 2:28 AM Gyan Mishra <[email-masked]> wrote:

>
> Hi Donatas
>
> By default most vendors generate the LL via EUI64 MAC address with FFFE
> bit stuff.
>
> I think the standard next-hop-self rewrite to loopback routable would
> solve the recursive issue.  However maybe cases where the external peering
> link must have reachability for NMS telemetry or maybe other reasons.
>
> On Tue, Dec 28, 2021 at 4:12 AM Donatas Abraitis <
> [email-masked]> wrote:
>
>> Hi, folks!
>>

-- 
Donatas

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 29 Dec 2021 16:50:59 -0500

Route reflectors end up being an irritating related case similar to third party nexthops.

It's perfectly fine for a route reflector to reflector to reflect BOTH the global and the link-local address - as long as all parties are on the same interface.

-- Jeff

> On Dec 29, 2021, at 9:45 AM, Donatas Abraitis <[email-masked]> wrote:
> 
> >If it were desirable by operators to configure hardcoded intuitive LL address would there be any issue in the capabilities advertisement should pick up the configured LL?
> 
> >Also in BGP only DC fabrics where no IGP exists for recursion, however in NVO3 multi tiered fabrics where the leaf, spine and BL/BS nodes sit in the same AS with spine acting as RR, how would recursive routing work to the LL next hop attribute as the LL is only locally reachable?  
> 
> Mostly this is just by design. LLAs are more designed for point-to-point networks, like in L3 only (Clos topology). And yes, iBGP, RR should keep using GUA with rewritten next-hops.
> 
> 
> 
> On Wed, Dec 29, 2021 at 2:28 AM Gyan Mishra <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Hi Donatas
> 
> By default most vendors generate the LL via EUI64 MAC address with FFFE bit stuff.  
> 
> If it were desirable by operators to configure hardcoded intuitive LL address would there be any issue in the capabilities advertisement should pick up the configured LL?
> 
> Also in BGP only DC fabrics where no IGP exists for recursion, however in NVO3 multi tiered fabrics where the leaf, spine and BL/BS nodes sit in the same AS with spine acting as RR, how would recursive routing work to the LL next hop attribute as the LL is only locally reachable?  
> 
> I think the standard next-hop-self rewrite to loopback routable would solve the recursive issue.  However maybe cases where the external peering link must have reachability for NMS telemetry or maybe other reasons.
> 
> On Tue, Dec 28, 2021 at 4:12 AM Donatas Abraitis <[email-masked] <mailto:[email-masked]>> wrote:
> Hi, folks!
> 
> https://datatracker.ietf.org/doc/html/draft-white-linklocal-capability <https://datatracker.ietf.org/doc/html/draft-white-linklocal-capability>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>
> -- 
> Gyan Mishra
> Network Solutions Architect 
> Email [email-masked] <mailto:[email-masked]>
> M 301 502-1347
> 
> 
> 
> 
> -- 
> Donatas
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>

---

From: Gyan Mishra <[email-masked]>
Date: Wed, 29 Dec 2021 19:05:12 -0500

In an NVO CLOS the links are all P2P peers so would be loopback to loopback
peers typically and not connected iBGP peers and would rely on IGP for
recursion.

So for this capability feature all parties meaning connected iBGP peer and
no IGP recursion would be the RR iBGP scenario.

Why can’t this work for iBGP loopback to loopback peering with IGP
recursion as long as next hop self rewrite is done on the leafs and BL/BS
edge nodes.

Kind Regards

Gyan

On Wed, Dec 29, 2021 at 4:51 PM Jeffrey Haas <[email-masked]> wrote:

> It's perfectly fine for a route reflector to reflector to reflect BOTH the
> global and the link-local address - as long as all parties are on the same
> interface.
>
> -- Jeff
>
>

*Gyan Mishra*

*M 301 502-1347*

---

From: Donatas Abraitis <[email-masked]>
Date: Thu, 30 Dec 2021 10:20:06 +0200

Hello, Gyan,

by saying P2P, I meant BGP unnumbered setup in my mind, sorry :)

On Thu, Dec 30, 2021 at 2:05 AM Gyan Mishra <[email-masked]> wrote:

>
> In an NVO CLOS the links are all P2P peers so would be loopback to
> loopback peers typically and not connected iBGP peers and would rely on IGP
> for recursion.
>
>> It's perfectly fine for a route reflector to reflector to reflect BOTH
>> the global and the link-local address - as long as all parties are on the
>> same interface.
>>
>> -- Jeff
>>
>>
>> >Also in BGP only DC fabrics where no IGP exists for recursion, however
>> in NVO3 multi tiered fabrics where the leaf, spine and BL/BS nodes sit in
>> the same AS with spine acting as RR, how would recursive routing work to
>> the LL next hop attribute as the LL is only locally reachable?
>>
>>> Also in BGP only DC fabrics where no IGP exists for recursion, however
>>> in NVO3 multi tiered fabrics where the leaf, spine and BL/BS nodes sit in
>>> the same AS with spine acting as RR, how would recursive routing work to
>>> the LL next hop attribute as the LL is only locally reachable?
>>>

-- 
Donatas

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 30 Dec 2021 12:50:55 -0500

Gyan,

I'm not following the NVO work in particular, and my observations here are BGP fundamentals that each implementation has to deal with.

A route reflector CANNOT safely propagate a link-local route it receives from peer A to peer B unchanged without A and B being on the same link; e.g. a shared Ethernet.

A route reflector's only choice if it propagated the route in a situation where A and B are not on the same link is to do a nexthop self and insert itself into the forwarding.  Alternatively, it may simply not propagate the route.

The same thing is true if A, B, and C are doing eBGP.  If C receives a route from A with a link-local nexthop and intends to propagate it to B, and A and B are on the same link, A may send the global nexthop (if any) and link-local nexthop to B unchanged as a third-party nexthop.  If C resets the nexthop when advertising to B, it must reset both the global and the link-local portions.

RFC 2545's wording about the behaviors of nexthop propagation are ambiguous enough to have lead to a significant number of implementation faults in multiple implementations over the years.  Enough so that my personal advice to implementors is to have a knob that disables sending the link-local nexthop.

The scenarios where link-local is the ONLY nexthop simply mean that these scenarios all need to be cataloged and fully addressed if we're going to do any specification work.

And please note: BGP confederations add to the discussion.

-- Jeff

> On Dec 29, 2021, at 7:05 PM, Gyan Mishra <[email-masked]> wrote:
> 
> 
> In an NVO CLOS the links are all P2P peers so would be loopback to loopback peers typically and not connected iBGP peers and would rely on IGP for recursion.
> 
> So for this capability feature all parties meaning connected iBGP peer and no IGP recursion would be the RR iBGP scenario.
> 
> Why can’t this work for iBGP loopback to loopback peering with IGP recursion as long as next hop self rewrite is done on the leafs and BL/BS edge nodes.
> 
> Kind Regards 
> 
> Gyan
> 
> On Wed, Dec 29, 2021 at 4:51 PM Jeffrey Haas <[email-masked] <mailto:[email-masked]>> wrote:
>> On Dec 29, 2021, at 9:45 AM, Donatas Abraitis <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 30 Dec 2021 13:39:15 -0500

Jeff

Responses in-line

Happy Holiday!

On Thu, Dec 30, 2021 at 12:50 PM Jeffrey Haas <[email-masked]> wrote:

> Gyan,
>
> I'm not following the NVO work in particular, and my observations here are
> BGP fundamentals that each implementation has to deal with.
>

  Gyan> Most operators are migrating or have migrated to a L3 POD type
architecture within the DC pushing L3 out to the edge CLOS folded spine
topology.  Most operators use NVO3 for the critical tenant isolation
functionality that you get with L3 VPN extending that same model to the
DC.  However there are operators cases where L3 BGP no NVO DC designs using
CLOS topology w/o NVO.  There are also for both NVO and  BGP no NVO, BGP
only designs for more stability RFC 7938.  So a lot of variety to account
for.

>
> A route reflector CANNOT safely propagate a link-local route it receives
> from peer A to peer B unchanged without A and B being on the same link;
> e.g. a shared Ethernet.
>

   Gyan> Agreed

>
> A route reflector's only choice if it propagated the route in a situation
> where A and B are not on the same link is to do a nexthop self and insert
> itself into the forwarding.  Alternatively, it may simply not propagate the
> route.
>

Gyan> Agreed

>
> The same thing is true if A, B, and C are doing eBGP.  If C receives a
> route from A with a link-local nexthop and intends to propagate it to B,
> and A and B are on the same link, A may send the global nexthop (if any)
> and link-local nexthop to B unchanged as a third-party nexthop.  If C
> resets the nexthop when advertising to B, it must reset both the global and
> the link-local portions.
>

Gyan> Agreed.  This is critical and should be included in the specification

>
> RFC 2545's wording about the behaviors of nexthop propagation are
> ambiguous enough to have lead to a significant number of implementation
> faults in multiple implementations over the years.  Enough so that my
> personal advice to implementors is to have a knob that disables sending the
> link-local nexthop.
>

Gyan> Agreed.  We definitely need an update to RFC 2545 to fix the gaps

>
> The scenarios where link-local is the ONLY nexthop simply mean that these
> scenarios all need to be cataloged and fully addressed if we're going to do
> any specification work.
>

Gyan> Gyan

> And please note: BGP confederations add to the discussion.

Gyan> Agreed.  Every permutation needs to be accounted for. I would be
happy to collaborate and co-author the draft.

*Gyan Mishra*

*M 301 502-1347*

---

From: Donatas Abraitis <[email-masked]>
Date: Thu, 30 Dec 2021 21:44:05 +0200

Gyan,

we have a repository in Github, if you want to contribute as well, ping me
personally, I'll add permission to you to contribute.

On Thu, Dec 30, 2021 at 8:39 PM Gyan Mishra <[email-masked]> wrote:

>> I'm not following the NVO work in particular, and my observations here
>> are BGP fundamentals that each implementation has to deal with.
>>
>
>> So for this capability feature all parties meaning connected iBGP peer
>> and no IGP recursion would be the RR iBGP scenario.
>>

-- 
Donatas

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 30 Dec 2021 22:50:29 -0500

Thank you

Gyan

On Thu, Dec 30, 2021 at 2:44 PM Donatas Abraitis <[email-masked]>
wrote:

>>> A route reflector's only choice if it propagated the route in a
>>> situation where A and B are not on the same link is to do a nexthop self
>>> and insert itself into the forwarding.  Alternatively, it may simply not
>>> propagate the route.
>>>
>>
>> Gyan> Agreed
>>
>>>
>>> The scenarios where link-local is the ONLY nexthop simply mean that
>>> these scenarios all need to be cataloged and fully addressed if we're going
>>> to do any specification work.
>>>
>>
>> Gyan> Gyan
>>
>>>
>>>> Mostly this is just by design. LLAs are more designed for
>>>> point-to-point networks, like in L3 only (Clos topology). And yes, iBGP, RR
>>>> should keep using GUA with rewritten next-hops.
>>>>
>>>>
>>>>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 31 Dec 2021 10:54:28 -0800

Gyan,

If no IGP is used, in most cases, underlay follows RFC7938(single-hop eBGP over connected link with next-hop reset at every hop), if an overlay (really eVPN) is used, usually it is a multi-hop eBGP (eBGP over eBGP) between loopbacks (if daisy-chained) with next-hop unchanged (next-hop=VTEP) recursively resolved over connected; if route-server alike setup is used (at large scale makes lots of sense) then all eBGP speakers in a PoD are meshed with RSs and RSs are meshed between them. Note that next-hop is still unchanged (next-hop=VTEP); the LL next-hop capability would only apply to underlay AFI/SAFI.

Happy New Year!

Cheers,
Jeff

> On Dec 30, 2021, at 10:39, Gyan Mishra <[email-masked]> wrote:
> 
> 
> Jeff
> 
> Responses in-line
> 
> Happy Holiday!
> 
>   Gyan> Most operators are migrating or have migrated to a L3 POD type architecture within the DC pushing L3 out to the edge CLOS folded spine topology.  Most operators use NVO3 for the critical tenant isolation functionality that you get with L3 VPN extending that same model to the DC.  However there are operators cases where L3 BGP no NVO DC designs using CLOS topology w/o NVO.  There are also for both NVO and  BGP no NVO, BGP only designs for more stability RFC 7938.  So a lot of variety to account for.
>> 
> Gyan> Agreed.  Every permutation needs to be accounted for. I would be happy to collaborate and co-author the draft.
>> 
>> -- Jeff
>> 
>> 
>>>>>>> On Tue, Dec 28, 2021 at 4:12 AM Donatas Abraitis <[email-masked]> wrote:
>>>>>>> Hi, folks!
>>>>>>> 
>>>>>> Email [email-masked]
>>>>>> M 301 502-1347
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Donatas
>>> Email [email-masked]
>>> M 301 502-1347
>>> 
>>> 
>> 
> -- 
> 
> 
> Gyan Mishra
> Email [email-masked]
> M 301 502-1347
> 
>

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 31 Dec 2021 18:07:11 -0500

Jeff

Agreed!

Thanks for chiming in!

That is the NVO scenario with RFC 7938 BGP only underlay we have deployed
IPv4 underlay in the past as you described for large scale DC stability
within a POD.

Agreed that LL next hop capability would only apply to underlay.

Happy New Year!!

Gyan

On Fri, Dec 31, 2021 at 1:54 PM Jeff Tantsura <[email-masked]>
wrote:

> Gyan,
>
> If no IGP is used, in most cases, underlay follows RFC7938(single-hop eBGP
> over connected link with next-hop reset at every hop), if an overlay
> (really eVPN) is used, usually it is a multi-hop eBGP (eBGP over eBGP)
> between loopbacks (if daisy-chained) with next-hop unchanged
> (next-hop=VTEP) recursively resolved over connected; if route-server alike
> setup is used (at large scale makes lots of sense) then all eBGP speakers
> in a PoD are meshed with RSs and RSs are meshed between them. Note that
> next-hop is still unchanged (next-hop=VTEP); the LL next-hop capability
> would only apply to underlay AFI/SAFI.
>
> Happy New Year!
>
> Cheers,
> Jeff
>

*Gyan Mishra*

*M 301 502-1347*