Subject: [Idr] Link Local Next Hop Handling for BGP

From: Donatas Abraitis <[email-masked]>
Date: Sun, 7 Nov 2021 18:44:03 +0200

Hi, folks!

we want to continue with a long-standing draft
https://datatracker.ietf.org/doc/html/draft-white-linklocalnh to take some
further actions to move forward.

More and more issues come dealing with inter-operability handling IPv6
link-local next-hops. Especially it's tricky how to handle 32-bytes length
next-hops and every implementation handles this independently which makes
things even worse.

Some open-source daemons set NH length to 32-bytes, but the global address
(GA) is set to `::`. How this should be handled correctly? In such a case,
a DUT device can be in trouble while setting next-hop at receiving side (if
global is preferred).

If IPv6 GA does not exist, sending LL(LL) with 32-bytes is the correct way
or not again?

Controversially, is it OKay sending GA(::) (I didn't see and even expect
this to happen, but...).

Waiting for comments!

-- 
Donatas

---

From: Gert Doering <[email-masked]>
Date: Sun, 7 Nov 2021 22:18:27 +0100

Hi,

> next-hops and every implementation handles this independently which makes

Without having read the draft (yet) I can confirm that the current
way some BGP implementations do IPv6 nexthops is just needless brittlenss
- if I peer between GUAs on the same link, there is no good reason for
installing the resulting routes with a LLA NH, but many good reasons
to *not* do so.

Certain vendors point the the BGP4 MP RFCs and claim "it must be that
way" and refuse to fix it.

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                      Vorstand: Sebastian v. Bomhard, Michael Emmer
Joseph-Dollinger-Bogen 14        Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                 HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444         USt-IdNr.: DE813185279

---

From: Donatas Abraitis <[email-masked]>
Date: Mon, 8 Nov 2021 11:15:20 +0200

Gert,

>if I peer between GUAs on the same link, there is no good reason for

You are right unless using unnumbered stuff inside data center
environments.

-- 
Donatas

---

From: Gert Doering <[email-masked]>
Date: Mon, 8 Nov 2021 10:21:23 +0100

Hi,

> You are right unless using unnumbered stuff inside data center

If I use unnumbered, I am not using GUAs...

But maybe my statement should be qualified better:

 - for eBGP peerings between directly adjacent routers, using GUA
   addresses from a common network ("connected") to speak BGP between
   them, router should never install LLA NHs unless explicitly told
   so (like, because the other side only sends a LLA NH to redirect
   traffic to a 3rd router that is also on-link but has no GUA).

The long-standing behaviour of at least IOS is "you peer between two 
eBGP routers on an IXP, BGP session is between GUA addresses, but the 
FIB installed uses a LLA NH for that very peer" - which brings an extra
dependency on "is LLA ND working correctly?", and we've seen quite a 
bit of blackholing caused by "GUA ND worked, LLA ND did not, but BGP did 
not know, so happily claimed 'path valid' and packets were unhappy".

Gert Doering
        -- NetMaster

---

From: Donatas Abraitis <[email-masked]>
Date: Mon, 8 Nov 2021 11:28:30 +0200

But you also can peer directly with LLA, which is again not well defined in
https://datatracker.ietf.org/doc/html/rfc2545#section-3 where LLA comes as
GUA?

-- 
Donatas

---

From: Gert Doering <[email-masked]>
Date: Mon, 8 Nov 2021 10:48:30 +0100

Hi,

> https://datatracker.ietf.org/doc/html/rfc2545#section-3 where LLA comes as

That ("peer via LLA") is not the use case I am worried about.  We have
a RFC that describes this scenario and its drawbacks, and it seems fairly
well-defined.

I wouldn't *do* that, though.

Gert Doering
        -- NetMaster

---

From: Donatas Abraitis <[email-masked]>
Date: Mon, 8 Nov 2021 11:57:41 +0200

>We have a RFC that describes this scenario and its drawbacks, and it seems
fairly
well-defined.

Which RFC do you have in mind?

-- 
Donatas

---

From: Gert Doering <[email-masked]>
Date: Mon, 8 Nov 2021 11:00:34 +0100

Hi,

> Which RFC do you have in mind?

7404 Using Only Link-Local Addressing inside an IPv6 Network. M.
     Behringer, E. Vyncke. November 2014. (Format: TXT, HTML) (Status:
     INFORMATIONAL) (DOI: 10.17487/RFC7404)

Gert Doering
        -- NetMaster

---

From: Donatas Abraitis <[email-masked]>
Date: Mon, 8 Nov 2021 12:08:51 +0200

7404 is more like a _generic_ RFC, but with this draft we are scoping ONLY
how to handle MP NH + the length. 7404 does not talk about that at all, as
I see.

-- 
Donatas

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 8 Nov 2021 14:42:31 -0500

> how to handle MP NH + the length. 7404 does not talk about that at all, as

A lot of the underlying behaviors for link-local peering comes from a much
older internet-draft:

https://datatracker.ietf.org/doc/html/draft-kato-bgp-ipv6-link-local-00

(LLNH authors, you might want to consider adding this to your citations.)

Wearing both my hat as an implementor and as a chair, I'd love to see us
come to some conclusion over this set of issues.  We are, as the LLNH draft
notes, guaranteed to have some interesting backward compatibility issues.

A fundamental headache that's caused many of these issues is implementations
have often been unwilling to accept a link-local address when only a single
nexthop is present.  Instead, it wants it to be present in the RFC 2545
second nexthop position.  

Various vendors have worked around this by having the double nexthop form
with wildly different behaviors of what is placed in each of the fields.
Adding my own employer, Juniper, to the mix, the code will in many versions
simply encode the LLNH twice.

Offering my short opinion on the desired solution, I'd like to see
implementations encode a single LLNH.  After that, the rest of the code
should be to deal with implementation quirks, which we have a good start to
listing their variety in the draft.

As much as I'd prefer that the draft could be kept very short, the current
draft is in need of addressing how nexthops are used when LLNH is the only
option.  

Even RFC 2545 has interesting quirks if you're not careful.  For example, if
you receive a global and a link local nexthop from an EBGP speaker, and are
propagating it via IBGP to a peer on a different link, you need to strip the
LLNH.  While that might seem "obvious", several implementations strip the
LLNH simply because the peering session was internal; it doesn't handle a
pair of peers on the same link.

Single LLNH needs to always be reset as nexthop-self when passed to another
link.

Etc.

Mostly, the text covering nexthop behaviors in RFC 4271 needs to be
addressed along with clarifying RFC 2545 double-nexthop scenarios.

This will sadly make for a much more complicated draft.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 8 Nov 2021 21:54:49 +0100

Hi,

Let's also keep in mind existence of this draft:

https://datatracker.ietf.org/doc/html/draft-kumar-idr-link-local-nexthop-02

which while perhaps not most optimal solves the problem quite neatly and
well.

Thx,
R.

On Mon, Nov 8, 2021 at 8:42 PM Jeffrey Haas <[email-masked]> wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 8 Nov 2021 17:03:15 -0500

Thanks, Robert.  I had forgotten about that one.

I've added all 3 documents to a new section of the WG wiki so we can keep track of them:
https://trac.ietf.org/trac/idr/wiki/idr-draft-status <https://trac.ietf.org/trac/idr/wiki/idr-draft-status>

-- Jeff

> On Nov 8, 2021, at 3:54 PM, Robert Raszuk <[email-masked]> wrote:
> 
> Hi, 
> 
> https://datatracker.ietf.org/doc/html/draft-kumar-idr-link-local-nexthop-02 <https://datatracker.ietf.org/doc/html/draft-kumar-idr-link-local-nexthop-02>
> 
> which while perhaps not most optimal solves the problem quite neatly and well. 
> 
> Thx,
> R.
> 
> On Mon, Nov 8, 2021 at 8:42 PM Jeffrey Haas <[email-masked] <mailto:[email-masked]>> wrote:
> https://datatracker.ietf.org/doc/html/draft-kato-bgp-ipv6-link-local-00 <https://datatracker.ietf.org/doc/html/draft-kato-bgp-ipv6-link-local-00>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>

---

From: Donatas Abraitis <[email-masked]>
Date: Tue, 9 Nov 2021 11:09:28 +0200

Hi, All,

LLNH capability seems a bit too much here (in my opinion), since almost
all(?)
open-source BGP daemons already handle 16/32-bytes NH on their own on
sending,
and receiving sides. Can't say much about non open-source implementations.

That means if daemonA sends ::(LLA) as 32-bytes, the receiving side daemonB
takes only LLA
address as next-hop. This is what `draft-white-linklocalnh` defines
actually too.

LLNH capability defines: The value of the Length of Next Hop Network
Address field on a
MP_REACH_NLRI attribute shall be set to 16.
This is basically the same as `draft-white-linklocalnh` says:

*If an implementation intends to send a single link local forwarding
 address in the Next Hop field of the MP_REACH_NLRI, it MUST set the
 length of the Next Hop field to 16 and include only the IPv6 link   local
address in the Next Hop field.*

Jeffrey, you said, Juniper just sends LLA(LLA), FRR does the same. This is
again, not defined
behavior anywhere, right?

`draft-kato-bgp-ipv6-link-local` also does not describe LLA(LLA) behavior.

`draft-white-linklocalnh` tells us that we should ignore whether it's GUA
or LLA
if NH length is 16-bytes. Or if NH length is 32-bytes we may ignore the
second LLA or ::(LLA) (in this case :: as GUA).

On Tue, Nov 9, 2021 at 12:03 AM Jeffrey Haas <[email-masked]> wrote:

> https://trac.ietf.org/trac/idr/wiki/idr-draft-status
>
> -- Jeff
>

-- 
Donatas

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 9 Nov 2021 15:35:01 -0500

Donatas,

> and receiving sides. Can't say much about non open-source implementations.

Having worked for a commercial router vendor that has had to deal with
customer interop issues in this space for a number of years, I'd suggest
that "It's always messier than you think".  

> behavior anywhere, right?

Correct.  Although in fairness this is probably more due to overly general
code and not dealing with the link-local peering cleanly than by design.

> second LLA or ::(LLA) (in this case :: as GUA).

draft-white has the advantage that it's started to describe the various
broken scenarios.  Understanding those cohesively is important for trying to
figure out what our options are.

The advantage of using a capability is that it can let two conforming
implementations know that they aren't talking to buggy implementations with
unknown behaviors.  While it's not ideal that we have such a situation, it
cleanly deals with our incremental deployment situation: Implementations are
inconsistently broken, and can't readily change their default behavior.  

Many implementations will provide knobs to try to interoperate with someone
else's bugs.  But it'd be nicer if we simply worked on trying to be
interoperable by default.  A capability can let us do that.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 9 Nov 2021 22:03:01 +0100

> interoperable by default.  A capability can let us do that.

And just to highlight the importance of the above paragraph from Jeff ...

Imagine MSDC fabric and zoo of BGP implementations any node (example TOR)
needs to peer with. Both towards the core and towards the compute nodes.
Adjusting a cfg knob on both ends especially when you swap the other end
will be a huge mess at scale.

While I do not understand why we have 32 bytes nh to start with - or who
was so clever and came with such an idea - the capability to support 16
octet nh seems to be operationally the easiest option (IMHO).

Best,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 9 Nov 2021 16:13:22 -0500

Robert,

> octet nh seems to be operationally the easiest option (IMHO).

"I was not there at the time".  That said, I have multiple rumors that I've
carried with me when I discuss this with newer BGP developers.  I'm happy to
share the rumors and at some point would appreciate if someone can get our
elders to comment on it.  (And some IDR archive archaeology may be useful...)

The high level comment that always came up was that as we were figuring out
IPv6, the comment was always that "directly connected IPv6 nodes should
always talk to each other using link local addresses".  This seems to be the
over-arching motivation.

Another motivation is that IPv6 architecture provided for multiple global
addresses from different subnets to be configured on the same link.  Given
the churn over time about address selection in such circumstances, including
the link local address meant the global address for directly connected
networks was less important.

If you consider each case discussing what constitutes a "first party" or a
"third party" check in RFC 4271, considerations about mis-matched globally
assigned subnets become more interesting.  In long practice, we didn't have
this problem.

And a complete rumor that I've never had anyone substantiate is some early
implementations couldn't deal with nexthops that were global.  The
link-local nexthop in such a rumored implementation was how they addressed
that issue.

Rumors all aside, it's always been a messy feature and has haunted me
through at least three implementations now.

-- Jeff

---

From: Donatas Abraitis <[email-masked]>
Date: Tue, 9 Nov 2021 23:30:01 +0200

Agree with both about 16 versus 32 nh length, as you said _over-arching
motivation_ :)

Makes sense. So, if we have a capability, then in such a case we are able to
handle LLA(LLA), ::(LLA) encoding issues. Is that what you meant?
Otherwise, handle
as https://datatracker.ietf.org/doc/html/rfc2545#section-3 defines?

-- 
Donatas

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 9 Nov 2021 16:48:52 -0500

Donatas,

> handle LLA(LLA), ::(LLA) encoding issues. Is that what you meant?

What I mean is that we can make a choice in the canonical representation of
a standalone BGP Link-Local Nexthop.  

My preference is a single 16 bit value.

By announcing the capability and having the capability negotiated,
implementations don't need to set interop knobs to get a particular set of
fields populated a particular way or worry about how they are received.

Implementations that don't support this new proposed capability would
still require those knobs for whatever variety of inconsistent
implementations they want to deal with.  draft-white was a nice start to
cataloging the brokenness.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 9 Nov 2021 22:52:08 +0100

Hi,

> My preference is a single 16 bit value.

Well that would be the best choice. And this is as said earlier also what I
would prefer.

Alternatively we could try to negotiate both 16 & 32 and if both sides
support 16 use it, but if not fallback to 32 as still an automated option.

Rgs,
R.

---

From: Donatas Abraitis <[email-masked]>
Date: Wed, 10 Nov 2021 00:06:29 +0200

>My preference is a single 16 bit value.

Sounds good to me as well. Just not sure about the flow (sorry, not much
familiar with the processes yet) if we can update this draft by copying the
idea of
https://datatracker.ietf.org/doc/html/draft-kumar-idr-link-local-nexthop-02
or we should contact the authors and try to resurrect `
draft-kumar-idr-link-local-nexthop-02` or just start a new draft by
combining ideas?

On Tue, Nov 9, 2021 at 11:52 PM Robert Raszuk <[email-masked]> wrote:

> Well that would be the best choice. And this is as said earlier also what
> I would prefer.
>

-- 
Donatas

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 9 Nov 2021 23:08:48 +0100

I am sure Jeff can help with the formal guidelines, but for me it is always
in good taste to contact the original authors of the idea to check if they
are willing to continue the work (resurrecting the old or making a new
draft).

Best,
R.

---

From: Donatas Abraitis <[email-masked]>
Date: Wed, 10 Nov 2021 00:12:22 +0200

Thank you, will try to contact at first.

> On 2021-11-10, at 00:08, Robert Raszuk <[email-masked]> wrote:
> 
> 
> 
> I am sure Jeff can help with the formal guidelines, but for me it is always in good taste to contact the original authors of the idea to check if they are willing to continue the work (resurrecting the old or making a new draft).
> 
> Best,
> R.
> 
>> On Tue, Nov 9, 2021 at 11:06 PM Donatas Abraitis <[email-masked]> wrote:
>> Sounds good to me as well. Just not sure about the flow (sorry, not much familiar with the processes yet) if we can update this draft by copying the idea of https://datatracker.ietf.org/doc/html/draft-kumar-idr-link-local-nexthop-02
>> or we should contact the authors and try to resurrect `draft-kumar-idr-link-local-nexthop-02` or just start a new draft by combining ideas?
>> 
>>> Well that would be the best choice. And this is as said earlier also what I would prefer. 
>>> 
>>> Alternatively we could try to negotiate both 16 & 32 and if both sides support 16 use it, but if not fallback to 32 as still an automated option. 
>>> 
>>> Rgs,
>>> R.
>>> 
>>>

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 7 Nov 2021 23:54:25 -0500

Some history with IPv6 and why with all routing protocols the next hop is
set to LL is due to on-link nature of direct adjacency and  ND process
ICMPv6 ND NS/NA process where the ICMPv6 packets are sent with source
address is Link Local w/ Destination Solicit Node Multicast or target
address DAD process.

For directly adjacent neighbors on a P2P link the  Link local cache is
built first then the GUI. In my experience I have not run into any issues
where the LL cache was not built and the GUI was which would break
forwarding.

I would think that is a router bug as that is rare and should never happen.

>From an operational perspective it is confusing that the BGP next hop BGP
RIB  is GUI and  the next hop local adjacency uRIB is set to link local.

I have deployed  implementations where we have hardcoded the LL to be more
intuitive then the default EUI64  MAC based address default and that has
worked well setting the LL to the GUI address.  Caveat in hardcoding the LL
on each interface is you cannot write into the 50 bits all 0s in the fe80::
prefix so the IID portion that is hardcoded have to make sure its fe80::(64
bits) or it breaks ND process.  Making the next hop intuitive is really
nice as it makes it easier for operations to identify the neighbor device
in troubleshooting then looking at a EUI64 next hop.

As far as using LL for IGP on all p2p links in a DC that works very well &
does simplify deployment.  Cisco has a knob "ipv6-enable" which enables
processing of IPv6 packets without an IPv6 address configured.  Juniper has
a similar knob.  From an operations perspective this does yield issues with
hop by hop ping & trace as the LL is only locally reachable.

BGP peering works well using LL address and even with IPv4 eBGP peering
using RFC 3927 127.0.0.x addresses works well.

Use of LL for eBGP peering solves the inconsistency issue easily and works
well eliminates the dependency on both LL & GUI ND cache entry.  Also works
well using LL for IPv4 peering.  As for IGP on P2P links configuring an
IPv6 address is better from an operations standpoint but using all LL is
much easier to deploy as no address planning is needed.

Kind Regards

Gyan

On Tue, Nov 9, 2021 at 5:12 PM Donatas Abraitis <[email-masked]>
wrote:

> I am sure Jeff can help with the formal guidelines, but for me it is
> always in good taste to contact the original authors of the idea to check
> if they are willing to continue the work (resurrecting the old or making a
> new draft).
>
> Best,
> R.
>

-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 8 Nov 2021 00:57:01 -0500

When an interface is enabled DAD (Duplicate Address Detection) runs on the
interface and first transitions the LL address from tentative state to
Valid and then the ND cache entry is populated for the LL address.  Then
DAD runs again on all the GUIs configured on the interface and then the ND
cache is populated for the GUI.

So if DAD processing failed that is why the interface would get stuck in a
tentative state and thus missing ND cache entry.  As the LL is generally
default EUI64 is used  it is a very rare that LL ND cache did not populate
to run into the BGP issue described.

Once the GUA IPv6 address is configured on the interface, DAD runs.
However, as the GUA is configured there is always a possibility of dup
address misconfiguration, and DAD will catch it and put the interface in a
tentative state.  To get out of the tentative state some vendors have the
option to clear the ND state or can bounce interface or remove &
reconfigure the IPv6 address usually is the best option.

Gyan

On Sun, Nov 7, 2021 at 11:54 PM Gyan Mishra <[email-masked]> wrote:

> On Tue, Nov 9, 2021 at 5:12 PM Donatas Abraitis <
> [email-masked]> wrote:
>

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 10 Nov 2021 06:14:44 -0500

Gyan,

> I would think that is a router bug as that is rare and should never happen.

To add speculation, consider the case of a third-party nexthop.  In such
cases, having the link-local means you can avoid some of the churn where the
ND cache wasn't in sync for a nexthop you've learned for a host you may not
have exchanged any traffic with.

> in troubleshooting then looking at a EUI64 next hop.

Once upon a time, Linux used to encode its interface ID for ipv6 link local
in those "must be zero" bits in its data structures.  I've not read that
source in years so it may have changed. :-)

> hop by hop ping & trace as the LL is only locally reachable.

My memory is RFC 8335 was intended to address some of this.

-- Jeff

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 8 Nov 2021 10:01:09 -0500

Jeffrey

> have exchanged any traffic with.

Gyan>  Understood the motivation.  Would that be moreso router to router
peering or  DC NVO3 overlay extension BGP to host VM scenario or both.

>    Gyan> RFC 4291 states the 54 bits proceeding the fe80:: must be 0's.  I
have seen not having that 0 break ND and a nightmare to troubleshoot

2.5.6 <https://datatracker.ietf.org/doc/html/rfc4291#section-2.5.6>.
Link-Local IPv6 Unicast Addresses

   Link-Local addresses are for use on a single link.  Link-Local
   addresses have the following format:

   |   10     |
   |  bits    |         54 bits         |          64 bits           |
   +----------+-------------------------+----------------------------+
   |1111111010|           0             |       interface ID         |
   +----------+-------------------------+----------------------------+

   Link-Local addresses are designed to be used for addressing on a
   single link for purposes such as automatic address configuration,
   neighbor discovery, or when no routers are present.

   Routers must not forward any packets with Link-Local source or
   destination addresses to other links.

> Gyan> Excellent!

-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 10 Nov 2021 09:59:39 -0500

> On Nov 8, 2021, at 10:01 AM, Gyan Mishra <[email-masked]> wrote:
> 
>    Gyan> RFC 4291 states the 54 bits proceeding the fe80:: must be 0's.  I have seen not having that 0 break ND and a nightmare to troubleshoot

This was understood.

It's a common coding technique when you rely on minimal memory use to leverage well-understood requirements like this and simply steal bits.

If you KNOW that you must have zero fields, you can do things like insert a 32-bit interface index into that field.  

As long as you have consistent mutators and accessors in your API, you don't notice.  Just beware of raw memcpy. :-)

-- Jeff

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 8 Nov 2021 13:07:32 -0500

On Wed, Nov 10, 2021 at 9:59 AM Jeffrey Haas <[email-masked]> wrote:

>>    Gyan> RFC 4291 states the 54 bits proceeding the fe80:: must be 0's.
> I have seen not having that 0 break ND and a nightmare to troubleshoot
>
>
> This was understood.
>
> It's a common coding technique when you rely on minimal memory use to
> leverage well-understood requirements like this and simply steal bits.
>
> If you KNOW that you must have zero fields, you can do things like insert
> a 32-bit interface index into that field.
>
> As long as you have consistent mutators and accessors in your API, you
> don't notice.  Just beware of raw memcpy. :-)
>
> -- Jeff
>

Gyan> Agreed

-- 

*Gyan Mishra*

*M 301 502-1347*