Subject: [Idr] draft-haas-idr-bfd-subcode-00.txt

From: Jeffrey Haas <[email-masked]>
Date: Wed, 5 Jan 2022 17:26:40 -0500

[Speaking as an individual contributor]

Working Group,

While we are fighting through BGP NOTIFICATION codes in the other thread, I
wanted to bring to your attention a simple proposal: A cease subcode
specifically for notifying your peer that you're taking the session down
because BFD said so.

The motivation is hopefully spelled out clearly in the draft text.

As a BGP implementor that regularly has to bounce bug reports to a different
functional group because of mis-diagnosis, my hope is the Working Group will
find this worthy of adoption. (And quick WGLC.)

-- Jeff

----- Forwarded message from [email-masked] -----

Date: Wed, 05 Jan 2022 10:12:13 -0800
From: [email-masked]
To: [email-masked]
Subject: I-D Action: draft-haas-idr-bfd-subcode-00.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.

        Title           : BGP Cease Notification Subcode For BFD
        Author          : Jeffrey Haas
	Filename        : draft-haas-idr-bfd-subcode-00.txt
	Pages           : 5
	Date            : 2022-01-05

Abstract:
   The Bidirectional Forwarding Detection protocol (BFD) is used to
   detect loss of connectivity between two forwarding elements,
   typically with low latency.  BFD is leveraged by routing protocols,
   including the Border Gateway Protocol (BGP), to use that detection of
   loss of connectivity to bring down the protocol connections faster
   than the native protocol timers.

   This document defines a Subcode for the BGP Cease NOTIFICATION
   message for when a BGP connection is being closed due to a BFD
   session going down.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-haas-idr-bfd-subcode/

There is also an HTML version available at:
https://www.ietf.org/archive/id/draft-haas-idr-bfd-subcode-00.html

Internet-Drafts are also available by rsync at rsync.ietf.org::internet-drafts

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

----- End forwarded message -----

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 5 Jan 2022 22:47:26 +0000

> find this worthy of adoption. (And quick WGLC.)

It would probably work in practice, but does it work in theory?

Nick

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 5 Jan 2022 15:22:51 -0800

Jeff,

I support this work and its progress in IDR.

Few editorial comments:

BFD is a L3 protocol, and is employed between routers, not forwarding elements (that could operate at a lower layer)

“BGP implementation” -> BGP speaker

“BFD session goes Down” -> BFD session state changes to Down (BFD session could go down due to state changing to AdminDown which would not cause registered BGP speaker to change its respective state)

“Uses a Hold Timer to terminate session” -> BGP terminates session on Hold Timer expiry when a BGP speaker does not receive…

Cheers,

Jeff

From: Jeffrey Haas
Sent: Wednesday, January 5, 2022 2:26 PM
To: [email-masked]
Subject: [Idr] draft-haas-idr-bfd-subcode-00.txt

Working Group,

because BFD said so.

-- Jeff

----- Forwarded message from [email-masked] -----

Author          : Jeffrey Haas

Pages           : 5

Date            : 2022-01-05

Abstract:

session going down.

[email-masked]

----- End forwarded message -----

Idr mailing list

[email-masked]

https://www.ietf.org/mailman/listinfo/idr

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 6 Jan 2022 07:51:26 -0500

Thanks, Jeff, for your comments.  I've incorporated the bulk of them for the next version.

> On Jan 5, 2022, at 6:22 PM, Jeff Tantsura <[email-masked]> wrote:
> Few editorial comments:

This text was largely lifted from the introduction for RFC 5880.  I did introduce the typo of "elements" rather than "engines" and have corrected that.

The intent of that text has always been to acknowledge that BFD may not be tightly coupled to specific subsystems in someone's implementation.  BFD might be running on the line card, but still having its services for such sessions protecting a separate routing engine.

-- Jeff

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 6 Jan 2022 07:44:58 -0800

Hi Jeff,
I've put some notes below in-lined tagged GIM>>. Greatly appreciate you and
the authors' consideration and feedback.

Regards,
Greg

On Wed, Jan 5, 2022 at 3:23 PM Jeff Tantsura <[email-masked]>
wrote:

> BFD is a L3 protocol, and is employed between routers, not forwarding
> elements (that could operate at a lower layer)
>
GIM>> In RFC 5880 the BFD is characterized as:
   a protocol intended to detect faults in the
   bidirectional path between two forwarding engines, including
   interfaces, data link(s), and to the extent possible the forwarding
   engines themselves, with potentially very low latency.
Also, would point that RFC 5880 has defined the BFD Control message. So
far, at IETF we've standardized IP and MPLS encapsulation, and are
discussing BIER encapsulation. Hence, saying that BFD is the Layer 3
protocol might be not entirely accurate (I can even imagine a BFD Control
message encapsulated in Ethernet).

> “BFD session goes Down” -> BFD session state changes to Down (BFD session
> could go down due to state changing to AdminDown which would not cause
> registered BGP speaker to change its respective state)
>
> “Uses a Hold Timer to terminate session” -> BGP terminates session on Hold
> Timer expiry when a BGP speaker does not receive…
>
>
>
>
>
> Cheers,
>
> Jeff
>
>
>
> *From: *Jeffrey Haas <[email-masked]>
> *Sent: *Wednesday, January 5, 2022 2:26 PM
> *To: [email-masked]
> *Subject: *[Idr] draft-haas-idr-bfd-subcode-00.txt
>
>
>

---

From: Jeff Tantsura <[email-masked]>
Date: Thu, 6 Jan 2022 15:48:48 -0800

Hi Greg,

Please consider the context of the draft, which is IP reachability of a BGP peer.

Cheers,

Jeff

From: Greg Mirsky
Sent: Thursday, January 6, 2022 7:45 AM
To: Jeff Tantsura
Cc: Jeffrey Haas; [email-masked]
Subject: Re: [Idr] draft-haas-idr-bfd-subcode-00.txt

Hi Jeff,

I've put some notes below in-lined tagged GIM>>. Greatly appreciate you and the authors' consideration and feedback.

Regards,

Greg

On Wed, Jan 5, 2022 at 3:23 PM Jeff Tantsura <[email-masked]> wrote:

Jeff,

Few editorial comments:

Also, would point that RFC 5880 has defined the BFD Control message. So far, at IETF we've standardized IP and MPLS encapsulation, and are discussing BIER encapsulation. Hence, saying that BFD is the Layer 3 protocol might be not entirely accurate (I can even imagine a BFD Control message encapsulated in Ethernet).

Cheers,

Jeff

From: Jeffrey Haas

Working Group,

because BFD said so.

-- Jeff

----- Forwarded message from [email-masked] -----

Author          : Jeffrey Haas

Pages           : 5

Date            : 2022-01-05

Abstract:

session going down.

[email-masked]

----- End forwarded message -----

Idr mailing list

[email-masked]

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 6 Jan 2022 22:58:22 -0500

Hi Jeff

I support progressing this draft and the premise behind it to help
operators troubleshooting links where BGP is taken down by BFD in partial
failures.

Normally with BGP with  you get a BGP notification hold timer expired when
a link goes down or partial failure.

When BGP is registered with BFD with tuned down timers for fast convergence
upon link down you get a BFD down log message.  Upon link down connected
interface goes down fast external failover kicks in and takes the link down
immediately without BFD.

The most common use case for BFD is 1-way fiber partial failures for WAN
and LAN links where BFD goes down and pulls down  BGP but you don’t get a
BGP notification and only a BFD log message.

I agree with solution using the RFC 8538 BGP GR notification sub code for
the 1-way use case.

Kind Regards

Gyan

On Thu, Jan 6, 2022 at 6:49 PM Jeff Tantsura <[email-masked]>
wrote:

> *From: *Greg Mirsky <[email-masked]>
> *Sent: *Thursday, January 6, 2022 7:45 AM
> *To: *Jeff Tantsura <[email-masked]>
> *Cc: *Jeffrey Haas <[email-masked]>; [email-masked]
> *Subject: *Re: [Idr] draft-haas-idr-bfd-subcode-00.txt
>
>
>
> Hi Jeff,
>
> I've put some notes below in-lined tagged GIM>>. Greatly appreciate you
> and the authors' consideration and feedback.
>
>
>
> Regards,
>
> Greg
>
>
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: "Dale W. Carder" <[email-masked]>
Date: Fri, 7 Jan 2022 12:57:08 -0600

This looks pretty straightforward and a good addition.

In section 3 paragraph 3, that functionality would be really useful
operationally.

   When there is a total loss of connectivity between two BGP Speakers,
   it may not be possible for the NOTIFICATION message to have been
   sent.  Even so, BGP implementations may provide this reason as part
   of their operational state; e.g. bgpPeerLastError in the BGP MIB.
   [RFC4273].

If a speaker is using this codepoint, it should use it consistently.
Ideally, I think the notification to the peer and what it reports to 
the management systems should match?  That would help with the head 
scratching.  Should that paragraph be edited such as

   Even so, BGP implementations SHOULD provide this reason as part
   of their operational state; e.g. bgpPeerLastError in the BGP MIB

or separate this out into its own requirement, and more rigidly say 
that implementations are using this subcode MUST also reflect that in
bgpPeerLastError?

best,
Dale

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 7 Jan 2022 11:40:55 -0800

Gyan,

You won’t experience  holdtime expiry on link down event, also syslog entry would be clear about the reason BGP session went down.

All modern BGP implementations implemented (default to my memory) “interface tracking”, so when an interface over which the peer is resolved going down, BGP process is notified and respective BGP sessions (each peer IP is registered with in RIB) then brought down.

Cheers,
Jeff

> On Jan 6, 2022, at 19:58, Gyan Mishra <[email-masked]> wrote:
> 
> 
> 
> Hi Jeff
> 
> I support progressing this draft and the premise behind it to help operators troubleshooting links where BGP is taken down by BFD in partial failures.
> 
> Normally with BGP with  you get a BGP notification hold timer expired when a link goes down or partial failure.
> 
> When BGP is registered with BFD with tuned down timers for fast convergence upon link down you get a BFD down log message.  Upon link down connected interface goes down fast external failover kicks in and takes the link down immediately without BFD. 
> 
> The most common use case for BFD is 1-way fiber partial failures for WAN and LAN links where BFD goes down and pulls down  BGP but you don’t get a BGP notification and only a BFD log message.
> 
> I agree with solution using the RFC 8538 BGP GR notification sub code for the 1-way use case.
> 
> Kind Regards 
> 
> Gyan
> 
> 
>> On Thu, Jan 6, 2022 at 6:49 PM Jeff Tantsura <[email-masked]> wrote:
>> Hi Greg,
>> 
>>  
>> 
> Gyan Mishra
> Network Solutions Architect 
> Email [email-masked]
> M 301 502-1347
> 
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 7 Jan 2022 14:53:52 -0500

Gyan,

> On Jan 6, 2022, at 10:58 PM, Gyan Mishra <[email-masked]> wrote:

Unpacking the stack of dependencies for a failure is always fun. :-)

In the event where you have an event impacting the interface, many BGP implementations will happily process an interface-down event to quickly stop the session.  Within a similar timeframe, BFD may have noticed the interface has gone down.

The question often becomes what event does BGP notice first?

My preference for this document is not to handle every set of conditions that may cause BGP to stop a session, especially since many of them are implementation specific and perhaps not amenable to standardization.  The BFD one condition is at least clear... and all too common. 

As an implementation note, JUNOS treats a received interface-down event from the internal infrastructure as a "config change" and uses that for the CEASE subcode.  The infrastructure doesn't distinguish the down event from configuration vs. the system moving the state to down due to loss of link, e.g.

-- Jeff

> On Thu, Jan 6, 2022 at 6:49 PM Jeff Tantsura <[email-masked] <mailto:[email-masked]>> wrote:
> Hi Greg,
> 
>  
> 
> From: Greg Mirsky <mailto:[email-masked]>
> To: Jeff Tantsura <mailto:[email-masked]>
> Cc: Jeffrey Haas <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> On Wed, Jan 5, 2022 at 3:23 PM Jeff Tantsura <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Jeff,
> 
>  
> From: Jeffrey Haas <mailto:[email-masked]>
> To: [email-masked] <mailto:[email-masked]>
> ----- Forwarded message from [email-masked] <mailto:[email-masked]> -----
> 
>  
> From: [email-masked] <mailto:[email-masked]>
> To: [email-masked] <mailto:[email-masked]>
> https://datatracker.ietf.org/doc/draft-haas-idr-bfd-subcode/ <https://datatracker.ietf.org/doc/draft-haas-idr-bfd-subcode/>
>  
> https://www.ietf.org/archive/id/draft-haas-idr-bfd-subcode-00.html <https://www.ietf.org/archive/id/draft-haas-idr-bfd-subcode-00.html>
>  
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/i-d-announce <https://www.ietf.org/mailman/listinfo/i-d-announce>
> Internet-Draft directories: http://www.ietf.org/shadow.html <http://www.ietf.org/shadow.html>
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt <ftp://ftp.ietf.org/ietf/1shadow-sites.txt>
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>
>  
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>
>  
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>
> -- 
> Email [email-masked] <mailto:[email-masked]>
> M 301 502-1347
> 
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 7 Jan 2022 15:54:17 -0500

Dale,

> On Jan 7, 2022, at 1:57 PM, Dale W. Carder <[email-masked]> wrote:

I think that's a good change, and I've incorporated it.

I'm hesitant to make this a MUST because of a level of ambiguity about when such state is set in implementations.  One valid interpretation is that this data is only set for received notifications rather than sent for RFC 4273.  Elsewhere, direction is specified.

The BGP MIBv2 work that was never completed in IDR, but which served as the basis for a number of vendor enterprise MIBs, distinguishes between the sent and received errors - along with providing access to the DATA.

https://datatracker.ietf.org/doc/html/draft-ietf-idr-bgp4-mibv2-15 <https://datatracker.ietf.org/doc/html/draft-ietf-idr-bgp4-mibv2-15>

See Bgp4V2PeerErrorsEntry

For comparison, see https://www.juniper.net/documentation/software/topics/concept/mib-jnx-bgpmib2.txt <https://www.juniper.net/documentation/software/topics/concept/mib-jnx-bgpmib2.txt>, jnxBgpM2PeerLastErrorSent.

And similar for openconfig:
https://github.com/openconfig/public/blob/master/release/models/bgp/openconfig-bgp-neighbor.yang <https://github.com/openconfig/public/blob/master/release/models/bgp/openconfig-bgp-neighbor.yang> - last-notification-error-code/subcode.

-- Jeff

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 7 Jan 2022 20:18:19 -0500

Hi Jeff

Thanks for the correction.  When a link goes down I agree you don’t get the
hold down expiry or any BGP log messages, it’s when the link is Up and the
timer pops you get the hold time expired.  There are cases where BGP peer
session is via L2 switch and link maybe remain always up so you rely on
 hold time expiry, and that scenario is where BFD tuned timers come into
play.

For link down event cisco and juniper and most other vendors have a default
feature called “fast external failover” so upon link down event BGP detects
down event and fails over immediately.

Cisco

https://community.cisco.com/t5/networking-documents/bgp-fast-external-fallover-command-overview/ta-p/3142577

Juniper

https://www.juniper.net/documentation/en_US/junose15.1/topics/reference/command-summary/bgp-fast-external-fallover.html

Kind Regards

Gyan

On Fri, Jan 7, 2022 at 2:40 PM Jeff Tantsura <[email-masked]>
wrote:

> Gyan,
>
> You won’t experience  holdtime expiry on link down event, also syslog
> entry would be clear about the reason BGP session went down.
>
> All modern BGP implementations implemented (default to my memory)
> “interface tracking”, so when an interface over which the peer is resolved
> going down, BGP process is notified and respective BGP sessions (each peer
> IP is registered with in RIB) then brought down.
>
> Cheers,
> Jeff
>
> When BGP is registered with BFD with tuned down timers for fast
> convergence upon link down you get a BFD down log message.  Upon link down
> connected interface goes down fast external failover kicks in and takes the
> link down immediately without BFD.
>
>> “Uses a Hold Timer to terminate session” -> BGP terminates session on
>> Hold Timer expiry when a BGP speaker does not receive…
>>
>>
>>
>>
>>
>> Cheers,
>>
>> Jeff
>>
>>
>>

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 7 Jan 2022 21:22:47 -0500

Hi Jeffrey

On Fri, Jan 7, 2022 at 2:53 PM Jeffrey Haas <[email-masked]> wrote:

> In the event where you have an event impacting the interface, many BGP
> implementations will happily process an interface-down event to quickly
> stop the session.  Within a similar timeframe, BFD may have noticed the
> interface has gone down.
>

   Gyan> My understanding is in that link down event there is a feature
called fast-external-failover that I know Cisco and Juniper support and
maybe others that upon link down BGP is immediately brought down.

> The question often becomes what event does BGP notice first?

   Gyan> True good point.  My thoughts are with link down BGP notices the
link down and use the knob above to take down BGP immediately and does not
the BFD event.  I believe the knob can be disabled so you can use BFD to
bring down BGP.

>
> My preference for this document is not to handle every set of conditions
> that may cause BGP to stop a session, especially since many of them are
> implementation specific and perhaps not amenable to standardization.  The
> BFD one condition is at least clear... and all too common.
>

   Gyan> Agreed

>
> As an implementation note, JUNOS treats a received interface-down event
> from the internal infrastructure as a "config change" and uses that for the
> CEASE subcode.  The infrastructure doesn't distinguish the down event from
> configuration vs. the system moving the state to down due to loss of link,
> e.g.
>

Gyan> Is what you described the same as the feature I described above

*Gyan Mishra*

*M 301 502-1347*