Subject: Re: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Mon, 31 Jan 2022 14:49:43 -0000

Hi Les,

Your scenario below is indeed something we have encountered in our production network in the non-strict scenario, due to "flapping" links, where routing protocol could come up before BFD due to "break-in-middle" link issue (interface stayed up, so routing protocol remained active). Strict mode will address this issue. 

Another point to add is that we do have as a standard on our interfaces to safeguard against flapping link by configuring interface hold-time/carrier-delay. However, this is only useful in situations where the link physically goes down (and fast detection is automatic in most implementation). 

Nowadays, it is also common to see the "break-in-middle" failures. we use BFD to detect this sort of failure within sub-second. And to dampen this sort of break-in-middle failures, we will need to use BFD holdtime/dampening. 

Thanks

Albert

From: [email-masked] At: 01/30/22 14:38:37 UTC-5:00To:  [email-masked],  [email-masked]
Cc:  Albert Fu (BLOOMBERG/ 120 PARK ) ,  [email-masked],  [email-masked],  [email-masked]
Subject: RE: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04

Robert – 
  
Here is what you said (emphasis added): 
  
<snip> 
But the timer I am suggesting is not related to BFD operation, but to OSPF (and/or ISIS). It is not about BFD sessions being UP or DOWN. It is about allowing BFD for more testing (with various parameters (for example increasing test packet size in some discrete steps) before OSPF is happy to bring the adj. up. 
<end snip> 
  
Point #1: If you want BFD to do more testing (such as MTU testing) then clearly you need extensions to BFD (such as https://datatracker.ietf.org/doc/draft-ietf-bfd-large-packets/ ) 
  
Point #2: The existing timers (as Ketan points out are mentioned in Section 5) are applied today at the OSPF level precisely because OSPF does not currently have strict-mode operation. So in a flapping scenario you could see the following  behavior: 
  
a)BFD goes down 
b)OSPF goes down in response to BFD 
c)OSPF comes back up  
d)Link is still unstable – so traffic is being dropped some of the time – but perhaps OSPF adjacency stays up (i.e., OSPF hellos get through often enough to keep the OSPF adjacency up) 
  
So some implementations have chosen to insert a delay following “b”. This doesn’t guarantee stability, but hopefully makes it less likely. And because OSPF today does NOT wait for BFD to come up, the delay has to be implemented at the OSPF  level. 
  
Once you have strict mode support, the sequence becomes: 
  
a)BFD goes down 
b)OSPF goes down in response to BFD 
c)BFD comes back up 
d)OSPF comes back up 
  
Now, if the concern is that BFD comes back up while the link is still unstable, the way to address that is to put a delay either before BFD attempts to bring up a new session or a delay after achieving UP state before it signals UP to its  clients – such as OSPF. This is a better solution because all BFD clients benefit from this. Ad if the link is still unstable, it is more likely that the BFD session will go down during the delay period than it would be for OSPF because the BFD timers are  significantly more aggressive. 
(BTW, this behavior can be done w/o a BFD protocol extension – it is purely an implementation choice.) 
  
From a design perspective, dampening is always best done at the lowest layer possible. In most cases, interface layer dampening is best. If that is not reliable for some reason, then move one layer up – not two layers up. 
  
   Les 

From: Robert Raszuk <[email-masked]> 
Sent: Sunday, January 30, 2022 10:05 AM
To: Ketan Talaulikar <[email-masked]>
Cc: Les Ginsberg (ginsberg) <[email-masked]>; Acee Lindem (acee) <[email-masked]>; [email-masked]; Albert Fu <[email-masked]>; lsr <[email-masked]>
Subject: Re: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04 

Hi Ketan, 

I would like to point out that the draft discusses the BFD "dampening" or "hold-down" mechanism in Sec 5. We are aware of BFD implementations that include such mechanisms in a protocol-agnostic manner. 

BFD dampening or hold-time are completely orthogonal to my point. Both have nothing to do with it.  

Those timers only fire when BFD goes down. In my example BFD does not go down. But we want to bring up the client adj. only after X ms/sec/min etc ...of normal BFD operation if no failure is detected during that timer. 

This draft indicates that OSPF adjacency will "advance" in the neighbor FSM only after BFD reports UP.  

And that is exactly too soon. In fact if you do that today without waiting some time (if you retire the current OSPF timer) you will not help at all in the case you are trying to address.  

Reason being that perhaps 200 ms after BFD UP it will go down, but OSPF adj. will get already established. It is really pretty simple.  

Thx, 

Robert. 

PS. And yes I think ISIS should also get fixed in that respect.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 31 Jan 2022 15:59:15 +0100

Les & Ketan

> Nowadays, it is also common to see the "break-in-middle" failures. we use
> BFD to detect this sort of failure within sub-second. And to dampen this
> sort of break-in-middle failures, we will need to use BFD
> holdtime/dampening.
>

Another data point to the above and this discussion which Albert is
co-author of.

Ref:
https://datatracker.ietf.org/doc/html/draft-ietf-idr-bgp-bfd-strict-mode

Please see the below paragraph which clearly says *BGP BFD Hold time*:

   If the BFD session does not transition to the Up state, and the
   HoldTimer has been negotiated to a non-zero value, the BGP FSM will
   close the session appropriately.  If the HoldTimer has been
   negotiated to a zero value, the session should be closed after a time
   of X.  This time X is referred as "BGP BFD Hold time".  The proposed
   default BGP BFD Hold time value is 30 seconds.  The BGP BFD Hold time
   value is configurable.

To me it is clear that BGP BFD Hold time is on the client side and here
affects BGP FSM.

Thx,
Robert.

From: [email-masked] At: 01/30/22 14:38:37 UTC-5:00
> To: [email-masked], [email-masked]
> Cc: Albert Fu (BLOOMBERG/ 120 PARK ) <[email-masked]>, [email-masked],
> [email-masked], [email-masked]
> But the timer I am suggesting is not related to BFD operation, but to OSPF
> (and/or ISIS). It is not about BFD sessions being UP or DOWN. It is about *allowing
> BFD for more testing (with various parameters (for example increasing test
> packet size in some discrete steps)* before OSPF is happy to bring the
> adj. up.
>
> <end snip>
>
>
>
> Point #1: If you want BFD to do more testing (such as MTU testing) then
> clearly you need extensions to BFD (such as
> https://datatracker.ietf.org/doc/draft-ietf-bfd-large-packets/ )
>
>
>
> Point #2: The existing timers (as Ketan points out are mentioned in
> Section 5) are applied today at the OSPF level precisely because OSPF does
> not currently have strict-mode operation. So in a flapping scenario you
> could see the following behavior:
>
>
>
> a)BFD goes down
>
> d)Link is still unstable – so traffic is being dropped some of the time –
> but perhaps OSPF adjacency stays up (i.e., OSPF hellos get through often
> enough to keep the OSPF adjacency up)
>
>
>
> So some implementations have chosen to insert a delay following “b”. This
> doesn’t guarantee stability, but hopefully makes it less likely. And
> because OSPF today does NOT wait for BFD to come up, the delay has to be
> implemented at the OSPF level.
>
>
>
> Now, if the concern is that BFD comes back up while the link is still
> unstable, the way to address that is to put a delay either before BFD
> attempts to bring up a new session or a delay after achieving UP state
> before it signals UP to its clients – such as OSPF. This is a better
> solution because all BFD clients benefit from this. Ad if the link is still
> unstable, it is more likely that the BFD session will go down during the
> delay period than it would be for OSPF because the BFD timers are
> significantly more aggressive.
>
> From a design perspective, dampening is always best done at the lowest
> layer possible. In most cases, interface layer dampening is best. If that
> is not reliable for some reason, then move one layer up – not two layers up.
>
>
>
>    Les
>
>
>
>
>
> *From:* Robert Raszuk <[email-masked]>
> *Sent:* Sunday, January 30, 2022 10:05 AM
> *To:* Ketan Talaulikar <[email-masked]>
> *Cc:* Les Ginsberg (ginsberg) <[email-masked]>; Acee Lindem (acee) <
> [email-masked]>; [email-masked]; Albert Fu <
> [email-masked]>; lsr <[email-masked]>
> *Subject:* Re: [Lsr] Working Group Last Call for "OSPF Strict-Mode for
> BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04
>
>
>
> Hi Ketan,
>
>
>
> I would like to point out that the draft discusses the BFD "dampening" or
> "hold-down" mechanism in Sec 5. We are aware of BFD implementations that
> include such mechanisms in a protocol-agnostic manner.
>
>
>
> Those timers only fire when BFD goes down. In my example BFD does not go
> down. But we want to bring up the client adj. only after X ms/sec/min etc
> ...of normal BFD operation if no failure is detected during that timer.
>
>
>
> And that is exactly too soon. In fact if you do that today without waiting
> some time (if you retire the current OSPF timer) you will not help at all
> in the case you are trying to address.
>
>
>
> Reason being that perhaps 200 ms after BFD UP it will go down, but OSPF
> adj. will get already established. It is really pretty simple.
>
>
>
> Thx,
>
> Robert.
>
>
>

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Mon, 31 Jan 2022 17:30:47 +0000

Albert –

We are in full agreement.

Delays in bringing BFD backup after a previous failure may well be warranted in the break-in-middle scenarios.
I am not convinced this needs to be standardized – seems quite appropriate as an implementation choice. But if any discussion were to occur in RFCs, I think it should be in some BFD document.

As this draft is focused on OSPF protocol extensions, I don’t think BFD dampening needs to be discussed. In any case it should not alter the interaction between BFD and protocols. If it takes longer for BFD to come up that just means the OSPF adjacency will not come up either – which is exactly the behavior that is desired.

    Les

From: Albert Fu (BLOOMBERG/ 120 PARK) <[email-masked]>
Sent: Monday, January 31, 2022 6:50 AM
To: Les Ginsberg (ginsberg) <[email-masked]>; [email-masked]; [email-masked]
Cc: Acee Lindem (acee) <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04

Hi Les,

Thanks

Albert

From: [email-masked]<mailto:[email-masked]> At: 01/30/22 14:38:37 UTC-5:00
To: [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>
Cc: Albert Fu (BLOOMBERG/ 120 PARK ) <mailto:[email-masked]> , [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>
Subject: RE: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04

Robert –

<snip>

a)BFD goes down

a)BFD goes down

   Les

From: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Sent: Sunday, January 30, 2022 10:05 AM
To: Ketan Talaulikar <[email-masked]<mailto:[email-masked]>>
Cc: Les Ginsberg (ginsberg) <[email-masked]<mailto:[email-masked]>>; Acee Lindem (acee) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Albert Fu <[email-masked]<mailto:[email-masked]>>; lsr <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04

Hi Ketan,

Thx,
Robert.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 31 Jan 2022 18:39:23 +0100

Les,

BFD does not have a notion to come up in a hidden way, start testing
the link and only after some defined per client protocol or per interface
peer period signal to the client its "UP" state.

BFD holdtime as defined in RFC5880 is to keep BFD sessions and therefore
data probes down (for example to smooth in time control plane churn when
1000s of BFD peers are to all come up in the same time) so it is completely
unrelated to the discussion about BFD clients and link testing before
brining OSPF adj up.

If you want to extend BFD further you are welcome to take it to the BFD WG.
I am cc-ing Greg here as he is one of the best active BFD experts.

But if so this draft should wait till that happens. Alternatively as it has
been suggested we could choose to keep BFD simple and have such timer on
the client side. The behaviour on the client is trivial - Do your client
action (bring IGP adj. up or BGP session up or insert static to the RIB etc
...) if timer X elapsed from BFD session going UP and during that time
there was no transition to DOWN.

Regards,
R.

On Mon, Jan 31, 2022 at 6:31 PM Les Ginsberg (ginsberg) <[email-masked]>
wrote:

> Delays in bringing BFD backup after a previous failure may well be
> warranted in the break-in-middle scenarios.
>
> I am not convinced this needs to be standardized – seems quite appropriate
> as an implementation choice. But if any discussion were to occur in RFCs, I
> think it should be in some BFD document.
>
>
>
> As this draft is focused on OSPF protocol extensions, I don’t think BFD
> dampening needs to be discussed. In any case it should not alter the
> interaction between BFD and protocols. If it takes longer for BFD to come
> up that just means the OSPF adjacency will not come up either – which is
> exactly the behavior that is desired.
>
>
>
>     Les
>
>
>
>
>
> *From:* Albert Fu (BLOOMBERG/ 120 PARK) <[email-masked]>
> *Sent:* Monday, January 31, 2022 6:50 AM
> *To:* Les Ginsberg (ginsberg) <[email-masked]>; [email-masked];
> [email-masked]
> *Cc:* Acee Lindem (acee) <[email-masked]>;
> [email-masked]; [email-masked]
> *Subject:* RE: [Lsr] Working Group Last Call for "OSPF Strict-Mode for
> Your scenario below is indeed something we have encountered in our
> production network in the non-strict scenario, due to "flapping" links,
> where routing protocol could come up before BFD due to "break-in-middle"
> link issue (interface stayed up, so routing protocol remained active).
> Strict mode will address this issue.
>
>
>
> Another point to add is that we do have as a standard on our interfaces to
> safeguard against flapping link by configuring interface
> hold-time/carrier-delay. However, this is only useful in situations where
> the link physically goes down (and fast detection is automatic in most
> implementation).
>
>
>

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Mon, 31 Jan 2022 17:49:56 +0000

Robert –

The paragraph you quote below has to do with BGP behavior in the event “BFD session does not transition to the Up state”.
There is no disagreement about what the protocol (BGP or OSPF) should do in this case. The point of strict-mode is to “wait-for-BFD”.

You, however, are trying to introduce some additional requirements. To this end you said:

“What I find missing in the draft is a mutually (between OSPF peers) timer fired after BFD session is up which in OSPF could hold on allowing BFD to do some more testing before declaring adj to be established. I think just bringing OSPF adj immediately after the BFD session is up is not a good thing.”

So apparently you want BFD to signal UP – but have the protocol do nothing until BFD completes some additional testing. What then was the point of BFD signaling UP to OSPF? And since you want the additional testing to be done by BFD, what new signal should BFD send to OSPF when this is done?
The point of BFD sending UP to its clients is to indicate that BFD thinks the link has been verified from the BFD perspective. I do not see the point of sending two such signals. If you think current BFD testing is inadequate please ask for extensions to BFD (in the BFD WG).

You also said:

“BFD is a great tool to tell you if the end to end path is UP or DOWN. It was not designed to give you any characteristics or metrics for the path quality.”

I agree. But if you are now proposing that protocol adjacencies should not come up until certain link quality metrics are met (e.g., link loss, delay) – you are moving into an area that is completely out of scope of this draft.
I won’t dig deeper into what could be a very lengthy discussion. If you really want to pursue this idea, I suggest you write a new draft.

   Les

From: Robert Raszuk <[email-masked]>
Sent: Monday, January 31, 2022 6:59 AM
To: Albert Fu <[email-masked]>; Les Ginsberg (ginsberg) <[email-masked]>; Ketan Talaulikar <[email-masked]>
Cc: Acee Lindem (acee) <[email-masked]>; [email-masked]; lsr <[email-masked]>
Subject: Re: [Lsr] Working Group Last Call for "OSPF Strict-Mode for BFD" - draft-ietf-lsr-ospf-bfd-strict-mode-04

Les & Ketan

Another data point to the above and this discussion which Albert is co-author of.

Ref: https://datatracker.ietf.org/doc/html/draft-ietf-idr-bgp-bfd-strict-mode

Please see the below paragraph which clearly says BGP BFD Hold time:

To me it is clear that BGP BFD Hold time is on the client side and here affects BGP FSM.

Thx,
Robert.

Robert –

<snip>

a)BFD goes down

a)BFD goes down

   Les

Hi Ketan,

Thx,
Robert.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 31 Jan 2022 19:52:47 +0100

> what new signal should BFD send to OSPF when this is done?

None. Lack of DOWN signal is enough for OSPF to proceed.

Thx,
R.

On Mon, Jan 31, 2022 at 6:50 PM Les Ginsberg (ginsberg) <[email-masked]>
wrote:

> Robert –
>
>
>
> The paragraph you quote below has to do with BGP behavior in the event
> “BFD session does not transition to the Up state”.
>
> There is no disagreement about what the protocol (BGP or OSPF) should do
> in this case. The point of strict-mode is to “wait-for-BFD”.
>
>
>
> *“What I find missing in the draft is a mutually (between OSPF peers)
> timer fired after BFD session is up which in OSPF could hold on allowing
> BFD to do some more testing before declaring adj to be established. I think
> just bringing OSPF adj immediately after the BFD session is up is not a
> good thing.”*
>
>
>
> So apparently you want BFD to signal UP – but have the protocol do nothing
> until BFD completes some additional testing. What then was the point of BFD
> signaling UP to OSPF? And since you want the additional testing to be done
> by BFD, what new signal should BFD send to OSPF when this is done?
>
> The point of BFD sending UP to its clients is to indicate that BFD thinks
> the link has been verified from the BFD perspective. I do not see the point
> of sending two such signals. If you think current BFD testing is inadequate
> please ask for extensions to BFD (in the BFD WG).
>
>
>
> You also said:
>
>
>
> “BFD is a great tool to tell you if the end to end path is UP or DOWN. It
> was not designed to give you any characteristics or metrics for the path
> quality.”
>
>
>
> I agree. But if you are now proposing that protocol adjacencies should not
> come up until certain link quality metrics are met (e.g., link loss, delay)
> – you are moving into an area that is completely out of scope of this draft.
>
> I won’t dig deeper into what could be a very lengthy discussion. If you
> really want to pursue this idea, I suggest you write a new draft.
>
>
>
>    Les
>
>
>
> *Sent:* Monday, January 31, 2022 6:59 AM
> *To:* Albert Fu <[email-masked]>; Les Ginsberg (ginsberg) <
> [email-masked]>; Ketan Talaulikar <[email-masked]>
> [email-masked]; lsr <[email-masked]>

---

From: Ketan Talaulikar <[email-masked]>
Date: Tue, 1 Feb 2022 11:47:03 +0530

Hi Robert,

The comparison between the BFD strict-mode of operation proposals for OSPF
and BGP are not directly comparable since their FSMs are different. In
OSPF, the FSM is going to wait indefinitely in Init state until the BFD
session is set up, while that is not the case for BGP and hence BGP needs
the "BGP BFD Hold time" to close/restart the neighbor FSM.

Please correct me if I am reading this wrong.

What the OSPF draft discusses in Sec 5 is a "hold-down" wait period where
even though the BFD session is established the protocol FSM does not
proceed further until a period of time has passed to ensure the stability
of the BFD session.

Thanks,
Ketan

On Mon, Jan 31, 2022 at 8:29 PM Robert Raszuk <[email-masked]> wrote:

>> Cc: Albert Fu (BLOOMBERG/ 120 PARK ) <[email-masked]>,
>> [email-masked], [email-masked],
>> [email-masked]
>> But the timer I am suggesting is not related to BFD operation, but to
>> OSPF (and/or ISIS). It is not about BFD sessions being UP or DOWN. It is
>> about *allowing BFD for more testing (with various parameters (for
>> example increasing test packet size in some discrete steps)* before OSPF
>> is happy to bring the adj. up.
>>
>> <end snip>
>>
>>
>>
>> [email-masked]>; [email-masked]; Albert Fu
>> <[email-masked]>; lsr <[email-masked]>
>> And that is exactly too soon. In fact if you do that today
>> without waiting some time (if you retire the current OSPF timer) you will
>> not help at all in the case you are trying to address.
>>
>>
>>