Subject: RE: [idr] BGP planned maintenance requirements as an IDR WG doc

From: "DECRAENE Bruno RD-CORE-ISS" <[email-masked]>
Date: Wed, 10 Aug 2005 15:29:36 +0200

Hi Paul,

> At least one signalling method exists (CEASE) - why not implement 
> CEASE+wait first and get some experience of it before considering
> what protocol changes could make it work better?

Could you be a little more specific about your proposition?

>From your description, I understand that you close the BGP session
according to the BGP spec, then wait some time, then shut down the
interface and proceed with the maintenance.

If this is a correct understanding, let's take the following topology
(from the requirement draft):
                             '  
                             '  
                             '  
                       AS1   '      AS2  
                             '  

                       /-----------ASBR1-----P1----  
                      /                      |   
                     /                       |  
                 CUST                        | 
                     \                       | 
              X.Y/16  \                      | 
                       \-----------ASBR2-----P2---- 

                             '  
                             '  
                       AS1   '      AS2  
                             '  
                                      
                   Figure 1: Redundant peering example.  

We want to close CUST-ASBR1 link.

After sending the "CEASE" to CUST, ASBR1 removes the routes learned
through this eBGP session. You have 2 cases:

-0- BGP don't know the backup path, so it will remove the destination
from the RIB. ==> packets to the destination X.Y/16 (your customer) are
dropped on ASBR1.
-1- BGP knows the backup path, so it will install it in the RIB ==>
packets to that destination are sent back to P1. Unfortunately, P1
doesn't know yet that CUST-ASBR1 eBGP session is down. So it sends back
to packet to ASBR1 ==> packets are looped between ASBR1 and P1.

>I'd like to see:

> - implementation of the CEASE and wait solution
> - some field experience of it
> - maybe kind of informational ID discussing the previous two points

> *Then* and only then is it sensible to start talking about a 
> potential requirements draft for tackling any shortcomings in BGP wrt 
>maintainance, and possible suitable changes to it.

> Understand the problem first please (surely?).

Could you please have a look at slide 4 in our presentation?
It looks to me that your proposition has been tested (The BGP session
have been shutdown, the interface has not even been shutdown) and yet
you lose some trafic.

Regards,
Bruno

_______________________________________________
Idr mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/idr

---

From: Paul Jakma <[email-masked]>
Date: Wed, 10 Aug 2005 15:36:30 +0100 (IST)

> Could you be a little more specific about your proposition?

Put a delay in between NOTIFY/CEASE and updating your FIB.

> from the RIB. ==> packets to the destination X.Y/16 (your customer) are

Not sure what you mean by "BGP" here - ASBR1 or P1?

1. ASBR1 Sends CEASE/NOTIFY to CUST1
2. ASBR1 processes any routes it may have received from CUST1
 	2a. Form withdrawals and send to P1
 	2b. Queue up FIB updates to process with some delay X
3. After delay X expires, process the FIB updates due to 2a

P1 will get the withdrawal. If it doesn't know the backup path via 
P2, it's surely in trouble regardless of what you propose for BGP ;). 
If you mean ASBR1 is not aware - that's ok, it will continue 
forwarding to CUST1 for a while.

The delay X, if large enough, gives P1 sufficient time to process the 
withdrawal, calculate the new path via P2 and install it.

ASBR1, during this time, will continue forwarding any packets P1 
sends via it on to CUST1. If ASBR1 does not know about the backup 
path, that's ok, because when ASBR1 eventually updates it's FIB, P1 
should already have updated its FIB and will no longer be sending 
packets to it for CUST1.

Eventually P1 should pass an announcement for CUST1 (learned from P2) 
to ASBR1 and ASBR1 updates its FIB. ASBR1 shouldn't see any traffic 
at all directed towards CUST1 though - it will be idle.

> to packet to ASBR1 ==> packets are looped between ASBR1 and P1.

If ASBR1 knows the backup path, that's ok. Because it will not update 
its FIB for a while - sufficient time for P1 to update its own RIB 
and FIB to point CUST1 route in the direction of P2.

This is all essentially what several people have been considering 
already, eg the "Ordered FIB Updates" draft (i think you're aware of 
it, no?):

 	http://bgp.potaroo.net/ietf/idref/draft-francois-ordered-fib/

Essentially, seems to me there are two rules of thumb:

 	- for a path shutdown, FIB updates should lag RIB updates

 	- for a path addition, FIB updates should lead RIB updates
 	  and propogation.

(iirc) Francois had a very interesting presentation at the routing 
area meeting on this. They're looking deeply at this already, 
mathematically AIUI (havn't seen any papers on that side of things 
though), so hopefully some really useful techniques will come out of 
that in time.

> Could you please have a look at slide 4 in our presentation?

Where is your presentation exactly? I remember the presentation only 
vaguely at this stage - note that powerpoint format is inconvenient 
for me.

> It looks to me that your proposition has been tested (The BGP 
> session have been shutdown, the interface has not even been 
> shutdown) and yet you lose some trafic.

If the FIB was updated, i can imagine so, yes.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
We'll try to cooperate fully with the IRS, because, as citizens, we feel
a strong patriotic duty not to go to jail.
 		-- Dave Barry

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 10 Aug 2005 21:16:57 +0200

Paul,

> Put a delay in between NOTIFY/CEASE and updating your FIB.

You need at least to be able to distinguish between :
- a NOTIFY/CEASE that must be processed immediately
- a NOTIFY/CEASE that must be processed after N seconds (note that
selecting the good value for N is an issue)

>> from the RIB. ==> packets to the destination X.Y/16 (your customer) are

Note that this can happen simply if CUST-ASBR2 has a lower local-pref
than CUST-ASBR1

> P1 will get the withdrawal. If it doesn't know the backup path via P2,
> it's surely in trouble regardless of what you propose for BGP ;). If you
> mean ASBR1 is not aware - that's ok, it will continue forwarding to
> CUST1 for a while.
> 

The solution to be used will depend on how packets are forwarded inside
the AS. There are two possible solutions :
- pervasive BGP where all routers run BGP and consult their FIB to
forward each received transit packet
- encapsulation such as an MPLS-enabled network. In this case, the
ingress router selects the egress router and forward the packets
directly to the egress router (no transit router need to consult its BGP
FIB to forward the packet).

With pervasive BGP, the solution must ensure that no transient
forwarding loop will occcur during the update of the FIBs of all
routers. This is more complex than simply setting a delay of N seconds
on one router.

...
> already, eg the "Ordered FIB Updates" draft (i think you're aware of it,
> no?):
> 

Note that this document considers the ordered update of FIBs with only a
link-state routing protocol. It does not consider any interaction with BGP.

> (iirc) Francois had a very interesting presentation at the routing area
> meeting on this. They're looking deeply at this already, mathematically
> AIUI (havn't seen any papers on that side of things though), so
> hopefully some really useful techniques will come out of that in time.

There is a full paper on the FIB update order with mathematical proofs,
see Pierre Francois' INFOCOM paper available from
http://www.info.ucl.ac.be/people/OBO/biblio.html

Best regards

Olivier

---

From: Paul Jakma <[email-masked]>
Date: Fri, 12 Aug 2005 17:51:26 +0100 (IST)

Hi Olivier,

> selecting the good value for N is an issue)

The receiving router or the sender? As for selecting a good value, 
yes :), tis an issue.

> routers. This is more complex than simply setting a delay of N 
> seconds on one router.

Generally, yes, I'm sure it is. Seems it could work for the scenario 
presented though, if the two 'edge' routers (CUST-ASBR{1,2}) both 
delayed their FIB updates, thus giving time for interior iBGP 
speakers (P1, P2) to update their paths. (Ie if N is greater than the 
time it takes for P1 and P2, then there is no time during which 
either ASBR1 or ASBR2 will refuse to forward to P2).

It likely wouldn't work for more complex edge scenarios, but that's 
(hopefully) what you guys are considering ;).

> Note that this document considers the ordered update of FIBs with 
> only a link-state routing protocol. It does not consider any 
> interaction with BGP.

Indeed.

Your (plural) work applies generally though to topology updates, 
doesn't it?

Further, given iBGP should not result in complex forwarding (iBGP 
topology should nearly always be 'flat' in the abstract - nexthops 
are abstracted, should always point to the 'edge'), it means, from an 
iBGP POV, you should not have a complex topology to contend with.

Eg, in the example scenario given previously, consider some other 
router within AS 2, call it P3. It's interior path to the routers 
ASBR{1,2} is arbitrary (be it: many hops, lots of redundant paths; or 
maybe its directly to one or both of P1 and P2, who knows). However, 
as iBGP abstracts interior topology it's iBGP path to CUST is always:

 	x/y via CUST

The only sane place to distribute this external nexthop into iBGP (if 
at all, if using IGP - then we dont need to worry at all) is on 
either ASBR1 or ASBR2.

So the delayed update trick on ASBR1 will work fine, as long as the 
delay is sufficient for iBGP to converge on ASBR2.

Similarly, if CUST's sessions to ASBR1 and ASBR2 have different 
nexthops, in iBGP we will have one of:

 	x/y via CUST-nh1 (connected to ASBR1)

or

 	x/y via CUST-nh2 (connected to ASBR2)

installed throughout the rest of the iBGP domain. And again, the 
delayed FIB update on ASBR1 should allow anyone in the iBGP of AS2 
using the former path to update to the latter.

In both cases, while others converge, ASBR1 will still forward to 
CUST (as will ASBR2 obviously). Once convergence is complete, ASBR1 
should no longer see any packets destined for x/y at all, and it 
doesnt matter what it does really.

Ie, the abstracted topology which iBGP propogates makes analysis of 
this quite easy, doesn't it? At least for the shutdown case, not 
considering the 'up' case.

Obviously, if the nexthops are carried in IGP, then updates of those 
are an IGP problem - which is being looked at, AIUI (by yourself and 
others).

> There is a full paper on the FIB update order with mathematical 
> proofs, see Pierre Francois' INFOCOM paper available from 

Ah, bedankt! I'll have a read.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Old MacDonald had an agricultural real estate tax abatement.