Subject: [idr] BGP planned maintenance requirements as an IDR WG doc

From: "DUBOIS Nicolas ROSI/DAS/ARI" <[email-masked]>
Date: Fri, 5 Aug 2005 19:14:40 +0200

Hi Folks,
 
Following IETF 63, I'd like to suggest that BGP planned maintenance
requirements (draft-dubois-bgp-pm-reqs-02.txt) be considered as an IDR
working group document. 
 
Thanks,
 
Nicolas 
+33 1 44 44 28 17
_______________________________________________
Idr mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/idr

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 05 Aug 2005 16:00:09 -0400

In message <[email-masked]>
> requirements (draft-dubois-bgp-pm-reqs-02.txt) be considered as an IDR

What's the point?  There is BGP gracefull restart and there are
routers with warm standby route servers that can do things like
software upgrades without impacting service.  All line cards these
days are hot pluggable.

If you are taking a router completely out of service you can cost it
out of the IGP first then take down the EBGP peerings, then shut it
down.  These days better there is less need to do this.  For example,
on some routers you can do a hardware upgrade to a newer route
processor by installing it as a backup route processor and forcing a
swap of control.

Unless you need to swap out a faulty backplane there are routers today
that can be upgraded incrementally over a very wide range of capacity,
including increasing the backplane capacity, all without a reboot.

Isn't this requirement a few years late?

Curtis

ps - send email off list if you want details that are outside the
scope of the IDR discussion.

---

From: John Leslie <[email-masked]>
Date: Fri, 5 Aug 2005 18:13:16 -0400

> software upgrades without impacting service.  All line cards these

   I cannot support this I-D as a Working Group document either; but my
problem with it is that it fails to propose adequate algorithms for
graceful shutdown. (There are occasions where power is failing and
shutdown will be necessary: if we were to adopt it, we should at least
change the title.)

   It would be good to have a mechanism for deprecating routes a bit
more gracefully than prepending in hopes that some other route will be
preferred; and it would be good to have some mechanism to ensure that
the propagation of deprecating routes never causes packets to be
dropped for lack of an immediately-available alternate route.

   But I do not see that in this draft.

--
John Leslie <[email-masked]>

---

From: Paul Jakma <[email-masked]>
Date: Sat, 6 Aug 2005 15:46:11 +0100 (IST)

On Fri, 5 Aug 2005, DUBOIS Nicolas ROSI/DAS/ARI wrote:

> requirements (draft-dubois-bgp-pm-reqs-02.txt) be considered as an 
> IDR working group document.

I meant to ask this at the WG meeting, but exactly what do you intend 
to cover with this which isnt covered by graceful restart?

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
On the other hand, you have different fingers.

---

From: "DUBOIS Nicolas ROSI/DAS/ARI" <[email-masked]>
Date: Sun, 7 Aug 2005 14:05:45 +0200

Curtis,

Thanks for reading the draft, some comments:

If you've got a 2.5 gig line card and want to upgrade it to a 10 or 40 gig
line card, I still think you need to shut down the BGP session that is
running on it and in order to replace the card.
 
Of course you can do it gracefully with some route-map tricks, ... But on
large operationnal routers it can be quite complex, and it would be better
to have it automatic and directly embedded in the router software.

BTW, graceful restart is really efficient when all (all !) peers are
supporting it.If some of the peer are not supporting it, I wouldn't say it's
harmful but ... You never know. I do not know many service providers who can
successfully use it on a large ASBRs currently. In our case many of our
customers and peers do not support GR even in helper only mode. So GR is
still a "for future use" feature to me.

> down. 

This is not going to work in all situations.

Best regards 
	
Nicolas Dubois
Tel :  0144442817	

> -----Message d'origine-----
> De : [email-masked] [mailto:[email-masked]] De la 
> part de Curtis Villamizar
> Envoyé : vendredi 5 août 2005 22:00
> À : DUBOIS Nicolas ROSI/DAS/ARI
> Cc : DECRAENE Bruno RD-CORE; [email-masked]; FONDEVIOLE Benoit RD-CORE
> Objet : Re: [idr] BGP planned maintenance requirements as an 
> IDR WG doc 
> 
>

---

From: "DUBOIS Nicolas ROSI/DAS/ARI" <[email-masked]>
Date: Sun, 7 Aug 2005 14:08:44 +0200

Basically GR restart is:
	Forwarding plane stays up / control plane says I'll be back.

Planned maintenance requirement is:
	Forwarding going away, control plane going away / let's do it
gracefully.

Best regards,
	
Nicolas Dubois
Tel :  0144442817	

> De : Paul Jakma [mailto:[email-masked]] 
> Envoyé : samedi 6 août 2005 16:46
> Cc : [email-masked]; DECRAENE Bruno RD-CORE; FONDEVIOLE Benoit RD-CORE
> I meant to ask this at the WG meeting, but exactly what do 
> you intend to cover with this which isnt covered by graceful restart?
> 
> regards,
> --

---

From: "DUBOIS Nicolas ROSI/DAS/ARI" <[email-masked]>
Date: Sun, 7 Aug 2005 14:19:13 +0200

Hi John,

Thanks for reading the draft,

>> but my problem with it is that it fails to propose adequate algorithms
for graceful shutdown.

I think it is better to first agree that there is a need for a graceful
shutdown and then discuss of the acceptable solutions. Depending on the
scope of this graceful shutdown, there are many acceptable solutions.

I think a requirement should not explain the problem and the solution at the
same time.

>    It would be good to have a mechanism for deprecating 
> routes a bit more gracefully than prepending in hopes that 
> some other route will be preferred; and it would be good to 
> have some mechanism to ensure that the propagation of 
> deprecating routes never causes packets to be dropped for 
> lack of an immediately-available alternate route.
> 

I agree with that and it is the idea we would like to push with the draft.

Best regards,

Nicolas Dubois
Tel :  0144442817	

> part de John Leslie
> Envoyé : samedi 6 août 2005 00:13
> À : Curtis Villamizar
> Cc : [email-masked]; DUBOIS Nicolas ROSI/DAS/ARI
>    I cannot support this I-D as a Working Group document 
> either; but my problem with it is that it fails to propose 
> adequate algorithms for graceful shutdown. (There are 
> occasions where power is failing and shutdown will be 
> necessary: if we were to adopt it, we should at least change 
> the title.)
> 
>    It would be good to have a mechanism for deprecating 
> routes a bit more gracefully than prepending in hopes that 
> some other route will be preferred; and it would be good to 
> have some mechanism to ensure that the propagation of 
> deprecating routes never causes packets to be dropped for 
> lack of an immediately-available alternate route.
>

---

From: Paul Jakma <[email-masked]>
Date: Sun, 7 Aug 2005 15:04:19 +0100 (IST)

Hi Nicholas,

> 	Forwarding plane stays up / control plane says I'll be back.

Right. Though note that GR specifies a forwarding-state bit, so you 
can signal to your peer, /after/ the fact (when you bring your 
session back up), whether or not your forwarding plane stayed up.

> 	Forwarding going away, control plane going away / let's do it

Ok, I guess my question then is what you think can be done for this 
case, eg /lack/ of graceful-restart capability.

Further, as above, how can you even reliably distinguish between the 
two cases above (presuming GR is enabled)? Given the GR draft, you 
can't determine anything until the peer comes back. Ie, given GR, 
there is a window of uncertainty where you can not say whether 
forwarding stayed up or not, you only know when it comes back and you 
can examine the F bit in the GR capability advertisement for the 
appropriate AFI/SAFI.

Something to consider anyway (the interaction of the two).

A final thought: Does the second case even require any protocol work?

It strikes me that the second case can easily be handled entirely 
within the confines of existing specified BGP, by simply having an 
implementation take the relevant BGP session(s) down (ie sending a 
CEASE) and just waiting a while to let the relevant remote peers 
update their routes, before proceeding with actually removing the 
routes from RIB and FIB and then restarting.

I reckon the most protocol work this needs is just to have 
implementations advertise a "I reckon I need X milliseconds to clear 
Y number of routes" or even just "Hey, if you're going down, try give 
me X seconds advance notice if you can" capability, which could be 
used to supercede any implementation specific default.

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
Finagle's Seventh Law:
 	The perversity of the universe tends toward a maximum.

---

From: "john smith" <[email-masked]>
Date: Sun, 07 Aug 2005 14:22:57 +0000

Hi,

I think if *we know* that a bomb is going to fall on our town at X'oclock, 
we warn everyone to leave town beforehand.... and not wait till the last 
minute and send out a message?

So generally most operators simply send out an "email" to the peering NOCs.
Practical..no?

-JS

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 07 Aug 2005 13:01:39 -0700

> It strikes me that the second case can easily be handled entirely within 
> the confines of existing specified BGP, by simply having an 
> CEASE) and just waiting a while to let the relevant remote peers update 
> their routes, before proceeding with actually removing the routes from 
> RIB and FIB and then restarting.
> 
> implementations advertise a "I reckon I need X milliseconds to clear Y 
> number of routes" or even just "Hey, if you're going down, try give me X 
> seconds advance notice if you can" capability, which could be used to 
> supercede any implementation specific default.

As this mail is the closest to my line of understanding this problem let 
me ask a very basic question of course assuming that GR is for "the 
future" ...

Would it not be enough to simply withdraw the advertised routes before 
bringing the session down towards given peer ? If operator enters 
shutdown on BGP session this should not trigger any immediate BGP/TCP 
action .. instead one could send MP_UNREACH then at some decent timeout 
actually bring the sessions down.

No protocol changes required. All it would take is a local 
implementation little twick ;).

Cheers,
R.

---

From: Paul Jakma <[email-masked]>
Date: Sun, 7 Aug 2005 21:53:40 +0100 (IST)

On Sun, 7 Aug 2005, Robert Raszuk wrote:

[you forgot to attribute text of mine which you quoted]

>> the confines of existing specified BGP, by simply having an implementation 
>> take the relevant BGP session(s) down (ie sending a CEASE) and just waiting 
>> a while to let the relevant remote peers update their routes, before 
>> proceeding with actually removing the routes from RIB and FIB and then 
>> restarting.

> Would it not be enough to simply withdraw the advertised routes 
> before bringing the session down towards given peer ?

See above, shutting the session down implies the remote side should 
withdraw (if GR is /not/ involved). No need to send 160k+ withdraws.

What matters, AFAICT, is that you keep your forwarding in place for 
as long as it takes to allow the remote peers concerned to complete 
withdrawal of your routes. No?

> implementation little twick ;).

Indeed.

It could be made better though. The ultimate would be some kind of 
callback mechanism:

(router shutting down)		(remote peer)
Rzzz's     			Rb
----------------------------------------------
Notify/intend-shutdown	---->
 			       <updates its RIB and FIB>
 		        <---   Notify/shutdown-go-ahead

Course, NOTIFY is an error condition, and 1771 states:

 	"The BGP connection is closed immediately after sending it."

Which presents a problem. BGP would need a non-error notification 
method to do the above.

A "If you shutdown, try send me the NOTIFY Xs in advance, please" 
OPEN capability would be easier/quicker to do, and would at least 
remove need for an implementation to simply guess how long /other/ 
implementations might need.

Anyone reckon that'd be something to write-up? :) (would be vaguely 
complementary to the "Estimated Restart Time" field in the Graceful 
Capability).

regards,
-- 
Paul Jakma	[email-masked]	[email-masked]	Key ID: 64A2FF6A
Fortune:
fortune: cannot execute.  Out of cookies.

---

From: Curtis Villamizar <[email-masked]>
Date: Sun, 07 Aug 2005 18:12:46 -0400

In message <[email-masked]>
> running on it and in order to replace the card.

Only if its an EBGP session.  If its IBGP you don't need to take down
anything to replace a card unless your network has no internal
redundancy.

For the EBGP session we already have BGP Gracefull Restart but that
doesn't cover upgrading if you are going to pull the old card.

The forwarding that is kept in place during gracefull restart won't be
in place while neither card is in the slot.  Are you suggesting that
the BGP session should stay up and maintain a black hole while the
cards are swapped?  I hope not and that's not what I read.

In this particular example, both providers would bring up the 40 Gig
and then bring down the 10 Gig.  Of course since a lot of 40 Gig
interfaces are really concatonated 10 Gigs (4 x OC192), in this
example they'd add 3 10 gigs with no disruption at all.  Either way no
bits are lost if for a time there are two peerings before the old one
is shut down.

Of course if you are out of card slots there are some routers that let
you expand the backplane without disruption.  Not all routers do that.

> to have it automatic and directly embedded in the router software.

Sorry.  I used to maintain a tier-1 network and I can't see why it
would be so difficult to change policy to not advertise the EBGP into
IBGP, then not accept the EBGP, then shut down the EBGP.  Yes, I know
that today's NOC staff are not what they used to be but many of our
NOC staff were college students who didn't know what BGP was until
their on the jub training.  It seems that this would be your best
solution.

In the old days some bright operations engineer might write a perl
program which given a peering line could write the policy to disable
the EBGP to IBGP, then disable the EBGP, then take down the peering.
The operator could then paste that into the router config.  It sounds
like what you are asking for is a way for the router to shut down the
peering in a similar order but do so automaticly.

> still a "for future use" feature to me.

The document that you are asking to be considered looks to be just a
requirements document.

What you might be better off advocating is that more routers implement
BGP gracefull restart and more or your peers enable it.  In the
example you just gave gracefull restart doesn't help.  Given the draft
and then the example above, its not clear what problem you are trying
to address and what solution you have in mind.  Is it time for me to
reread the draft?

> This is not going to work in all situations.

Yes but your document adds nothing to improve it.

> Tel :  0144442817	

Regards,

Curtis

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Mon, 8 Aug 2005 09:13:31 -0400

Assume, for a moment, that there is a legitimate need for knowing 
when planned maintenance is scheduled. If that is the case, however, 
it would seem that it is as important to know when other critical 
resources will be out of service, ranging from an optical multiplexer 
that handles the physical layer for several BGP speakers, to security 
devices that ensure link level or firewall-to-firewall encryption 
between two ISPs.

The first will clearly affect routing, but would it be handled 
adequately with a BGP Maintenance Planned message for each affected 
router, without a message identifying the common resource? As more 
and more root cause analysis goes into network management systems, 
would it be even more useful to know the multiplexer is going down, 
to know the systems dependent on it, and, when the link goes down, 
suppress the consequential router alarms from the NMS? Again, 
assuming knowledge of related entities, the BGP speakers could 
receive a local signal that a resource on which they depend is going 
down, and to begin graceful shutdown N minutes before the scheduled 
maintenance?

In other words, if this is a good idea for BGP, it would seem a good 
idea for other critical components. The question might be split: is 
the requirement useful, and, if so, what should be the transport 
mechanism for the messages? Given the tendency to overload BGP into a 
generalized signaling method, perhaps the focus in IDR should be on 
the transport and the subset of BGP informational messages.

My sense is that the overall problem is more for the Operations than 
Routing area, and that alternate transport mechanisms should be 
examined. It may well be that the messages should be carried in BGP, 
as the de facto one known intercarrier signaling mechanism, but 
perhaps the BGP implementation should be closer to the OSPF Opaque 
LSA.  Certain messages carried in that manner could be locally 
significant in starting graceful shutdown, or a different operational 
procedure that assumes the forwarding path won't be back in the near 
term.

---

From: Pekka Savola <[email-masked]>
Date: Mon, 8 Aug 2005 18:17:14 +0300 (EEST)

On Sun, 7 Aug 2005, Curtis Villamizar wrote:
> IBGP, then not accept the EBGP, then shut down the EBGP.

There is a wide world beyond tier-1's.  I don't think this is so much 
an issue of clue, but rather the amount of time wasted (and the 
potential for typos and other similar results) when performing manual 
operations.

Where tier-1's might differ from the rest is that maybe they have more 
BGP maintenances that writing scripts etc. to deal with these events 
is worth the effort and their continuing maintenance.

For the rest, all of these actions would need to be done manually. 
Doing so is a pain in the ass.  As an operator of a smaller network, 
I'd certainly be happy to see the implementations move to sane or 
easily configurable default behaviour (which may or may not include 
protocol enhancements).

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Ted Seely <[email-masked]>
Date: Mon, 8 Aug 2005 13:06:32 -0400 (EDT)

> So generally most operators simply send out an "email" to the peering NOCs.

yes, and we also gracefully bleed the traffic off affected ckt(s) prior
to said scheduled maintenances.

in the case where it is not scheduled, you usually have to wait for the
router to finish booting anyway, so .....  :)  Perhaps one day we'll solve
that problem as well.

-ted

> -JS

Ted Seely
Principal Network Design Engineer
Internet Engineering - SprintLink
(W) 703.689.6425
(M) 703.967.3289
AIM - wanpro00
Yahoo IM - tseely01

"granular process independence, fault containment, and isolation"