Subject: [Idr] Re: Last Call: 'Graceful Restart Mechanism for BGP' to Proposed Standard (draft-ietf-idr-restart)

From: Pekka Savola <[email-masked]>
Date: Thu, 1 Jun 2006 10:13:25 +0300 (EEST)

On Tue, 30 May 2006, The IESG wrote:
> The IESG has received a request from the Inter-Domain Routing WG to consider
> the following document:
>
> - 'Graceful Restart Mechanism for BGP '
>   <draft-ietf-idr-restart-11.txt> as a Proposed Standard
>
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action.  Please send any comments to the
> [email-masked] or [email-masked] mailing lists by 2006-06-13.

The spec seemed reasonably clear, but seems to have a couple of open 
issues, as described below.

Btw, it's interesting that AFAIK no vendor enables graceful restart by 
default.  Some don't even enable graceful restart helper mode by 
default, but that's more difficult to find out because it seems very 
difficult to see whether this is enabled or not when GR is not 
enabled.  I wonder if vendors see some issues in enabling graceful 
restart or GR helper mode that this document does not bring up.

substantial
-----------

1) Is there an implementation report?  I couldn't find any.

In particular, I'd be interested about two things:
  - section 5 states: "In general the Restart Time SHOULD NOT be greater
    than the HOLDTIME carried in the OPEN." while default in two vendors
    I checked appeared to not honor this; Cisco has 120sec restart, 90 sec
    holdtime.  Juniper has 300 sec restart, 60 sec holdtime.

  - Forwarding State bit seems interesting especially to for planned outage
    use, but I does any vendor allow it to be configured?  I couldn't find
    any..

2) The deployment and security considerations sections are not 
sufficiently clear on the hazards of graceful restart.  For example, 
if the router crashes in a certain manner or shuts down due to power 
loss, the traffic is blackholed for up to Restart Time + following 
routing convergence (instead of just routing convergence).

This threat can be mitigated e.g., by using control-plane independent 
liveness detection mechanisms (e.g., BFD on hardware -- probably no 
one implemented that yet) or just using graceful restart for planned 
maintenance (but forwarding state bit doesn't appear to be 
configurable so this isn't an option in practice).  Also lowering 
restart time to something like 10 seconds (if you have multiple 
control processors) might help.

It is also not clear how the issues above differ whether the BGP 
speaker has one or multiple control processors.  I think many 
deployment require two but the assumptions about this aren't stated in 
the document.

editorial
---------
    This document proposes a mechanism for BGP that would help minimize
    the negative effects on routing caused by BGP restart.

==> maybe s/proposes/describes/ ?

5.1. Procedures for the Restarting Speaker

    When the Restarting Speaker restarts, possible it SHOULD retain, if
    possible, the forwarding state for the BGP routes in the Loc-RIB, and
    SHALL mark them as stale.

==> "possible it SHOULD retain" doesn't parse, maybe ", possible" was 
leftover from editing and should be removed..

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

_______________________________________________
Idr mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/idr

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Jun 2006 07:59:04 -0400

> restart or GR helper mode that this document does not bring up.

At least one vendor prefers that their customers take advantage of
redundant route processor support that makes it look to the external
world as if no route processor failure occurrred.  For most EBGP the
actual protocol exchange is on the line card processor.  For IBGP or
EBGP multihop the "tepid" route processor takes over (not quite hot
since recovery of some derived state occours on failover).

At least two vendors have done this although one is no longer in
business.  The remaining vendor is not one of the big two but has some
tier one and tier two customers.

> substantial

The practical reason for this is most route processors can't reload
and get to a state where BGP can come up in less than their HOLDTIME.
Bad things happen if the grace timer expires first.  I think this
reflects a worst case setting.

> routing convergence (instead of just routing convergence).

You need to also have IGP graceful restart running and the grace time
for that is often shorter.

The down side of a not-so-gracefull restart in which graceful restart
is indicated is a recovery in the lesser of route processor reboot
time or the grace period plus convergence.  In practical terms this is
almost always over a minute, sometimes a few minutes.

> control processors) might help.

Lets just leave it at "there are other ways to accomplish BGP
failover".

btw- Same vendor does BFD on line cards with line cards aware of route
processor liveness or the state of a route processor failover.  They
don't need gracefull restart since the failover is completely
transparent to the outside world but implement it (I think).

> deployment require two but the assumptions about this aren't stated in 

Depends on whether the extra control processors just sit there and
require intervention to be put into service or act as cold spares or
act as warm or hot spares.  I think how redundant route processors are
used is out of scope for the document though.  A hibernating spare
seems common (spare has read in config and that stuff and "has a
heartbeat" but has acquired no adjacenies or routing information).
Graceful restart seems appropriate for this sort of sparing and in
that case you could configure the graceful restart timer to a much
lower value than the default.

> leftover from editing and should be removed..

Your point about an implementation report is valid.  I think you'll
find most major vendors already implement this and support this enough
to make the effort to provide an implementation report so we'll just
have to wait for one.  I don't know for sure how many providers
deplyed this and whether some of those later decided to turn it off
but I do know that providers have insisted that it be implemented.
For now it is in use.  Maybe in a few years redundant route processors
will be more common but even then single route processor routers or
ones without a warm or hot spare will still exist for a long time.

Curtis

---

From: Yakov Rekhter <[email-masked]>
Date: Thu, 01 Jun 2006 06:17:39 -0700

Pekka,

> restart or GR helper mode that this document does not bring up.

Here is the URL for the implementation report:

     http://tools.ietf.org/wg/idr/draft-ietf-idr-bgp-gr-survey/draft-ietf-idr-bgp-gr-survey-01.txt

Yakov.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 1 Jun 2006 18:34:45 +0300 (EEST)

Hi,

I'll follow-up to Curtis inline, but first I'd like to make the 
comment about the implementation report: 
http://tools.ietf.org/wg/idr/draft-ietf-idr-bgp-gr-survey/draft-ietf-idr-bgp-gr-survey-01.txt

It's 4 years old, though fortunately (or unfortunately) the spec 
hasn't really changed that much since.  The main differences seem 
to be:
- now using uppercase language
- restarting router procedures now SHOULD retain forwarding state 
(instead of shall)
- BGP FSM update inclusion
- Selection_Deferral_Timer specification spelled out

I'd have liked greater detail in the implementation report, given that 
at least two vendors state compliance but do not note that (or why) 
they do not conform to a SHOULD NOT.  While this specific issue 
probably does not hinder interoperability, it raises a question 
about the credibility of the spec.

> reflects a worst case setting.

Correct, but this gets back to the point below -- "how applicable 
is this with just a single control processor?"

These seem to be two very distinct scenarios with very different 
requirements and applicability.

> for that is often shorter.

That does not apply to all the sessions though.

(Funnily enough, a major vendor has minimum GR restart-time for BGP 1 
second, but for IS-IS 30 seconds..)

> almost always over a minute, sometimes a few minutes.

You seem to make some assumptions here.  For most non-stub routers, 
there's always a backup path, and reconvergence doesn't need to take 
more than a second or two (we've certainly measured very short BGP 
reconvergence inside our network).  For stub edge routers which only 
connect single-homed users and are not used to transit traffic to/from 
any other router, always doing graceful restart may be better than 
non-GR.

> lower value than the default.

I personally believe that IETF documents should be useful to 
non-vendors as well.. and more than just bare bones spec required for 
interoperability would be useful to have. Given that we already have a 
section 'Deployment Considerations', it could easily be expanded to 
include useful information.  Likewise, security considerations would 
(perhaps in addition to above) need to better discuss the hazards of 
delayed convergence, i.e., routing packets into a black hole instead 
of rerouting around the problem.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 01 Jun 2006 13:20:24 -0400

Pekka,

Sorry to trim so much.

The things I pointed out were mostly regarding an alternative to
graceful restart and some of the implication of the various ways of
supporting a spare route processor (one or more).  That sort of thing
does seem beyond the scope of the document.

I do support this document and I don't feel that it needs to be a
review paper on the convergence properties of the various approaches
to handling route processor failure or planned maintenance.  At most
it should briefly describe how it is intended to improve this
situation, which is does in Section 5 "Operation".  Section 7
"Deployment Considerations" points out the transient blackhole or loop
problem if things change.  It is safe to assume that the reader knows
that blackholes and route loops are bad and knows in detail what the
consequences are.  I don't see that omiting an embedded review paper
makes this useful only to vendors unless you consider protocol
specification in general to be of no use to providers or researchers
or anyone but vendors.

Curtis

---

From: "John G. Scudder" <[email-masked]>
Date: Fri, 2 Jun 2006 11:10:57 -0400

Pekka,

Thanks for your comments.  I've addressed the substantive ones below. 
(No quibbles with the editorial.)

>    holdtime.  Juniper has 300 sec restart, 60 sec holdtime.

I think this demonstrates that we erred in using the SHOULD NOT 
language, and should weaken it to make it a suggestion, rather than 
using the SCARY ALL CAPS standards words.  There's no 
interoperability issue in doing so.  (BTW I believe you have the 
cisco numbers wrong, should be 120/180 which is in fact compliant 
with the spec as written.)

>    use, but I does any vendor allow it to be configured?  I couldn't find

I'm not aware of any though there could be.  I agree with your 
comment about planned outage use, and so does the spec -- such an 
implementation choice is specifically discussed in the last paragraph 
of section 5.1.

>routing convergence (instead of just routing convergence).

The deployment considerations section does mention the possibility of 
black holes and loops, though it doesn't specifically mention the 
power failure scenario.  Granted we don't present the algebra or all 
the possible failure cases, but I like to think that our audience is 
bright enough to do that.  I tend to prefer drafts and RFCs that 
assume I can think for myself and don't waste my time making me read 
through stuff that's self-evident.

That said, if there's significant support for expanding the 
deployment section, I don't see any real reason not to.  In any case 
we can explicitly mention the loss-of-FIB (e.g., power loss) scenario.

You also mention the security considerations section above but I 
don't see the connection?  (Your later comment to Curtis, "Likewise, 
security considerations would (perhaps in addition to above) need to 
better discuss the hazards of delayed convergence, i.e., routing 
packets into a black hole instead of rerouting around the problem" 
doesn't help me understand why you think black holing is a security 
rather than a deployment consideration.)

>This threat can be mitigated e.g., by using control-plane 
>independent liveness detection mechanisms (e.g., BFD on hardware -- 
>probably no one implemented that yet)

Yes, and as I'm sure you're aware this was one of the principal 
motivations behind BFD.  At least some implementations I'm aware of 
can do this.  It does seem worthwhile to me that we mention this in 
the spec.

>or just using graceful restart for planned maintenance (but 
>forwarding state bit doesn't appear to be configurable so this isn't 
>an option in practice).  Also lowering restart time to something 
>like 10 seconds (if you have multiple control processors) might help.
>
>deployment require two but the assumptions about this aren't stated 
>in the document.

Indeed, this is an implementation detail.  As such I think it's 
correct for the document to be silent on this point.

Regards,

--John

---

From: Pekka Savola <[email-masked]>
Date: Sat, 3 Jun 2006 11:22:54 +0300 (EEST)

On Fri, 2 Jun 2006, John G. Scudder wrote:
> I think this demonstrates that we erred in using the SHOULD NOT language, and 
> should weaken it to make it a suggestion, rather than using the SCARY ALL 
> CAPS standards words.  There's no interoperability issue in doing so.

Agreed, though picking the right restart time is still very important 
if you want the protocol to work well without negative impact in worst 
case scenarios.

>> 2) The deployment and security considerations sections are not sufficiently 
>> clear on the hazards of graceful restart.  For example, if the router 
>> crashes in a certain manner or shuts down due to power loss, the traffic is 
>> blackholed for up to Restart Time + following routing convergence (instead 
>> of just routing convergence).
>
> The deployment considerations section does mention the possibility of black 
> holes and loops, though it doesn't specifically mention the power failure 
> scenario.

The current text is in the context of changes in routing _after_ GR 
has been started.  A loop/blackhole as a result of initiating GR, and 
a loop that occurs because a routing update was missed due to ongoing 
GR are two very different things.  The document is silent about the 
former.

> Granted we don't present the algebra or all the possible failure 
> cases, but I like to think that our audience is bright enough to do 
> that. I tend to prefer drafts and RFCs that assume I can think for 
> myself and don't waste my time making me read through stuff that's 
> self-evident.

Degradation of the "BGP service model" is AFAICS one of the biggest 
reasons why graceful restart hasn't been used that much, and folks are 
pretty uncertain about turning it on.

If we want folks to endorse using our protocols, more text would not 
hurt.  If we don't care, I can see that publishing bare bones specs 
could be deemed sufficient.

> You also mention the security considerations section above but I don't see 
> the connection?  (Your later comment to Curtis, "Likewise, security 
> considerations would (perhaps in addition to above) need to better discuss 
> the hazards of delayed convergence, i.e., routing packets into a black hole 
> instead of rerouting around the problem" doesn't help me understand why you 
> think black holing is a security rather than a deployment consideration.)

The existing routing (and BGP) "service model" is that it reroutes 
around failures in the network if another path is available.

By introducing graceful-restart, you change BGP and this service model 
so that in certain common scenarios, packets get blackholed or looped 
up to restart-time instead instead of always rerouting around the 
problem.

This has an impact on availability of the network.  Availability is 
one important aspect of security.

>> or just using graceful restart for planned maintenance (but forwarding 
>> state bit doesn't appear to be configurable so this isn't an option in 
>> practice).  Also lowering restart time to something like 10 seconds (if you 
>> have multiple control processors) might help.
>> 
>> It is also not clear how the issues above differ whether the BGP speaker 
>> has one or multiple control processors.  I think many deployment require 
>> two but the assumptions about this aren't stated in the document.
>
> Indeed, this is an implementation detail.  As such I think it's correct for 
> the document to be silent on this point.

Maybe.  It is not required for interoperability, but it would be 
useful if we wanted the protocol to be useful.

For instance, the recommendation for the restart-time could be very 
different (e.g., 5 or 10 seconds instead of, say, 300) depending on 
whether multiple control processors are available or not.

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 03 Jun 2006 06:55:20 -0400

> GR are two very different things.  The document is silent about the 

If "about the former" means initiating GR to a router that died then
thats where BFD comes in.  BFD in now often implemented in the line
cards with a liveness check on the route processor such that the line
cards can quickly tell if the route processor is in its reload or
failover process and proceeding normally or has tripped up in this
process.  If the RP is going to need a full reload rather than a fast
failover or where failover is not supported, if the RP is dead rather
than reloading normally, the line card can indicate BFD deadness by
not responding and shortcut the GR time.  This is all implementation
detail and none of it belongs in the GR protocol spec.

> could be deemed sufficient.

Protocol specs are suppose to define the protocol and provide major
security and deployment considerations, and not provide excessive text
simply advocating their use.

> one important aspect of security.

If it exposed a denial of service attack it would be a security issue.
If it just has the potential to lengthen a failure or worsen a failure
(causing a black hole) then it is a deployment consideration.

AFAIK providers are using this (or similar approaches) to provide
gracefull software upgrade which for some routers occurs more often
than RP failures (ie: a software upgrade might occur once or twice a
year while RP failures might be many years but even more likely might
not occur for the life of the router).

(... as opposed to the pre-ECC memory days when RPs would die much
more often on "high sun spot days" - not likely to be related but
everything seems to get blamed on sun spots in a joking manner - might
also been related more to memory timing or noise on the boards than to
lack of ECC but that's another story).  :-)

> whether multiple control processors are available or not.

How about adding "The use of redundant control processors is outside
of the scope of this document except to mention that the
Selection_Deferral_Timer timer can be set to a much lower value in
implementations which support this feature and make use of BGP
Graceful Restart"?  This could be tacked onto the deployment
considerations section.

Curtis