Subject: RE: [idr] BGP planned maintenance requirements as an IDR WG doc

From: "DECRAENE Bruno RD-CORE-ISS" <[email-masked]>
Date: Mon, 8 Aug 2005 16:14:10 +0200

> ....Well but if they are heavy traffic bearing links it would 
> be better to inform your peers a few hours in advance so that 
> he can rewrite his policies.. it is also his $$ after all :-)

If it's my customer (Internet or VPN) this is my $$ if I broke the SLA.
In a BGP/MPLS VPN situation, if I manage the CE, it's my time/money to
rewrite it's policy.

> >Hi John,
> >
> > > Hi,
> > >
> > > I think if *we know* that a bomb is going to fall on our town at 
> > > X'oclock, we warn everyone to leave town beforehand....
> > > and not wait till the last minute and send out a message?
> >
> >That's exactly our point.
> >We know the forwarding will be shutdown. Why wait the last second to 
> >send a BGP CEASE? Why not inform all your BGP peers a few 
> >seconds/minutes before? In a way that would allow minimal 
> packet loss 
> >during BGP convergence.
> >
> >
> > > So generally most operators simply send out an "email" to the 
> > > peering NOCs.
> > > Practical..no?
> >
> >Warning the peer is one thing. Shuting down the session with minimal 
> >pack loss is still desirable.
> >
> >In some situation, we don't care about loosing some trafic 
> or both NOC 
> >we'll write some route map/policy to gracefully reroute the 
> packets (or 
> >believe so).
> >
> >In some other situation, customers do cares about their SLA, and 
> >modifying the route map each time is too error prone and time/money 
> >consuming.
> 
> _________________________________________________________________
> Express yourself instantly with MSN Messenger! Download today 
> it's FREE! 
> http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
> 
> 

_______________________________________________
Idr mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/idr

---

From: "john smith" <[email-masked]>
Date: Mon, 08 Aug 2005 14:33:01 +0000

Hi,

What about "his" customers? His SLAs may be more stringent than your SLAs...

I would agree with Howard in a way, GROW would be a nice place to bring this 
up.
As far as i know most people rely on exchanging information a few hours 
prior to a restart.

>From: "DECRAENE Bruno RD-CORE-ISS" <[email-masked]>
>To: "john smith" <[email-masked]>,<[email-masked]>
>Subject: RE: [idr] BGP planned maintenance requirements as an IDR WG doc
>Date: Mon, 8 Aug 2005 16:14:10 +0200
>

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

---

From: "john smith" <[email-masked]>
Date: Mon, 08 Aug 2005 14:47:54 +0000

any vendor has a hook into microsoft outlook calender with BGP? ;-)

>From: "john smith" <[email-masked]>
>To: [email-masked], [email-masked]
>Date: Mon, 08 Aug 2005 14:33:01 +0000
>
>Hi,
>
>I would agree with Howard in a way, GROW would be a nice place to bring 
>this up.

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 08 Aug 2005 22:42:06 +0200

John,
> As far as i know most people rely on exchanging information a few hours

To understand the requirements from Nicolas and Bruno, we have to
consider that the Internet has changed. Ten years ago, routing protocols
were used to support mainly researchers sending best effort packets. For
those users, loosing ten seconds worth of packets was not a problem.

Today, IP routing protocols are used to support IP and MPLS-based
networks carrying various types of mission critical data including :
- Voice over IP
- Video over IP
- Virtual Private Networks
- various types of leased line emulation

For many of those services, SPs are pushed by their customers to offer
stringent Service Level Agreements. If those SLAs are not met, the
customer is often eligible for a refund. To avoid losing packets,
Service Providers use various types of techniques. In the data plane,
marking, scheduling, buffer acceptance are commonly used to favor
important packets over others. In the control plane, several approaches
are used :
- tune the implementation of the routing protocols to achieve sub-second
convergence. This is possible with ISIS/OSPF, but is more challenging
with BGP
- use MPLS-based fast reroute techniques to protect important LSPs in
case of failures
- develope IP-based fast reroute techniques to provide similar
mechanisms for pure IP networks

All those solutions try to avoid as much as possible packet losses. The
document proposed by Nicolas and Bruno is another useful step in this
direction. Reacting to planned failures is important as measurements
done in ISP networks have shown that a large fraction of the failures
are indeed planned.

Another point to consider when evaluating this document is the
following. Today, most operators perform their maintenance operations
early in the morning, e.g. 4-6 am to reduce the impact of the
maintenance. From a human ressource viewpoint, this implies that special
staff is required during the night shift. This increases the costs of
the ISP. If it was possible to perform maintenance operations during the
day *without impacting traffic and causing any packet loss*, I'm sure
that ISPs would prefer to perform their maintenace during the day while
all the engineers are available to react in case of problems. This would
require the ability  to change any configuration without causing any
packet loss. Nicolas and Bruno's draft are one step in this direction
for eBGP.

Best regards,

Olivier

-- 
CSE Dept. UCL, Belgium - http://www.info.ucl.ac.be/people/OBO/

---

From: "john smith" <[email-masked]>
Date: Mon, 08 Aug 2005 21:09:05 +0000

>those users, loosing ten seconds worth of packets was not a problem.

ok

>- various types of leased line emulation

ok

>mechanisms for pure IP networks

ok.

>done in ISP networks have shown that a large fraction of the failures

Please let me know which is the institute invests time and money only to 
come to a conclusion that "failures are planned".
I would like to see the data and their defination of a so called "planned 
failure". :)

>all the engineers are available to react in case of problems.

How has the time of day got anything to do with the matter? Your peers will 
never be able to rewrite their policies in 3 minutes, whether you perform 
the "planned failure" at 2am or 2pm. You would have to give them a few hours 
of prior notification.

>This would

I think I just pointed out above how a just a few minutes of warning can 
lead to packet loss. While I do understand what you are trying to say, I do 
not get why you feel signalling a "planned failure" gives one any greater 
advantage than sending an email?

>Olivier
^^^^^^^^^^^^^^^^

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 08 Aug 2005 23:22:39 +0200

John,

> I would like to see the data and their defination of a so called
> "planned failure". :)

Basically, they assumed that if the failures occur during maintenance
hours they are caused by manual operations and thus are planned. See e.g. :
http://www.cambridge.intel-research.net/~gianluca/talks/icnp-tutorial.pdf
, slide 61

http://www.ripe.net/ripe/meetings/ripe-47/presentations/ripe47-routing-fcp.pdf,
slide 5

> How has the time of day got anything to do with the matter? Your peers
> will never be able to rewrite their policies in 3 minutes, whether you
> perform the "planned failure" at 2am or 2pm. You would have to give them
> a few hours of prior notification.

You assume that the peer should perform manual operations to handle the
planned failure. I would prefer a solution where the BGP routers in the
peer automatically adapt to the planned failure.

> lead to packet loss. While I do understand what you are trying to say, I
> do not get why you feel signalling a "planned failure" gives one any
> greater advantage than sending an email?
> 

If you send an email, you warn the human operators. If you send BGP
messages, you warn the routers. I would prefer a model where BGP routers
 can automatically react to planned failures compared to a model where a
human intervention is required to handle a planned failure elsewhere.

Olivier

---

From: "john smith" <[email-masked]>
Date: Mon, 08 Aug 2005 21:44:21 +0000

Well  i guess if they say so then they must be right.
Lets go ahead with this feature!! what are we waiting for! :)

>From: Olivier Bonaventure <[email-masked]>
>To: john smith <[email-masked]>
>CC: [email-masked], [email-masked]
>Subject: Re: [idr] BGP planned maintenance requirements as an IDR WG doc
>Date: Mon, 08 Aug 2005 23:22:39 +0200
>
>John,
>
> >

http://search.msn.com/

---

From: vijay gill <[email-masked]>
Date: Mon, 08 Aug 2005 22:47:51 -0400

> human intervention is required to handle a planned failure elsewhere.

We do a lot of maintenances on our routers and we have a tool that 
automatically sends out mail to each contact that is on that particular 
router. The reason we do this is that people get irate when sessions or 
traffic flows get disrupted (notice I said people, not routers) and they 
would like for prior notification so they aren't surprised when 
something alarms on their monitoring system. As someone who is on-call, 
having a BGP session flap will almost always result in a page. I'd 
rather my monitoring desk already be aware that there will be an event 
and therefore, not page me.

Also, for major peers etc, it is important to get notification, because 
it could be that we are in scheduled jeopardy and as such, we can 
disallow that maintenance if necessary, esp. if we have backup capacity 
being down. Eg, one of my peers is upgrading code in frankfurt, and I'm 
upgrading code in london and frankfurt is acting as a backup for london. 
Having the routers understand the failure is completely useless to me. I 
want people to understand so I can plan around the issue.

/vijay

---

From: "john smith" <[email-masked]>
Date: Tue, 09 Aug 2005 07:43:40 +0000

well all said and done,
I know atleast one router vendor who could possibly consider using the 
"rollback" feature to achieve this.

>From: vijay gill <[email-masked]>
>To: Olivier Bonaventure <[email-masked]>
>CC: [email-masked]
>Date: Mon, 08 Aug 2005 22:47:51 -0400

>
>Also, for major peers etc, it is important to get notification, because it 
>could be that we are in scheduled jeopardy and as such, we can disallow 
>that maintenance if necessary, esp. if we have backup capacity being down. 
>Eg, one of my peers is upgrading code in frankfurt, and I'm upgrading code 
>in london and frankfurt is acting as a backup for london. Having the 
>routers understand the failure is completely useless to me. I want people 
>to understand so I can plan around the issue.
>
>/vijay
>

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar - get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

---

From: Ted Seely <[email-masked]>
Date: Tue, 9 Aug 2005 14:01:17 -0400 (EDT)

> want people to understand so I can plan around the issue.

Well said Vijay.  This captures exactly why automation (signalled or
invoked) can burn you.  Large networks where large amounts of traffic
will swing ( 500M to 1G+ ) when circuits go down because of planned events
must be coordinated at the human level.

-ted