Subject: [GROW] draft-iops-grow-bgp-session-culling-00

From: Job Snijders <[email-masked]>
Date: Sun, 12 Mar 2017 23:16:55 +0100

Dear GROW,

With this BCP Internet-Draft we hope to draw some attention to good
practises which can be applied by IP networks or IXPs to mitigate
negative impact caused by maintenance operations on lower layer
networks. The idea is to promote the concept of breaking the
control-plane in a controlled fashion, before actually breaking the
data-plane.

Regarding the "Voluntary BGP Session Teardown Recommendations" - we all
too often see operators do the equivalent of 'yanking the cable out of
its socket' while not allowing any time for reconvergence. As far as I
am aware, there is no documentation which recommends to shutdown BGP
sessions and allow for some time for the traffic to subside due to
rerouting, and then commence with the maintenance.

Some background on the "Involuntary BGP Session Teardown
Recommendations": a number of years ago an (at the time novel) approach
was presented on how to reduce the negative impact of IXP maintenance.
Since then the idea gained popularity and is applied at more and more
IXPs. The video + pdf are available here:
https://ripe67.ripe.net/archives/video/116/

The approaches outlined in this document may not be required in every
network topology, for instance some IXPs have fantastic 'make before
break' methodologies accomplished through a layer of photonic switches.

I'd like to ask the chairs for a 10 minute slot in GROW's session at
IETF 98 to present on this draft.

Kind regards,

Job

ps. Some may point out that this is a rampant layering violation, to which I
will say: "yes". ;-)

----- Forwarded message from [email-masked] -----

Date: Sun, 12 Mar 2017 14:00:18 -0700
From: [email-masked]
To: [email-masked]
Subject: I-D Action: draft-iops-grow-bgp-session-culling-00.txt

A New Internet-Draft is available from the on-line Internet-Drafts directories.

        Title           : Mitigating Negative Impact of Maintenance through BGP Session Culling
        Authors         : Will Hargrave
                          Matt Griswold
                          Job Snijders
                          Nick Hilliard
	Filename        : draft-iops-grow-bgp-session-culling-00.txt
	Pages           : 9
	Date            : 2017-03-12

Abstract:
   This document outlines an approach to mitigate negative impact on
   networks resulting from maintenance activities.  It includes guidance
   for both IP networks and Internet Exchange Points (IXPs).  The
   approach is to ensure BGP-4 sessions affected by the maintenance are
   forcefully torn down before the actual maintenance activities
   commence.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-iops-grow-bgp-session-culling/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-iops-grow-bgp-session-culling-00

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

----- End forwarded message -----

---

From: Theodore Baschak <[email-masked]>
Date: Sun, 12 Mar 2017 18:29:35 -0500

I can attest to the technical validity of the solution proposed because
this is exactly what we do at the Manitoba Internet Exchange when we do IXP
maintenance. We shutdown the route server sessions for everyone and usually
notice many participants sessions are still up. Then we apply v4 and v6
ACLs to cull BGP sessions between IX participants. About 15 minutes later
there is no traffic flowing across the peering fabric and we are able to do
whatever maintenance is required without causing next-hops to be
unavailable and traffic to not flow. We have used this at least three times
in my memory, and it has worked exactly as described/intended.

Thank you to the authors for writing this BCP Internet-Draft. As an
operator of a relatively young IXP, it is very valuable to have these
documents and best practices out there. We really appreciate them.

Theodore Baschak - AS395089 - Hextet Systems
https://bgp.guru/ - https://hextet.net/
http://mbix.ca/ - http://mbnog.ca/

On Sun, Mar 12, 2017 at 5:16 PM, Job Snijders <[email-masked]> wrote:

> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>

---

From: Alejandro Acosta <[email-masked]>
Date: Mon, 13 Mar 2017 02:07:21 +0100

Excellent idea to write this down.  Congrats to the authors.

What do you think in including also some suggestions when bringing up
the BGP sessions?.  Sometimes it´s good idea to bring them up one by one
or something like that, the idea is to make the device to fill out the
forwarding table, create cache, perform ARP lookups, ND, and so on. To
bring up all the session at once many times is not that good.

Alejandro,

---

From: Gert Doering <[email-masked]>
Date: Mon, 13 Mar 2017 09:18:21 +0100

Hi,

> control-plane in a controlled fashion, before actually breaking the

I like the concept.

Wording-wise, there is room for misunderstanding in the current version,
2.2.1:

2.2.1.  Packet Filter Considerations

   The packet filter should be designed and specified in a way that:

   o  only affect link-local BGP traffic i.e. forming part of the
      control plane of the system described, rather than multihop BGP
      which merely transits

it says "link-local", but what it wants is not "fe80::" but "the prefixes
the intermediate network uses for on-link peering" (plus, maybe, fe80::).

So maybe "only affect *on-link* BGP traffic"?

[..]
> ps. Some may point out that this is a rampant layering violation, to which I

And a useful one :-)

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 13 Mar 2017 09:33:45 +0100 (CET)

> So maybe "only affect *on-link* BGP traffic"?

Good catch. Perhaps "intra on-link subnet BGP traffic" or something like 
that? "link-local" should not be used as a term though, I agree that might 
cause confusion.

Great work writing this down, hope more operators and IXPs implement these 
procedures.

Also, do we know why still so few use BFD on IXPs? Since all other 
mechanisms apart from BFD lacked consensus (there were L2 reporting 
protocol proposals I remember from 10+ years back), it would be great if 
BFD was actually deployed more.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Gert Doering <[email-masked]>
Date: Mon, 13 Mar 2017 09:41:00 +0100

Hi,

> BFD was actually deployed more.

Right.  Speaking for ourselves: until not so long ago, our routers
were just not capable of sustaining 100+ BFD sessions with usefully
short timers - we now have better peering boxes, so maybe it's time
to start talking to peers and spread the word.

That's the other part - knowledge of the peer operators at a typical
IXP is reaching new all-time lows every year, so having something that
will do the right thing on the fabric operators's side is much less work
than negotiating anything exceeding basic BGP setup with 100+ peers...

Gert Doering
       "disgruntled old fart..."
-- 
have you enabled IPv6 on something today...?

---

From: heasley <[email-masked]>
Date: Mon, 13 Mar 2017 16:19:10 +0000

> bring up all the session at once many times is not that good.

I'd expect this to prolong and exacerbate the 'path hunting', while the
min-advert-timer might help to squelch it if all sessions are enabled
at the same time - after the IGP settles, which is automatic in some
impl..

randy, link to path hunting paper?  i can't seem to find it.

---

From: Job Snijders <[email-masked]>
Date: Mon, 13 Mar 2017 18:05:31 +0100

> Excellent idea to write this down.  Congrats to the authors.

Thanks!

> bring up all the session at once many times is not that good.

At this moment I'm inclined to consider such 'staggered gradual
restoration' recommendations out of scope for this BCP. Mainly because I
am not aware of a generic, currently in-use recommendations and the
problem space might be somewhat undefined.

While I do know that staggered restoration was sometimes applied back in
the day (when the majority of backbone routers was cisco 7100-style
devices), and usually arranged manually through phonecalls, I'm not so
sure whether this still is in vogue.

Kind regards,

Job

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 13 Mar 2017 20:38:08 +0000

> problem space might be somewhat undefined.

I'd suggest that ixp operators are able to make this sort of decision by
themselves.

Nick

---

From: <[email-masked]>
Date: Tue, 14 Mar 2017 14:28:56 +0000

> From: GROW [mailto:[email-masked]] On Behalf Of heasley
> 

For the BGP shut, in section 2.1. " Voluntary BGP Session Teardown Recommendations" you could propose or at least reference BGP Graceful shutdown https://tools.ietf.org/html/draft-ietf-grow-bgp-gshut-06
In very short, it initiates the path hunting for the backup BGP path, _before_ the withdraw of the nominal path. Tests have shown that 0 packet loss is achievable (assuming that within the AS, tunneling is used in order to avoid micro-loops during iBGP convergence). But if one is not targeting 0 packet loss, which is typically the case in the Internet ecosystem, there is no requirement for tunneling.

In short, over eBGP, routes to be withdrawned are tagged with a "well known" community, in order to be de-preferred on the receiving side.

Some vendors have automated this. But one may also do it manually using BGP policies.

Regards,
--Bruno

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: <[email-masked]>
Date: Tue, 14 Mar 2017 15:07:32 +0000

> From: Job Snijders [mailto:[email-masked]]
> 
 > On Tue, Mar 14, 2017 at 02:28:56PM +0000, [email-masked] wrote:
 > > For the BGP shut, in section 2.1. " Voluntary BGP Session Teardown
 > > Recommendations" you could propose or at least reference BGP Graceful
 > > shutdown https://tools.ietf.org/html/draft-ietf-grow-bgp-gshut-06
 > > In very short, it initiates the path hunting for the backup BGP path,
 > > _before_ the withdraw of the nominal path. Tests have shown that 0
 > > packet loss is achievable (assuming that within the AS, tunneling is
 > > used in order to avoid micro-loops during iBGP convergence). But if
 > > one is not targeting 0 packet loss, which is typically the case in the
 > > Internet ecosystem, there is no requirement for tunneling.
 > >
 > > In short, over eBGP, routes to be withdrawned are tagged with a "well
 > > known" community, in order to be de-preferred on the receiving side.
 > >
 > I appreciate the effort and thought that has gone into gshut, but I am
 > not aware of actual deployments and as scuh certainly cannot vouch for
 > using this method as a 'best current practise'. it may be a 'future best
 > practise' - but that is not now.

Referencing does not necessarily imply recommending as BCP.

On a side note, I'd be interesting to know why reducing the impact of the maintenance using gshut is not considered as worth it, while it is for culling. Especially since the benefit of the latter is 90 second (and configurable)  while the former is minutes (and not configurable).

Kind regards,
--Bruno

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 15:59:05 +0100

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 16:16:24 +0100

On Tue, Mar 14, 2017 at 03:07:32PM +0000, [email-masked] wrote:
> On a side note, I'd be interesting to know why reducing the impact of
> the maintenance using gshut is not considered as worth it, while it is
> for culling. Especially since the benefit of the latter is 90 second
> (and configurable)  while the former is minutes (and not
> configurable).

well, to be frank: because one is an existing practise, the other isn't?

But I'm not closing the door: BCPs can be enhanced over time, which
makes them a very useful vehicle for such ongoing betterment of
interconnection maintenance strategies. Perhaps when gshut is either a
commonly accepted practise, and maybe has a different document status it
would be good to add it to the toolbox.

Kind regards,

Job

---

From: Gert Doering <[email-masked]>
Date: Tue, 14 Mar 2017 16:41:06 +0100

Hi,

On Tue, Mar 14, 2017 at 03:07:32PM +0000, [email-masked] wrote:
> On a side note, I'd be interesting to know why reducing the impact
> of the maintenance using gshut is not considered as worth it, while
> it is for culling. Especially since the benefit of the latter is
> 90 second (and configurable)  while the former is minutes (and not
> configurable).

How's the IXP operator going to introduce a gshut message into a BGP
session between IXP customer A and IXP customer B?

Gert Doering
        -- NetMaster

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 16:49:10 +0100

> session between IXP customer A and IXP customer B?

an IXP can't, and I am not under the impression that Bruno was
suggestion to do so. I took his comments as applicable to section 2.1

this is why the proposed draft contains two angles: one for IXPs and one
for ISPs, each with their different nuances.

Kind regards,

Job

---

From: Gert Doering <[email-masked]>
Date: Tue, 14 Mar 2017 16:55:25 +0100

Hi,

> for ISPs, each with their different nuances.

Indeed, for a direct ISP-ISP link, and the maintenance being controlled
by one of the peering parties, gshut would be a useful approach (if
it's known that the other party has deployed it).

Since the title of the draft is "session-culling" it feels somewhat
out of scope to go more into detail on gshut, but a reference might
be useful.

Gert Doering
        -- NetMaster

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 14 Mar 2017 14:29:01 -0400

we seem to be having quite the discussion on this document, ought it be a
WG draft ? should we vote to adopt? :)

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 18:36:36 +0000

> WG draft ? should we vote to adopt? :)

I would be in favor of adopting draft-iops-grow-bgp-session-culling as WG
document and continuing work on it as group.

Job

---

From: Gert Doering <[email-masked]>
Date: Tue, 14 Mar 2017 21:01:36 +0100

Hi,

> WG draft ? should we vote to adopt? :)

Yes, please make this a WG draft.  This is good stuff, it should have
stronger backing than "individual submission".

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 14 Mar 2017 20:02:37 +0000

Christopher Morrow wrote:
> we seem to be having quite the discussion on this document, ought it be
> a WG draft ? should we vote to adopt?

I'd say it has a place as a wg doc.

Nick

---

From: Christopher Morrow <[email-masked]>
Date: Tue, 14 Mar 2017 16:11:30 -0400

err, I should have just mailed an adoption call :) I'll do that and note
that nick/gert/job all already voted 'yes'.

---

From: David Farmer <[email-masked]>
Date: Tue, 14 Mar 2017 15:59:46 -0500

> stronger backing than "individual submission".

+1

-- 
===============================================
David Farmer               Email:[email-masked]
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

---

From: Job Snijders <[email-masked]>
Date: Wed, 15 Mar 2017 16:37:00 +0100

On Tue, Mar 14, 2017 at 04:55:25PM +0100, Gert Doering wrote:
> > > > On a side note, I'd be interesting to know why reducing the
> > > > impact of the maintenance using gshut is not considered as worth
> > > > it, while it is for culling. Especially since the benefit of the
> > > > latter is 90 second (and configurable)  while the former is
> > > > minutes (and not configurable).
> > > 
> > > How's the IXP operator going to introduce a gshut message into a
> > > BGP session between IXP customer A and IXP customer B?
> > 
> Indeed, for a direct ISP-ISP link, and the maintenance being
> controlled by one of the peering parties, gshut would be a useful
> approach (if it's known that the other party has deployed it).

I've come to understand that even if the remote party does not support
gshut, at least in one direction there will be benefit (downpreffing of
routes received from the BGP neighbor which is about to be shut down). 

> out of scope to go more into detail on gshut, but a reference might be
> useful.

Perhaps if the gshut draft is revived, a reference indeed is appropiate.
I may have been too soon in my dismissal. Ben Maddison aptly pointed out
that gshut is part of Ben's Current Practices. :-)

Kind regards,

Job

---

From: <[email-masked]>
Date: Wed, 15 Mar 2017 15:52:24 +0000

> From: Job Snijders [mailto:[email-masked]]  > Sent: Wednesday, March 15, 2017 4:37 PM
> 

You are right.
Most network usages are bidirectional, in which case improving one way has probably no benefit.
However, this may still be beneficial when:
- routing is asymmetric
- BGP convergence time is different in both directions (which is typically the case) and we can improve the slowest direction (typically the one receiving a higher number of routes, although hardware/software also plays a significant role)

 > Perhaps if the gshut draft is revived, a reference indeed is appropiate.

We will revive it, probably removing the non-transitive community part. 

Kind regards,
--Bruno

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 20 Mar 2017 16:30:58 -0400

> routes received from the BGP neighbor which is about to be shut down). 

This sort of "draining" behavior is pretty common when you can do it.  And
it's worth noting that most of what gshut needed was just a community people
were willing to use for such a depref.

> that gshut is part of Ben's Current Practices. :-)

Informational references say "go look at this time" not "you require this"
(normative).  You can refer to pretty much anything you want in
informational without impacting publication.

-- Jeff