Subject: [GROW] WGLC: draft-ietf-grow-ops-reqs-for-bgp-error-handling-04

From: Christopher Morrow <[email-masked]>
Date: Mon, 11 Jun 2012 16:21:49 -0400

Hello GROW-WG folk,
Please take this message as the start of a 2 week, ending 6/25/2012
(June 25, 2012) WGLC for the subject draft, link to current version:
  <http://www.ietf.org/internet-drafts/draft-ietf-grow-ops-reqs-for-bgp-error-handling-04.txt>

Abstract:
 "BGP-4 is utilised as a key intra- and inter-Autonomous System routing
   protocol in modern IP networks.  The failure modes as defined by the
   original protocol standards are based on a number of assumptions
   around the impact of session failure.  Numerous incidents both in the
   global Internet routing table and within Service Provider networks
   have been caused by strict handling of a single invalid UPDATE
   message causing large-scale failures in one or more Autonomous
   Systems.

   This memo describes the current use of BGP-4 within Service Provider
   networks, and outlines a set of requirements for further work to
   enhance the mechanisms available to a BGP-4 implementation when
   erroneous data is detected.  Whilst this document does not provide
   specification of any standard, it is intended as an overview of a set
   of enhancements to BGP-4 to improve the protocol's robustness to suit
   its current deployment."

-Chris
co-chair

---

From: Tom Hodgson <[email-masked]>
Date: Wed, 13 Jun 2012 16:24:06 +0100 (BST)

>  <http://www.ietf.org/internet-drafts/draft-ietf-grow-ops-reqs-for-bgp-error-handling-04.txt>

I have reviewed the latest diffs, and the draft in general and agree that 
this represents a useful documentation of the operator requirements and 
should proceed to the next stage.

-- 
Tom Hodgson
[email-masked]

---

From: Gaurab Raj Upadhaya <[email-masked]>
Date: Mon, 18 Jun 2012 01:44:40 +0100

On 6/13/12 4:24 PM, Tom Hodgson wrote:
>> Hello GROW-WG folk, Please take this message as the start of a 2
>> week, ending 6/25/2012 (June 25, 2012) WGLC for the subject
>> draft, link to current version: 

The
latest draft -05 should proceed to the next level.

thanks
 -gaurab

- -- 

http://www.gaurab.org.np/

---

From: "George, Wes" <[email-masked]>
Date: Tue, 19 Jun 2012 11:05:00 -0400

I've read this draft and I think it is useful and ready to proceed.
Two minor comments:

In section 4, it may be useful to specifically note the need for a flag, trap, or other notification method (to the user, rather than to the neighbor) to identify when BGP is in the process of recovering RIB consistency, including which routes are considered inconsistent (if this info is available). While section 6 discusses the need for more information about the state of BGP when it is recovering from these errors, I saw no reference to this specific item. Since this is likely to be a transient state, it is also helpful to log this information to aid in root cause analysis of transient problems.

Also, there are a lot of uses of RFC2119 words in ways that could be interpreted as normative despite the lack of a reference to 2119. It may be useful to either go ahead and make them so (all caps, include RFC2119 boilerplate) or to consider alternate words to avoid confusion. While this is an informational document and not a protocol specification, there are a lot of specific requirements of what we are expecting changes to the implementation to do and not do such that normative words would likely be helpful both to implementers and to those who may follow this draft with one to make protocol changes to address the requirements set forth.

Thanks,

Wes George

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Christopher Morrow
> Sent: Monday, June 11, 2012 4:22 PM
> To: [email-masked]; [email-masked] [email-masked]
> Subject: [GROW] WGLC: draft-ietf-grow-ops-reqs-for-bgp-error-handling-04
>
> Hello GROW-WG folk,
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: "John G. Scudder" <[email-masked]>
Date: Tue, 19 Jun 2012 14:28:42 -0400

It so happens that we are in the middle of an extended WG adoption debate for draft-uttaro-idr-bgp-persistence-01 over at IDR. The problem statement from that draft is kind of buried in S. 1.1:

   BGP Persistence targets the different use case of a catastrophic
   failure when the BGP control plane can remain down for a longer time
   (e.g. hours).

This relates to draft-ietf-grow-ops-reqs-for-bgp-error-handling insofar as draft-uttaro specifically proposes an extension to BGP error handling and (as far as I can tell) its problem statement is not addressed in reqs-for-bgp-error-handling.

While I don't think it would be reasonable to ask reqs-for-bgp-error-handling to address every imaginable error mode, the fact that this is a current active topic in IDR does suggest it might be worth speaking to this one.

--John

P.S.: It's possible I missed a relevant section in reqs-for-bgp-error-handling, in which case please do point it out.

---

From: Pedro Andres Aranda Gutierrez <[email-masked]>
Date: Thu, 21 Jun 2012 07:23:59 +0200

I completely agree with the need to send as much root-cause info as
possible. I think that the user (i.e. admin of the router receiving
wrecked BGP-4 updates) has to be informed. But I also think it would
not do any harm to inform the neighbour causing the crash (à la "you
killed me with this packet!!!"). This information has to be used to
inform (via trap, log, etc.) the admin of the router causing the
damage, so that he takes action ASAP.

My .2 cents,
/Pedro A. Aranda

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 21 Jun 2012 01:48:42 -0400

http://tools.ietf.org/html/draft-ietf-idr-operational-message-00

On Wednesday, June 20, 2012 10:24 PM, Pedro Andres Aranda Gutierrez <> wrote:

>> In section 4, it may be useful to specifically note the need for a
>> flag, trap, or other notification method (to the user, rather than
>> to the neighbor) to identify when BGP is in the process of
>> recovering RIB consistency, including which routes are considered
>> inconsistent (if this info is available). While section 6 discusses
>> the need for more information about the state of BGP when it is
>> recovering from these errors, I saw no reference to this specific
>> item. Since this is likely to be a transient state, it is also
>> helpful to log this information to aid in root cause analysis of
>> transient problems.         
-- 
Jakob Heitz.

---

From: Pedro Andres Aranda Gutierrez <[email-masked]>
Date: Thu, 21 Jun 2012 08:22:48 +0200

OK, then let's state that operational messages (DUMP with malformed
packet) shall be used to notify the neighbor. The ideal scenario for
me is:

1.- Someone sends me a stinking packet and resets my router: I'm able
to notify him back automatically and then, if he doesn't react I can
a) shut the session down and b) ask him why he didn't react.

2.- I'm sending stinking packets: I receive a trap after I sent the
first and I can shut the router/session down and rollback any changes
or investigate where that stinking packet came from, and why it was
rejected.

I think we have still not learnt from the 2009 incidents, where it
took more than 6 hours to bring the situation back to normal and
that's a pity.

/PA

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 21 Jun 2012 08:45:48 +0200

Hi Pedro,

 > I think we have still not learnt from the 2009 incidents,

I think we actually did. The number of IETF documents are trying to 
address this both in IDR and GROW WGs.

It is just that it takes 2-3 years to come up with RFC and another 2-3 
years to implement such RFC these days .... Not to mention product 
marketing/management asking left and right about business case and 
frankly speaking operational quality improvements are not on top of 
their revenue generating list of enhancements ;(

Best,
R.

---

From: Rob Shakir <[email-masked]>
Date: Sat, 23 Jun 2012 22:04:18 +0100

Hi Wes,

Thanks for the comments.

> In section 4, it may be useful to specifically note the need for a flag, trap, or other notification method (to the user, rather than to the neighbor) to identify when BGP is in the process of reco…

This sounds like a good addition -- I'll add a statement regarding this into the -05 version of the draft.

> Also, there are a lot of uses of RFC2119 words in ways that could be interpreted as normative despite the lack of a reference to 2119. It may be useful to either go ahead and make them so (all caps…

I'd be interested in further comments on this -- the intention of the document originally was to frame the problem space, as well as put forward requirements for solutions. If there is value in putting forward these requirements with 2119 wording, then I'll go ahead and do so.

Kind regards,
r.

---

From: Rob Shakir <[email-masked]>
Date: Sat, 23 Jun 2012 22:22:31 +0100

Hi Pedro,

> took more than 6 hours to bring the situation back to normal and

I think we have learnt from these incidents. During these incidents there were significant problems because sessions were torn down, and hence long-lived outages ensued. As is discussed at some length in the draft, this is not desirable for some operators, and in some operational deployments. The requirements outlined in this document are intended to provide means by which an implementation may balance the risk of inconsistency of routing information against preserving network operation. Primarily, therefore the requirements work out how to minimise the impact to overall network integrity during these cases, and then how to give operators much better visibility into what's happening.

It should be noted that where incidents that cause erroneous UPDATE messages occur today, a session MUST be torn down - and therefore a service outage is guaranteed for the subset of routing information carried by this session. Therefore your >6hr period is an outage, rather than a period of degraded service operation.

In both the cases that you have described, it seems like the assumption has been made that the advertising speaker is directly responsible for the erroneous nature of the UPDATE message. If we look at the AS4_PATH issue that we had in the DFZ a few years ago, the speaker that sent the UPDATE message was not responsible for the malformed contents of the attribute. In this case, there is little that the advertising speaker can actually do when one lets him know that he's sending "stinking packets".

If there are specific requirements that you don't feel are addressed in this document (particularly around the logging mechanisms), it would be great to hear about them. I've been working with a relatively large number of operators (many of whom do not not participate directly in the standards development process) and feel that this document is a good representation of these requirements.

Kind regards,
r.

---

From: Shane Amante <[email-masked]>
Date: Mon, 25 Jun 2012 15:15:29 -0600

I've read this draft and support it being published as an Informational RFC.

I do have some comments, which are not blocking, but that I do hope improve the clarity & readability of the draft.

1)  In Section 1.1, the following paragraph hints at the problems with current network architectures, but doesn't really come out and say what they are more directly.
---snip---
Traditional network architectures would deploy an Interior Gateway
Protocol (IGP) to carry infrastructure and customer prefixes, with an
Exterior Gateway Protocol (EGP) such as BGP being utilised to
propagate these prefixes to other Autonomous Systems.  [...]
---snip---
IMO, it would be better if the following section more directly said that scale, in terms of the sheer amount of routing information in BGP, has increased substantially, particularly in places like the core of a Route Reflection hierarchy with order _several_ million paths, and rising.  I wish I could point you at a link to a talk Danny & I gave to GROW and/or IDR at IETF 71 (?), but unfortunately I can't find a copy of the slides on the IETF's Web site.  You can see that it's in the GROW WG Agenda at IETF 71, but the slides link to a different presentation: <http://tools.ietf.org/wg/grow/agenda?item=agenda71.html>

Anyway, IMO, this is (from my vantage point) the single factor that makes outages not only much more visible, but /also/ (just as importantly) take longer to recover from ... (at the end of the day, CPU is finite, there are no "easy" answers wrt using multi-core processors to speed-up convergence of single AFI's, DRAM speeds are not keeping pace, etc.)  This may be useful to mention here to "set the stage", as it were, since throughout the draft there's a continuous (and, good) dialogue of not making convergence times worse or CPU load increase as a result of incorporating better recovery capabilities.  IOW, another benefit you don't really mention anywhere, but probably should, is that /theoretically/ some of these mechanisms may help further promote scalability of BGP.  Obviously, we won't know that until we are evaluating solutions and researchers have time to study them, etc.

Related to the above, there is also the following paragraph:
---snip---
Along with this change in role, the nature of the IP routing
information that is carried has changed.  BGP has become a ubiquitous
means by which service information can be propagated between devices.
---snip---
... it may be better to say that the original scope of BGP wrt *just* exchanging reachability information has expanded to include functions that *were* typically in the domain of 'traditional' provisioning activities, and leave it at that.  (You are obviously familiar with things like BGP A-D + VPLS-LDP vs. VPLS-BGP vs. RFC 4364 ... where the former divorces service discovery [provisioning] from signaling/network reachability, whereas the latter two tightly couple service discovery & network reachability).

2)  In Section 3, wrt the discussion on draft-ietf-idr-optional-transitive, it may be much more compelling to state that practically this solution has (currently) only been defined to be applicable to 3 out of the 29 total BGP path attributes that have been assigned by IANA, at this time.  And, even if you *could* to go farther, then only 10 out of the total 29 BGP path attributes are optional transitive.  So, the short story is: a lot more work in the area of session preservation
of iBGP sessions remains to be done, in particular wrt the other types of mandatory, discretionary, etc. attributes ... (assuming that such a thing is even technically possible with BGPv4, as we know it).

BTW, here's my list of optional, transitive attributes, with an "*" indicating those for which draft-ietf-idr-optional-transitive is applicable, in case you're wondering where I came up with my numbers above.  You don't need to include this in your draft.  I'm just providing it here in case someone wants to double-check my sources.
---snip---
7: Aggregator*
8: Community*
11: DPA - Destination Preference Attr
16: Extended Community*
17: AS4_PATH
18: AS4_Aggregator
22: PMSI-tunnel
23: Tunnel Encapsulation Attribute
25: IPv6 Address Specific Extended Community
128: Attribute Set
---snip---

4)  In Section 4, I can't really discern a strong (?) preference for one of the associated recommendations to recovering RIB consistency.  I *think* what you should say, more clearly, is there is a strong preference toward narrow, targeted refresh mechanisms covering specific routes over those broader solutions (recovering an entire Adj-RIB-In), to reduce the
impact on CPU, DRAM, etc.  Perhaps you could add a sentence at the end of the 4th paragraph, where you discuss draft-zeng-one-time-prefix-orf, along the lines of what I just mentioned above?  The good news is you state this very clearly, but much later, in Section 7.  Perhaps you can pull this sentence from Section 7 up to Section 4, as well?
---snip---
[...] It is recommended that where available, any automatic (or
manual) triggered recovery mechanism behaviour utilises such targeted
means in preference to any whole RIB refresh mechanism (such as
ROUTE-REFRESH).
---snip--- 

5)  Section 5 seems targeted solely at recreating the data structures at one side of a BGP session, (specifically a receiver of a BGP session).  However, who's to say the corruption is not actually in the Adj-RIB-Out of the BGP transmitter?  Thus, transmitting you an Adj-RIB-Out all over again, even "gracefully", is unlikely to do much good.  Perhaps it would be good to consider whether the "Receiving Speaker", in Graceful Restart terminology, should consider 'gracefully' purging and recreating his Adj-RIB-Out data structures, as well, before re-xmit'ing them to the "Restarting Speaker"?

Nits:
1)  From <http://tools.ietf.org/html/rfc4271#section-3.1>:
---snip---
For the purpose of this protocol, a route is defined as a unit of
information that pairs a set of destinations with the attributes of a
path to those destinations.  The set of destinations are systems
whose IP addresses are contained in one IP address prefix that is
carried in the Network Layer Reachability Information (NLRI) field of
an UPDATE message, and the path is the information reported in the
path attributes field of the same UPDATE message.
---snip---
... thus, to be more correct, I think you should scrub the document to say "route" in places where you're saying "prefix".  Some examples include:

http://tools.ietf.org/html/draft-ietf-grow-ops-reqs-for-bgp-error-handling-04#section-1.2
---snip---
o  It is unacceptable within modern deployments of the BGP-4 protocol
   that a single erroneous UPDATE packet affects prefixes that it
   does not carry.  This requirement therefore requires some
---snip---
s/affects prefixes/affects routes/

---snip---
this reset of the BGP-4 session results in interruption to
forwarding packets (by means of withdrawing prefixes installed by
BGP-4 into a device's RIB, and subsequently FIB).  To this end,
---snip---
s/withdrawing prefixes/withdrawing routes/

2)  In Section 2, this may be just wording, but I'm unclear on what you mean in the below where you say: "... handle the error in a manner focused on the NLRI contained within the message".  Are you trying to say something like "constrain the logging and/or ignoring of malformed UPDATE messages specific to the narrowest subset of routes for which malformed path attributes have been received?"  I'm not suggesting my wording is necessarily better, in this case, but perhaps you could look at being a bit more precise below?
---snip---
Where an UPDATE message is considered invalid by a BGP speaker due to
an error within a path attribute that is not the NLRI (where the
definition of NLRI includes reachability information encoded in the
MP_REACH_NLRI and MP_UNREACH_NLRI attributes as specified in
[RFC4760]) it is a requirement of any enhanced error handling
mechanism to handle the error in a manner focused on the NLRI
contained within the message.  
---snip---

3)  Section 2:
---snip---
contained within the message.  Since in this case, the message
received from the remote peer is syntactically valid, it is
considered that such an UPDATE is indicative of erroneous data within
a path attribute.  [...]
---snip---
s/path attribute/path attributes/

4)  In Section 4, there do you really mean "transitive" here or do you instead mean "transient"?
---snip---
It is of particular note for both means of recovering RIB consistency
described that these are effective only when considering transitive
errors within an implementation - for instance, should an RFC ...
---snip---
... also, in the following, do you mean "dynamic" instead of "transitive" in the below?
---snip---
[...] It is not advisable that a transitive
filter and advertisement mechanism is triggered by all error handling
events due to the load this is likely to place on the neighbour
receiving such a request.  Where this BGP speaker is a relatively
---snip---

5)  In Section 7.1, I think you probably want to s/NLRI/route/ in several places, e.g.:
---snip---
... upstream speaker to perform a best-path selection, and re-advertise a
new set of __NLRI__ before the downstream system is able to converge to a
new path.
---snip---

Thanks,

-shane