Subject: [GROW] draft-ietf-grow-ops-reqs-for-bgp-error-handling-04

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 14:56:49 +0000

Rob,

Following find my comments..

Thanks,
                Jim Uttaro

General Comment,

>From a philosophical perspective I agree with the goals of this draft but I do not agree with an approach that maintains a session in the face of a failure in the machinery. This is a bottom up approach which will always be a day late and a dollar short as we continue to patch what it means to be a viable session.. I believe the proper approach to meet your reqs ( and mine too!!! ) is a top down which does not change the BGP behavior but changes the response to that behavior.. The reality of today's BGP fields of use which are many and varied is that the control and the forwarding paths are orthogonal, the state being carried is not always paths as we usually consider them. Examples include RT-C, Flowspec which are used to create PWs and simulate an IGP etc... I do not believe changing the behavior of BGP session viability machinery to accomplish maintaining valid forwarding in the face of a control plane failure is the correct approach

The draft addresses a very important error condition but ( Update Error ).. This should be expanded to other areas where the failure of a session does not indicate that a subset or all of the routing state learned over said session is invalid.. I think the draft should address this directly.. Are the changes here configurable? Can I turn this off if I do not want this behavior for certain topologies, AFs?

Abstract

Can the scope be expanded? There are other failure modes, i.e Timer Expiry which today is not considered a failure mode. In reality what I have seen is that timer expiry occurs due to the fact that BGP threads cannot be serviced in a timely manner.  I think it would be best if we could put it all on the table..

I do think that this draft should bound the solution space. At the minimum the solutions proposed should meet a minimum set of the operators criteria in terms of managing the network, convergence, persistence, churn, forwarding impact etc...

Section 1.1

The following paragraph is based on the premise that the session being "down" results in a large impact.. This is certainly true for today's implementations which use the session ( Control Plane Construct ) to determine the viability of the forwarding state learned over said session. There are cases where this is the session and forwarding are parallel, but in many more cases control and forwarding planes are orthogonal.. I think we need to re-consider this assumption for many of the services BGP is being used for and base the response to error conditions on this reality..

" Both within Internet and multi-service routing architectures, a
   number of BGP sessions propagate a large proportion of the required
   routing information for network operation.  For Internet routing,
   these are typically BGP sessions which propagate the global routing
   table to an AS - failure of these sessions may have a large impact on
   network service, based on a single erroneous update.  In an multi-
   service environment, typical deployments utilise a small number of
   core-facing BGP sessions, typically towards route reflector devices.
   Failure of these sessions may also result in a large impact to
   network operation.  Clearly, the avoidance of conditions requiring
   these sessions to fail is of great utility to any network operator,
  and provides further motivation for the revision of the existing
   behaviour. "

Section 1.2

Bullet 1..

This is a very interesting point.. As you stated

"  Traditional network architectures would deploy an Interior Gateway

   Protocol (IGP) to carry infrastructure and customer prefixes, with an

   Exterior Gateway Protocol (EGP) such as BGP being utilised to

   propagate these prefixes to other Autonomous Systems. "

In this environment where BGP was predominantly used to advertised state between AS domains over dedicated peering points it would makes sense that a malformed update learned from a peer is a fairly good indication that the peer which is originating the update is suspect.. As there are other NHs available it would be prudent to not use the suspect session/forwarding path which in these cases were in parallel.. Maybe "treat as withdraw" is appropriate not sure, although the SP should be able to decide the course of action .

I would ask you to consider that there are two cases here.. The first is when a speaker learns an update from a peer where the NH for the paths in that update is the direct peer ( ASBR, PE ), and the second whereby the update is from a peer where the NH for the paths in the update is not the peer ( RR ). In the former case is there still a case that the original premise holds..The offending egress router should be disconnected from the topology. I don't think this is black and white and operators may want the flexibility of determining the behavior..

Bullet 2

Totally agree.. This is the one of the major requirements driving the BG Persistence Draft..

Bullet 3

Not sure exactly what the intent here is.. I am all for more robust NM and visibility...

Section 2

I think understand where you are coming from for the first set of errors.. Do we have a feel as to why there would be erroneous data? It would seem that whichever speaker created/modified etc.. the attr is experiencing some fundamental issue with the BGP machinery as it is not validating the context??

"Since in this case, the message

   received from the remote peer is syntactically valid, it is

   considered that such an UPDATE is indicative of erroneous data within

   a path attribute."

Section 2.1, 2.1.1, 2.1.2

Hmmm. Too be honest this make me a bit nervous.. This introduces classes of failure modes.. Downstream NM, Ops etc... will have to learn how to support these nuanced messages and intended meanings.. There is one thing certain, when a BGP Session goes down in today's world it merits the immediate attn of operations to figure out what went wrong and to fix it.. Solutions like GR or BGP Persistence do not change the trigger for Operations response..

Section 3

The premise is to modify when a NOTIFICATION message is sent thus mitigating the possibility that the session becomes invalid or to use a "treat as withdraw" for those paths sharing this common attribute.. The issue in mind is the notion of the session as the be all to end all.. We could as easily without any change to BGP use BGP Persistence to maintain the paths except for the ones that have the invalid attribute.. This is the simpler method, has the benefit of not changing BGP, or educating the world on the nuances of the changes etc...

I also do not fully understand "treat as withdraw" does this meant that the peer who has received an update with P1-PN with malformed attr then initiate a withdrawal to all of its peers?  Or simply assume that the paths have been received as a message?  Some sample topologies as to how this works would be a good addition to this section..

Section 4

I made some comments on a solution "Re: [Idr] I-D Action: draft-ietf-idr-error-handling-02.txt" which I think is intended to address some aspects of this reqs doc. It seems that there is a possibility of forwarding loops that can be created and the inability of BGP to recover without operator intervention.

" There are therefore risks of traffic blackholing, due to

   missing routing information, or forwarding loops.  Whilst this is

   deemed an acceptable compromise in the short term, clearly, it is

   suboptimal.  Therefore, a requirement exists to provide mechanisms by

   which a BGP speaker is able to recover the consistency of the Adj-

   RIB-In for a particular neighbour."

I do not think that the above draft which does not recover addresses this req..

I am not in support of solutions which create a scenario where BGP cannot recover without human intervention.

"It is of particular note for both means of recovering RIB consistency

   described that these are effective only when considering transitive

   errors within an implementation - for instance, should an RFC

   interpretation error within an implementation be present, regardless

   of the number of times a specific UPDATE is generated, it is likely

   that this error condition will persist (as it may with the existing

   behaviour defined by [RFC4271])."

I think the difference is that the operator knows that a serious situation is occurring by virtue of the session failing and can take appropriate action to correct. Forwarding can be maintained using Persistence or GR.. So IMO I can know a serious issue exists using existing BGP Machinery and well known procedures and at the same time maintain my forwarding..

Section 5

Why wouldn't we simply let the session fail and then use BGP Persistence or GR ;)

Section 6

Nothing is going to get people's attention like a failed BGP Session.. I can only speak for my org but this is what is actively monitored as the bases for the health of BGP.. Diagnostic, log etc... messages are of high volume and does not get folks attn.. IMO this is a major disadvantage to this approach.. We do not know how convey that something bad has happened to the BGP Machinery..

Section 7

Although I understand the distinction between the different classes of Update Errors, I am not sure that I understand how that translates into how the BGP Machinery is actually being impacted. The increased complexity of "understanding" the type of update error message, it's cause, possible remediation etc.. makes this approach really difficult to wrap ones arms around..

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 22 Jun 2012 17:57:01 +0200

Jim,

> We could as easily without any change to BGP use BGP Persistence to
> maintain the paths except for the ones that have the invalid
> attribute.. This is the simpler method, has the benefit of not
> changing BGP, or educating the world on the nuances of the changes
> etc…
+
> Why wouldn’t we simply let the session fail and then use BGP Persistence
> or GR ;)

Please observe that when the session is down you are not receiving 
withdraws or new best paths for those "good" prefixes (maybe 99% of 
them) which did not have any errors in their respective update messages.

Equating it with persistence proposal is therefor highly incorrect.

> I also do not fully understand “treat as withdraw” does this meant that
> the peer who has received an update with P1-PN with malformed attr then
> initiate a withdrawal to all of its peers?  Or simply assume that the
> paths have been received as a message?  Some sample topologies as to how
> this works would be a good addition to this section..

The speaker reacting on an error which can be addressed by 
"treat-as-withdraw" invalidates locally those prefixes received in the 
update message, runs local best path and as result if no other path is 
found withdraws those prefixes from all peers it has previously sent 
them to.

> I am not in support of solutions which create a scenario where BGP
> cannot recover without human intervention.

I think no one is. But we are - I think - not there yet for the routers 
to automatically fix their bugs, but only automatically signalling them 
the requested action ;(.

 > Nothing is going to get people’s attention like a failed BGP
 > Session..

True statement. But the entire assumption behind treat-as-withdraw is 
that your ops scripts parse the syslog messages indicating the issue to 
NOC with the same red color and buzz as bgp session down. Of course you 
need to rework your ops scripts/alarms for that to happen.

Rgs,
R.

PS.

Note that if the main BGP session is down (like in the persistence case) 
BGP Operational Messages can not any longer be exchanged between peers 
as TCP connection could have been reset (if no multisession is used and 
if we are talking about single SAFI). That just makes the issue worse 
especially when you do not like to have humans intervention.

---

From: Enke Chen <[email-masked]>
Date: Fri, 22 Jun 2012 11:00:13 -0700

Hi, folks:

It might help the discussion to refresh ourselves about several large 
outages in the last few years that prompted the work on the error 
handling requirements and solutions:

    o issue with AS4_PATH that resulted in session resets multiple hops 
away (two separate incidents)
    o session reset triggered by a single route with a new attribute

I remember that Rob had a presentation at the NANOG on the topic.

-- Enke

On 6/22/12 8:57 AM, Robert Raszuk wrote:
>> etc...
>> I also do not fully understand "treat as withdraw" does this meant that
> I think no one is. But we are - I think - not there yet for the 
> routers to automatically fix their bugs, but only automatically 
> signalling them the requested action ;(.
>
> > Nothing is going to get people's attention like a failed BGP
> > Session..
>
> that your ops scripts parse the syslog messages indicating the issue 
> to NOC with the same red color and buzz as bgp session down. Of course 
> you need to rework your ops scripts/alarms for that to happen.
>
> Rgs,
> R.
>
> PS.
>
> Note that if the main BGP session is down (like in the persistence 
> case) BGP Operational Messages can not any longer be exchanged between 
> peers as TCP connection could have been reset (if no multisession is 
> used and if we are talking about single SAFI). That just makes the 
> issue worse especially when you do not like to have humans intervention.
>
>
>
>
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Fri, 22 Jun 2012 18:22:10 +0000

+1..

Jim Uttaro

From: Enke Chen [mailto:[email-masked]]
Sent: Friday, June 22, 2012 2:00 PM
To: [email-masked]
Cc: [email-masked] List; [email-masked]; UTTARO, JAMES; Enke Chen
Subject: Re: [Idr] Fwd: [GROW] draft-ietf-grow-ops-reqs-for-bgp-error-handling-04

Hi, folks:

It might help the discussion to refresh ourselves about several large outages in the last few years that prompted the work on the error handling requirements and solutions:

   o issue with AS4_PATH that resulted in session resets multiple hops away (two separate incidents)
   o session reset triggered by a single route with a new attribute

-- Enke

Why wouldn't we simply let the session fail and then use BGP Persistence
or GR ;)

Please observe that when the session is down you are not receiving withdraws or new best paths for those "good" prefixes (maybe 99% of them) which did not have any errors in their respective update messages.

The speaker reacting on an error which can be addressed by "treat-as-withdraw" invalidates locally those prefixes received in the update message, runs local best path and as result if no other path is found withdraws those prefixes from all peers it has previously sent them to.

I think no one is. But we are - I think - not there yet for the routers to automatically fix their bugs, but only automatically signalling them the requested action ;(.

> Nothing is going to get people's attention like a failed BGP

True statement. But the entire assumption behind treat-as-withdraw is that your ops scripts parse the syslog messages indicating the issue to NOC with the same red color and buzz as bgp session down. Of course you need to rework your ops scripts/alarms for that to happen.

Rgs,
R.

PS.

Note that if the main BGP session is down (like in the persistence case) BGP Operational Messages can not any longer be exchanged between peers as TCP connection could have been reset (if no multisession is used and if we are talking about single SAFI). That just makes the issue worse especially when you do not like to have humans intervention.

Idr mailing list

[email-masked]<mailto:[email-masked]>

---

From: Rob Shakir <[email-masked]>
Date: Sat, 23 Jun 2012 22:33:32 +0100

> I remember that Rob had a presentation at the NANOG on the topic.

Hi Enke,

Thanks for this message. This work is absolutely motivated by incidents in live networks (both those that have publicly been described and those in private network deployment that are not as public).

I have talked to this work a number of times across a number of operator forums:

- NANOG51 / LINX / UKNOF:
	http://www.nanog.org/meetings/nanog51/presentations/Tuesday/shakir-bgp-error-handling_rob-shakir-FINAL2.pdf
	http://www.nanog.org/meetings/nanog51/presentations/Tuesday/bgp_err_hdling.wmv
- Netnod:
	http://rob.sh/files/RJS-Reinforcing_the_Kitchen_Sink-NETNOD-Autumn2011.pdf

The motivation for this work is improving the robustness of real-world networks, where the current error handling behaviour does not match the situation and deployment in which the protocol is deployed. 

As I have voiced in other posts to both the IDR and GROW mailing lists, these requirements put forward means to be able to balance the risk of things not being 100% correct in terms of protocol operation against the threat of complete outages for all routing information carried via a BGP session. If these risks are not acceptable to an operator, then the solutions implemented in answer to these requirements do not need to be enabled - however, at the moment, the protocol is completely constrained in terms of its behaviour -- sessions are torn down regardless of the impact of their failure.

Kind regards,
r.

---

From: Rob Shakir <[email-masked]>
Date: Sat, 23 Jun 2012 23:39:03 +0100

Hi Jim,

Thanks very much for the detailed review of this document.

On 22 Jun 2012, at 15:56, UTTARO, JAMES wrote:

> From a philosophical perspective I agree with the goals of this draft but I do not agree with an approach that maintains a session in the face of a failure in the machinery. This is a bottom up approach which will always be a day late and a dollar short as we continue to patch what it means to be a viable session.. I believe the proper approach to meet your reqs ( and mine too!!! ) is a top down which does not change the BGP behavior but changes the response to that behavior.. The reality of today’s BGP fields of use which are many and varied is that the control and the forwarding paths are orthogonal, the state being carried is not always paths as we usually consider them. Examples include RT-C, Flowspec which are used to create PWs and simulate an IGP etc… I do not believe changing the behavior of BGP session viability machinery to accomplish maintaining valid forwarding in the face of a control plane failure is the correct approach
>  

The scope for this draft was particularly to handle errors that occurred in the DFZ (and in some private networks) where UPDATE messages with malformed contents representing a subset of NLRI carried via a session (1 prefix in numerous cases) resulted in complete failure of these sessions. Particularly, errors where it was a malformed optional transitive attribute that was "tunnelled" across multiple speakers who did not parse it were particularly destructive in the DFZ. Jonathan Oddy, Andy Davidson and myself spent some effort analysing one of these incidents back in 2008/2009 -- http://mailman.nanog.org/pipermail/nanog/2009-January/006816.html. 

One of the complexities in this problem space is that there are multiple underlying philosophies as to how such error behaviour should be handled:

1. Conservative -- any error in messages received from a neighbour are indicative that it is not a viable route to any prefix it advertises. Therefore where error conditions occur, disconnect and remove all routes from the neighbour from the RIB.

2. Balanced risk -- some errors are not indicative of an error on the remote speaker, or are localised to specific NLRI/prefixes that are propagated via this speaker. Therefore take a balanced approach of applying error handling mechanisms to those NLRI, but continue to trust the integrity of the speaker for the remainder of routing information.

Essentially, this draft's scope was to explain the requirements that exist to manage the risk of taking the latter view. At the moment, there is a risk to an operator presented by the first (a malformed UPDATE may break the BGP sessions that run between their PEs and RRs, or those that propagate their routes to the Internet DFZ, and cause a service outage), and there is no option to balance the impact of that risk against the impact of BGP being incorrect for a subset of prefixes propagated via those sessions.

I would expect all solutions implemented in response to these requirements to be optional. If the risk of incorrectness is unacceptable to you/an operator, then you should absolutely not enable any of these mechanisms. In a number of networks that I have operated, designed and architected, I am prepared to accept the risk of incorrectness, as I consider it acceptable when compared to the risk of complete service outages in terms of impact to my customers during such incidents. At the moment, without the work described through the requirements outlined in this draft I do not have the means to make that call...

> I do think that this draft should bound the solution space. At the minimum the solutions proposed should meet a minimum set of the operators criteria in terms of managing the network, convergence, persistence, churn, forwarding impact etc…  

I think there was clear consensus amongst operators with whom I have spoken to work on the problem space of erroneous UPDATEs - particularly in response to observed incidents. The very real risk of expanding the scope of this document even further to handle any error in the BGP protocol is that we spend another few years cataloguing such conditions, whilst doing nothing about the ones that we have identified.

>  Section 1.1
>  
> The following paragraph is based on the premise that the session being “down” results in a large impact.. This is certainly true for today’s implementations which use the session ( Control Plane Construct ) to determine the viability of the forwarding state learned over said session. There are cases where this is the session and forwarding are parallel, but in many more cases control and forwarding planes are orthogonal.. I think we need to re-consider this assumption for many of the services BGP is being used for and base the response to error conditions on this reality..
>  
> “ Both within Internet and multi-service routing architectures, a
>    behaviour. “

I do not understand what the assertion that you are making here is. Please could you explain it to me? The errors that are being discussed relate to where a subset of NLRI are advertised within an erroneous UPDATE message, and the resulting impact of the current protocol behaviour on all other NLRI carried on that session. The cases of IP "transit" sessions, and RR-PE sessions are only examples of cases where there is a large amount of routing information carried over a single session - and hence it is of utility to avoid these sessions failing where they do not necessarily need to based on the impact to overall network service.

This point (to me anyway) seems entirely related to the control-plane -- it points out that an operator has cases where one really wants to keep the impact of errors down to the particular subset of routing information that is affected. This point is entirely in the protocol, rather than implying any behaviour about forwarding (i.e., no implication is made that the NLRI identified as carried in the erroneous UPDATE are installed in the FIB, but rather that all *other* NLRI continue to be installed in the RIB).

I'll work through the points that you have made in this mail -- but I'd like to respond with a specific point here comparing more targeted handling of errors in UPDATE messages with persistence-type approaches. 

- Persistence/hold-up is not acceptable to all operators in all deployments. Particularly, where no liveliness detection mechanism for the next-hop is available (i.e., where it is not possible or practical to determine the NH's reachability in the forwarding plane other than with with the routing protocol itself) then one would not want to accept the risk of running persistence.

- Therefore, an operator has two options for these deployments -- stick with the error handling behaviour that is available in BGP right now, which gives him no flexibility in terms of focusing responses to errors in a manner that is proportional to the actual error; or alternatively, accept some additional complexity and risk such that particular error handling is targeted to the NLRI that are contained in an UPDATE message that is found to be erroneous.

Given there are deployments where I cannot deploy persistence (e.g., towards an Internet network peer on an IXP where deployment of BFD/OAM mechanisms are very limited so BGP really tells me whether the other peer is "there" at all) yet in these cases I do not want to affect all NLRI when one erroneous UPDATE is received then it would seem that I need answers to the requirements in this draft. 

Your e-mail seems to imply that you do not have cases where you will not be able to deploy persistence and that you are accepting of session-level error handling in these cases. Would this be a fair assessment?

Kind regards,
r.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Sun, 24 Jun 2012 02:06:29 +0000

Rob,

	Comments In-Line...

Thanks,
	Jim Uttaro

-----Original Message-----
From: Rob Shakir [mailto:[email-masked]] 
Sent: Saturday, June 23, 2012 6:39 PM
To: UTTARO, JAMES
Cc: [email-masked]'; 'idr wg'
Subject: Re: draft-ietf-grow-ops-reqs-for-bgp-error-handling-04

Hi Jim,

[Jim U>] Got it..

[Jim U>] This is based on the current behavior of BGP.. If the NH is still viable you could allow the session to fall and persist the good state that has already been learned..

Essentially, this draft's scope was to explain the requirements that exist to manage the risk of taking the latter view. At the moment, there is a risk to an operator presented by the first (a malformed UPDATE may break the BGP sessions that run between their PEs and RRs, or those that propagate their routes to the Internet DFZ, and cause a service outage), and there is no option to balance the impact of that risk against the impact of BGP being incorrect for a subset of prefixes propagated via those sessions.
[Jim U>] Yes I have seen this..

I would expect all solutions implemented in response to these requirements to be optional. If the risk of incorrectness is unacceptable to you/an operator, then you should absolutely not enable any of these mechanisms. In a number of networks that I have operated, designed and architected, I am prepared to accept the risk of incorrectness, as I consider it acceptable when compared to the risk of complete service outages in terms of impact to my customers during such incidents. At the moment, without the work described through the requirements outlined in this draft I do not have the means to make that call...
[Jim U>] I do not understand how it is possible to make this configurable on a per session or AS basis..I would think all speakers participating in a routing context would have to adhere to the same rules for a consistent view across domains.. In my reading of the IDR draft it seems that it would be a MUST.. Maybe I should not be considering that IDR draft as the actual realization of the reqs..

I think there was clear consensus amongst operators with whom I have spoken to work on the problem space of erroneous UPDATEs - particularly in response to observed incidents. The very real risk of expanding the scope of this document even further to handle any error in the BGP protocol is that we spend another few years cataloguing such conditions, whilst doing nothing about the ones that we have identified.
[Jim U>] No doubt.. But  I have seen other errors that relate to overloading BGP, topological isolation etc... that have created huge outages in my network.. So, My thinking is that we should try to consider all of the challenges the protocol faces in terms of erroneous error conditions. 

I do not understand what the assertion that you are making here is. Please could you explain it to me? The errors that are being discussed relate to where a subset of NLRI are advertised within an erroneous UPDATE message, and the resulting impact of the current protocol behaviour on all other NLRI carried on that session. The cases of IP "transit" sessions, and RR-PE sessions are only examples of cases where there is a large amount of routing information carried over a single session - and hence it is of utility to avoid these sessions failing where they do not necessarily need to based on the impact to overall network service.
[Jim U>] The solution space here seems explicitly targeted to the internet IPV4 AF. Not sure if there is a dependency here on the control/forwarding planes being in parallel. I believe we need to consider all AFs that BGP is used for.. As the code that would be developed would be applicable to these AF also ( I presume ). So as an example would be RT-C, is this solution applicable I don't know I am simply asking.. If we want to develop this and have it specific to the internet use case lets state that clearly. If not, then let's consider the other applications BGP supports..

This point (to me anyway) seems entirely related to the control-plane -- it points out that an operator has cases where one really wants to keep the impact of errors down to the particular subset of routing information that is affected. This point is entirely in the protocol, rather than implying any behaviour about forwarding (i.e., no implication is made that the NLRI identified as carried in the erroneous UPDATE are installed in the FIB, but rather that all *other* NLRI continue to be installed in the RIB).
[Jim U>] See above.. My point is how can we ensure reliability across AFs...There is no doubt that mal-formed updates are an issue I just do not know how they affect other AFs, in those case it may be more appropriate to tear down and persist instead. Can we address these other use cases?

- Persistence/hold-up is not acceptable to all operators in all deployments. Particularly, where no liveliness detection mechanism for the next-hop is available (i.e., where it is not possible or practical to determine the NH's reachability in the forwarding plane other than with with the routing protocol itself) then one would not want to accept the risk of running persistence.
[Jim U>] Agreed.. Persistence requires that the NH be viable. 

- Therefore, an operator has two options for these deployments -- stick with the error handling behaviour that is available in BGP right now, which gives him no flexibility in terms of focusing responses to errors in a manner that is proportional to the actual error; or alternatively, accept some additional complexity and risk such that particular error handling is targeted to the NLRI that are contained in an UPDATE message that is found to be erroneous.
[Jim U>] My concern here is that the entire support structure is based on session viability, the expectation here is that operators need to develop new approaches to understanding if there is a routing anomaly in their topology and develop tools/procedures which are different.

[Jim U>] Agreed.. 

Your e-mail seems to imply that you do not have cases where you will not be able to deploy persistence and that you are accepting of session-level error handling in these cases. Would this be a fair assessment?
[Jim U>] My work primarily revolves around the following BGP use cases VPN ( L2/L3 ), 3107, Multi-Cast, RT-C, Flowspec not Internet.. In these environments I would prefer that the session fail, persistence is activated, and well known procedures are used to correct the problem. I am in no way saying that the use case described here for the internet application is not viable or the correct approach.. But the fact remains that the code base will be used across all AFs and this solution requires different OpS/Troubleshooting knowledge and support. I would prefer consistency in the behavior.. 

Kind regards,
r.

---

From: Rob Shakir <[email-masked]>
Date: Sun, 24 Jun 2012 10:09:46 +0100

Hi Jim,

Further comments in-line as [rjs].

> [Jim U>] This is based on the current behavior of BGP.. If the NH is still viable you could allow the session to fall and persist the good state that has already been learned..

[rjs]: Absolutely, this is the current behaviour. The problem with taking a whole session down in this case is that you now take a risk of inconsistency for all NLRI across that session for the duration that you hold onto the learned NLRI. If one avoids being in the situation where the session is down (e.g., by applying treat-as-withdraw behaviour in cases where one can determine the NLRI) then all other NLRI on the session continue to be updated as they need to be. It is only the NLRI that were included in the erroneous UPDATE that may be affected for looping/black-holing.

> [Jim U>] I do not understand how it is possible to make this configurable on a per session or AS basis..I would think all speakers participating in a routing context would have to adhere to the sam…

[rjs]: The IDR draft is the solution for some of the requirements -- particularly those described in Section 3 of the GROW draft.

[rjs]: I do not see why this behaviour needs to be consistent across domains?

[rjs]: Essentially, if I receive an invalid UPDATE message, and apply treat-as-withdraw, if the advertising speaker did not know that this was erroneous then I end up with a different view of what is in the RIB than the advertising speaker does. If this was a prefix I had no other route to, then I may black-hole, if it was one where it was a more-specific of some larger prefix, then we end up with the potential for loops.

[rjs]: If I am prepared to accept the black-holing or loops for the NLRI in the erroneous UPDATE as a risk, in favour of keeping the remaining NLRI working (and being updated/withdrawn if they change), then this is a local decision and I do not need to imply any behaviour of the neighbouring domains.

> [Jim U>] No doubt.. But  I have seen other errors that relate to overloading BGP, topological isolation etc... that have created huge outages in my network.. So, My thinking is that we should try t…

[rjs] Right, perhaps a better title for the draft would be "Operational Requirements for Enhanced Error Handling for UPDATE Messages in BGP-4" -- given that there were incidents that affected networks that particularly were focused on the errors in UPDATEs, it seemed logical that this was the place to start for this work. 

> [Jim U>] The solution space here seems explicitly targeted to the internet IPV4 AF. Not sure if there is a dependency here on the control/forwarding planes being in parallel. I believe we need to c…

[rjs] I'd say that it's not just applicable to IPv[46] in the Internet - but to numerous AFIs (there is a definite use-case for these solutions in L3VPN environments for instance). I am not saying that this is applicable or desirable to be turned on for all AFIs -- but it seems to me that this is a per-operator, per-deployment decision, not a per-AFI one. For instance, if we get an RTC UPDATE that is malformed, an operator may not want to tear down a session if it also carries other AFIs (e.g., VPNv[46] also) - in that case, the operator may want to treat this UPDATE as withdrawing the {as, route-target} NLRI (consider that we have no *standardised* multi-session mechanism yet, and there are potential scaling impacts of multiple sessions).

> [Jim U>] See above.. My point is how can we ensure reliability across AFs...There is no doubt that mal-formed updates are an issue I just do not know how they affect other AFs, in those case it may…

[rjs]: The consideration (that I see) that is missing from the draft w.r.t this point seems to be more "what would break if one utilises treat-as-withdraw and this is not {IPv[46],VPNv[46]} etc. Is this what you feel needs to be addressed?

[rjs]: I think the general though process of:
	1/ In some cases, we want avoid affecting all NLRI based on an error in a subset of the received NLRI.
	2/ In this case, we may need to recover from this inconsistency.
	3/ In some cases, we may want to give a session a chance to restart where we could not handle the error gracefully.
	4/ When errors occur on these sessions, there is great operational benefit of flagging them more explicitly.

[rjs]: is applicable across all AFIs almost. The distinction is really whether one considers that 1/ is a valid behaviour/risk for an AFI? 

> [Jim U>] My concern here is that the entire support structure is based on session viability, the expectation here is that operators need to develop new approaches to understanding if there is a rou…

[rjs]: Yes, this is true. We introduce a more complex failure mode where a subset of prefixes are affected. *Any* new mechanism changing the tear-down and flush-the-RIB behaviour of BGP will need some different approaches to be developed. The intention of §6 and §7 of the draft is to highlight where things do become more complex and define requirements for ensuring that the right toolset exists around this behaviour.

[rjs]: Simplifying networks/removing complexity should always be a goal -- but we need to understand where simple behaviour does not provide the levers to limit the impact of errors occurring, and balance the complexity of introducing new mechanisms against the benefit to our network's operation. 

[rjs]: I would like to think that the document highlights where complexity is being introduced -- and highlights the impact of turning some of these knobs on. If the complexity is not acceptable within a deployment, these mechanisms should not be deployed. However, where it is, we don't have the knobs to do anything today! If the document doesn't address this to your satisfaction, please let me know, and I'd be happy to make sure that the complexity is further highlighted.

> [Jim U>] My work primarily revolves around the following BGP use cases VPN ( L2/L3 ), 3107, Multi-Cast, RT-C, Flowspec not Internet.. In these environments I would prefer that the session fail, per…

[rjs]: I am involved in multiple network deployments across the company in which I work, both public and private. I understand that there are different requirements across different networks, and in different deployment cases. 

[rjs]: The problem of any mechanism is that it *may* be available for places where it is not applicable. We could analyse persistence and say that for Internet deployments, it has the potential to be harmful -- a device that has failed will continue to look like a valid path when it might not be, where a network in the Internet DFZ is likely to have a number of alternate paths that could be used. In my view, in this draft, we're just taking the "lowest common denominator" approach - where we say, "how can we build mechanisms that could be applied across all AFIs?". I think in all the use cases you mentioned, there are deployments where these mechanisms are applicable. In all cases, one valid behaviour would be that we keep the subset of NLRI that were not affected from a neighbour, but do not use those that are not necessarily trustworthy because they were linked to an invalid message in the protocol. Of course, operationally, you may not want to do this, based on the risk of inconsistency and the impact of failure. The key point here is that my network deployment may differ to yours, and its up to both of us to decide for our networks, but we both need the tools to be able to make these choices. The problem with a consistent approach is that I don't see how we can have a consistent approach that isn't based on the lowest common denominator AFI - and even then, our logic may differ in terms of where we want to turn it on.

Kind regards,
r.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Mon, 25 Jun 2012 22:47:35 +0000

Rob,

	Comments In-Line..

Thanks,
	Jim Uttaro

-----Original Message-----
From: Rob Shakir [mailto:[email-masked]] 
Sent: Sunday, June 24, 2012 5:10 AM
To: UTTARO, JAMES
Cc: [email-masked]'; 'idr wg'
Subject: Re: draft-ietf-grow-ops-reqs-for-bgp-error-handling-04

Hi Jim,

>>[Jim U>] The assumption being that the error was caused by an upstream speaker and is therefore not truly indicative of an issue over the session where the error manifests itself. This seems to make sense in the IPV4 case. I am still a bit concerned as I do not understand how the following is addressed.

- There is no way of knowing if the adjacent peer is the speaker that is actually responsible for the malformed attr or is coming from an upstream speaker. I can think of no way of knowing this.. Can it be inferred from the notion that an error is of the syntactic or semantic variety?
- There seems to be no threshold when the session is actually taking out of service. It would seem that some number of these type of errors would indicate a major issue is taking place and should be addressed by severing the speaker that is advertising paths with the malformed attr into the topology. A large number of these error will create a large number of withdrawn messages being generated from many peers. What are your thoughts on how this should be addressed?

>>[Jim U>] Can you explain this

>>[Jim U>] Yes.. I am not sure I like the notion of forwarding loops especially for large flows.. 

>>[Jim U>] I guess what I meant was the other paths that are considered good would be treated differently.. So in an environment where only paths with the mal-formed attr are affected by this error condition as opposed to an environment where all paths are affected ( withdrawn ) would create a inconsistent view of the "good" paths across AS domains.. So not so much the "bad" paths but the "good" paths and how they may be treated differently..

>>[Jim U>] Agreed

>>[Jim U>] Quite honestly AFs such as RT-C, Flowspec, etc... where the info being propagated is more akin to "configuration" not path info should persist regardless of the session.. This goes to the heart of the discussion of BGP is used for many fields of use that require persistence. It is not only paths that use BGP for dissemination.. I would prefer that this solution is limited to AFs that disseminate reachability/path info not configuration info..
>> 

>>[Jim U>] We need to look at AFs and decide what is the behavior required for the content being disseminated.. 
> 

Kind regards,
r.

---

From: Rob Shakir <[email-masked]>
Date: Thu, 28 Jun 2012 09:48:49 +0100

Hi Jim,

Apologies for the delay in replying to this message. Further discussion in-line marked [rjs].

> - There is no way of knowing if the adjacent peer is the speaker that is actually responsible for the malformed attr or is coming from an upstream speaker. I can think of no way of knowing this.. C…

[rjs]: This is true, only where we have a mechanism such as the partial bit in the optional transitive attribute can we infer that the directly attached neighbour did not look at the session. What the semantic and critical errors that are called out in the draft relate to is the impact of the error on the resulting UPDATE message, rather than the direct neighbour being responsible for it.

> - There seems to be no threshold when the session is actually taking out of service. It would seem that some number of these type of errors would indicate a major issue is taking place and should b…

[rjs]: This was something that has been discussed on the list previously. There are two key questions in this space:

- Do you expect errors that are indicative of a whole box failure, that are not related to a large change on the device (e.g., code upgrade) that affect all prefixes in a manner that the UPDATE message is formed well enough to extract the NLRI?

- Is the state of reaching all prefixes withdrawn (with UPDATEs withdrawing all NLRI being sent to all neighbours) an acceptable state? I think there is (of course) a scaling impact of such UPDATES being transmitted and parsed to all downstream neighbours, but the impact of such an event is really dependent on assuming that large proportions of the UPDATEs generated become erroneous.

I don't think that I can categorically state that the answer to the former is no, but I am not aware of the case. In my view, larger scale issues on the BGP speaker (e.g., things that affect memory integrity etc.) result in failures that produce output that is not well enough formed to fall within the "semantic" errors described in the draft. I would (of course) welcome implementor and tester's feed back on this point.

>>> [Jim U>] Can you explain this

[rjs]: See later point about "good"/"bad" paths. 

>>> [Jim U>] Yes.. I am not sure I like the notion of forwarding loops especially for large flows.. 

[rjs]: The potential for loops exists in some specific scenarios I think -- especially those where there is a covering aggregate advertised to a speaker, and a more specific that advertised within that aggregate. If this is the case, then in some cases, rather than forwarding back to the device advertising the more specific (i.e., the one that was withdrawn). I think the below example shows something like this - if 10.0.0.0/24 is advertised from C to B, then A forwards packets destined to 10.0.0.0/24, during the time that this prefix is withdrawn, then this will loop. Now, I think that this is a feature of this topology anyway, since where B-C is down, then there will be loops for 10/8 in B-D.

                         <-- 10.0.0.0/8 --
[ A ] --0.0.0.0/0--> [ B ] ---0.0.0.0/0---> [ D ]
                      |
                  10.0.0.0/24
                      |
                    [ C ]

[rjs]: There is then a discussion as to whether one would actually expect such topologies to occur in practical terms. Really, I'd rather expect that there are blackholes (e.g., I only had one path to A, and it got withdrawn, if anyone forwards me packets destined for A, then I drop them) or (more likely in an Internet DFZ perspective) I converge to an alternate path I had to that NLRI.

[rjs]: The reason that this is highlighted in the text is that introducing behaviour into the protocol such that loops may occur is obviously a compromise to protocol correctness, that may be a compromise to overall network forwarding integrity. It's important that this risk is understood, and balanced against the wider impact of session tear down. 

>>> [Jim U>] I guess what I meant was the other paths that are considered good would be treated differently.. So in an environment where only paths with the mal-formed attr are affected by this error…

[rjs]: I'm not sure I fully understand here:
	- Today: UPDATE is received from element A and found to be erroneous - session is reset, downstreams do not see any paths where A was the best-path in the RIB. 
	- With this draft: UPDATE is received from element A, found to be erroneous, downstreams still see all other paths where A is the best-path in the RIB.

[rjs]: I'm not sure that this is so much inconsistency of what the "good" paths look like - both the receiving and downstream elements still consider A's paths as valid, other than the ones that were included in the erroneous UPDATE. In both cases, the NLRI contained in the erroneous UPDATE is also not propagated downstream (session reset, or treat-as-withdraw stops the further propagation).

>>> [Jim U>] Quite honestly AFs such as RT-C, Flowspec, etc... where the info being propagated is more akin to "configuration" not path info should persist regardless of the session.. This goes to th…

[rjs]: The persistence discussion is a further optimisation over this work I feel, it addresses (as you correctly said before) more failure cases. In the case that one UPDATE containing modifications to this configuration information is invalid, is it worth making the rest of it "stale" (and not able to be updated)? I think that in the case, you also want to keep as much of the RIB/config info up-to-date as possible, therefore targeting the error handling mechanism to the contained NLRI still seems advantageous.

[rjs]: Now, the question may be whether treat-as-withdraw is suitable in these cases -- is it better to remove the flow specification, or RT from those installed, or keep it and know that it might be stale? I'd be interested to hear your thoughts here.

[rjs]: On the point of addressing this per-AF, perhaps the text to add to the draft is that behaviour such as treat-as-withdraw must (MUST?) be configurable on a per-AFI basis? The problem with stating something like this, is what does one do when there is no multi-session, and it is disabled for one AFI, yet enabled for another? 

Thanks,
r.