Subject: [GROW] I-D Action: draft-ietf-grow-diverse-bgp-path-dist-05.txt

From: [email-masked]
Date: Thu, 15 Sep 2011 06:58:18 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Global Routing Operations Working Group of the IETF.

	Title           : Distribution of diverse BGP paths.
	Author(s)       : Robert Raszuk
                          Rex Fernando
                          Keyur Patel
                          Danny McPherson
                          Kenji Kumaki
	Filename        : draft-ietf-grow-diverse-bgp-path-dist-05.txt
	Pages           : 22
	Date            : 2011-09-15

   The BGP4 protocol specifies the selection and propagation of a single
   best path for each prefix.  As defined today BGP has no mechanisms to
   distribute paths other then best path between its speakers.  This
   behaviour results in number of disadvantages for new applications and
   services.

   This document presents an alternative mechanism for solving the
   problem based on the concept of parallel route reflector planes.
   Such planes can be build in parallel or they can co-exit on the
   current route reflection platforms.  Document also compares existing
   solutions and proposed ideas that enable distribution of more paths
   than just the best path.

   This proposal does not specify any changes to the BGP protocol
   definition.  It does not require upgrades to provider edge or core
   routers nor does it need network wide upgrades.  The authors believe
   that the GROW WG would be the best place for this work.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-grow-diverse-bgp-path-dist-05.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-grow-diverse-bgp-path-dist-05.txt

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 15 Sep 2011 16:08:12 +0200

Hi,

Authors of draft-ietf-grow-diverse-bgp-path-dist-05 would like to start 
WGLC. The current version 5 of the document has been stable for last two 
revisions. One shipping commercial implementation report has been 
documented in draft-mynam-grow-diverse-path-impl.

Many thx,
R.

-------- Original Message --------
Subject: [GROW] I-D Action: draft-ietf-grow-diverse-bgp-path-dist-05.txt
Date: Thu, 15 Sep 2011 06:58:18 -0700
From: [email-masked]
To: [email-masked]
CC: [email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts 
directories. This draft is a work item of the Global Routing Operations 
Working Group of the IETF.

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-grow-diverse-bgp-path-dist-05.txt
_______________________________________________
GROW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/grow

---

From: "George, Wesley" <[email-masked]>
Date: Wed, 21 Sep 2011 16:22:16 -0400

I've read this draft. I'll note that I'm already in the acknowledgements section, but I can't find an on-list review that I wrote on this draft prior to this point unless I sent it privately to the authors, in which case I apologize if I'm retreading old discussion.
A nit regarding the acknowledgement - my first name is Wes, not George.

Some Nits
Abstract: %s/build/built in parallel, /co-exit/co-exist
Now that the draft is a WG doc, "..the authors believe..." (last sentence) can probably be removed (from intro section as well).

Intro: "other then best path" - suggest rewording to something like "secondary and tertiary paths" or "alternate paths which are not considered best path"

2. %s/"reduction of time of reachability restoration"/faster reachability restoration

Other stuff:
2.1 - when discussing overhead and scale concerns for add paths, perhaps a citation to 4984 would be appropriate? I've made similar comments to the SIDR folks, and I think generally anything that adds a non-trivial amount of impact to the growth curve of the routing system needs to consider this. Also, why is this citing implementation data from 2009? Is there now an implementation that supports add-paths, or is this just a holdover from earlier versions of the draft? If there is now an implementation, it would be good to revisit this section.

4. This asserts that no code changes are necessary to RR clients. I'm not sure I totally agree with that... If the idea is to have a primary (best) RR and then N additional paths, the general assumption is that the N, N1, ... RRs are carrying routes that are less and less preferred. How does this system avoid the same sort of inconsistency of best path choice among different routers in the network if there is no way to identify those paths as secondary? I think you need some way to determine if the alternate routes are intended to be ECMP routes or backup routes...
You may be able to cover this without code changes by using alternate configurations of other BGP preference indicators (MED, Localpref, metric, etc), perhaps with inbound route policy on the client or outbound on the RR, but since things like metric may be different based on where something is in the network, that may lead to inconsistency if used by itself. Even then, the draft doesn't discuss how this should be managed.

4.1 Also, there's a definite scaling consideration on the RR clients that isn't really discussed here - they are now going to be storing some number of additional routes and paths that is linearly related to the number of additional planes that are implemented. The addition of more RR sessions that presumably carry a portion of the full routing table now drives a non-trivial increase in memory footprint and processing overhead (and potentially convergence time for slower boxes). In the simplest case of 2 primary route-reflectors (for diversity), and 1 2nd-best path RR, you've added one session. If you want to carry a 3rd-best RR or have redundant 2nd-best RRs, you've added 4 sessions. It's fair to say that after a certain number of alternate paths, you start having less routes because there are only so many alternative exits, but otherwise there is a potentially large problem even if it's not quite as bad as addpaths.
I might recommend that you do some analysis of the routing table to know where this threshold makes a difference, based on how many alternate paths an average route carries. In addition to being a scaling consideration, it also helps to inform what value of N becomes diminishing returns because most networks don't have that many backup paths. I envision this being something like "80% of routes have 4 or less paths, so moving beyond 4 planes may add overhead without much benefit..."

Further scaling considerations occur in the core if the P routers must know about both the primary and secondary paths (and therefore peer/mesh with both R1 and R1'). You may be able to draw the conclusion that this is better than full mesh (5.1 #3) and therefore superior from a scaling perspective, but this at least needs to be discussed. It may be that the only way to avoid that particular issue is to operate with a BGP-free core...

It may be appropriate to add a separate scaling considerations discussion to your deployment considerations (section 6) to discuss some of the above.

There may be additional operational considerations from the perspective of route analysis - if you have either a homebuilt or off the shelf set of software that does route analysis for the purpose of event root-cause analysis, anomaly detection, capacity planning/failure analysis, etc, it has to be aware of these additional planes such that it returns the proper response when evaluating the routing table to determine what the expected behavior should be in the real network. This is especially important when it uses the table to determine how traffic will reroute during different failure scenarios. These tools may act like a participant in the mesh rather than a client in order to get a pure view of the table, and that may lead to undesired results if the multiple planes aren't taken into account. There may also be considerations for looking glass implementations and the actual information that is visible on the RRs and RR clients as the result of standard BGP show commands to aid in troubleshooting and verification.

Thanks,

Wes George

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, September 15, 2011 9:58 AM
To: [email-masked]
Cc: [email-masked]
Subject: [GROW] I-D Action: draft-ietf-grow-diverse-bgp-path-dist-05.txt

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 21 Sep 2011 23:24:39 +0200

Hello Wes,

> I've read this draft. I'll note that I'm already in the
> acknowledgements section, but I can't find an on-list review that I
> wrote on this draft prior to this point unless I sent it privately to
> the authors, in which case I apologize if I'm retreading old
> discussion.

That was based on your review and off-list mail of June 23rd 2010.

> A nit regarding the acknowledgement - my first name is

Apologies for that. Will be fixed.

> Some Nits Abstract: %s/build/built in parallel, /co-exit/co-exist Now
> that the draft is a WG doc, "..the authors believe..." (last
> sentence) can probably be removed (from intro section as well).
>
> Intro: "other then best path" - suggest rewording to something like
> "secondary and tertiary paths" or "alternate paths which are not
> considered best path"
>

Will fix.

> Other stuff: 2.1 - when discussing overhead and scale concerns for
> add paths, perhaps a citation to 4984 would be appropriate?

I would prefer not to mix the growing internet scale concerns from some 
of the operational practices/configuration based based scale concerns.

> I've made
> similar comments to the SIDR folks, and I think generally anything
> that adds a non-trivial amount of impact to the growth curve of the
> routing system needs to consider this.

I think there is substantial difference for local vs global size 
increase of the routing system. Here in this work all concerns are 
regarding to the local one.

> Also, why is this citing
> implementation data from 2009? Is there now an implementation that
> supports add-paths, or is this just a holdover from earlier versions
> of the draft? If there is now an implementation, it would be good to
> revisit this section.

The point to cite this was to compare with original 2002 time when 
add-paths was defined to indicate how long it took for implementations 
to finally start looking at this. I will reword this section to better 
reflect the intention and avoid using absolute dates.

> 4. This asserts that no code changes are necessary to RR clients. I'm
> not sure I totally agree with that... If the idea is to have a
> primary (best) RR and then N additional paths, the general assumption
> is that the N, N1, ... RRs are carrying routes that are less and less
> preferred. How does this system avoid the same sort of inconsistency
> of best path choice among different routers in the network if there
> is no way to identify those paths as secondary? I think you need some
> way to determine if the alternate routes are intended to be ECMP
> routes or backup routes... You may be able to cover this without code
> changes by using alternate configurations of other BGP preference
> indicators (MED, Localpref, metric, etc), perhaps with inbound route
> policy on the client or outbound on the RR, but since things like
> metric may be different based on where something is in the network,
> that may lead to inconsistency if used by itself. Even then, the
> draft doesn't discuss how this should be managed.

I stand by the claim that no code change is needed on clients. Moreover 
no even additional policy change is required either.

The best way to illustrate this is to compare presence of additional BGP 
paths on the clients in the scenario where clients are interconnected 
with full IBGP mesh or would get all paths with add-path. In neither 
case there is a notion of RR telling client which path is best or which 
is second best .. and there is number of good reasons for that (one is 
that for RR numbering paths can be different then for client, the other 
one is that when we would withdraw any path advertised and ordered we 
would need to re advertise with new order all remaining paths - that 
amount of churn is non negligible).

Each client's BGP best path is capable of making safe (loop free) 
autonomous choice of paths in PIC/fast connectivity restoration/ibgp 
multipath cases.

> 4.1 Also, there's a definite scaling consideration on the RR clients
> that isn't really discussed here - they are now going to be storing
> some number of additional routes and paths that is linearly related
> to the number of additional planes that are implemented. The addition
> of more RR sessions that presumably carry a portion of the full
> routing table now drives a non-trivial increase in memory footprint
> and processing overhead (and potentially convergence time for slower
> boxes). In the simplest case of 2 primary route-reflectors (for
> diversity), and 1 2nd-best path RR, you've added one session. If you
> want to carry a 3rd-best RR or have redundant 2nd-best RRs, you've
> added 4 sessions. It's fair to say that after a certain number of
> alternate paths, you start having less routes because there are only
> so many alternative exits, but otherwise there is a potentially large
> problem even if it's not quite as bad as addpaths. I might recommend
> that you do some analysis of the routing table to know where this
> threshold makes a difference, based on how many alternate paths an
> average route carries. In addition to being a scaling consideration,
> it also helps to inform what value of N becomes diminishing returns
> because most networks don't have that many backup paths. I envision
> this being something like "80% of routes have 4 or less paths, so
> moving beyond 4 planes may add overhead without much benefit..."

It is absolutely correct to say that more paths client carries the more 
CPU cycles and memory will be used to process and store them.

However there is one observation to be made ... in 99% of cases I have 
seen for distributing more then best path intra-domain the sufficient 
number of paths per net on each client is 2.

Moreover this is configurable by the operator modulo RR support 
capabilities for more planes then 2.

IMHO cost of bringing additional paths for control plane is quite well 
understood today. Moreover it is quite implementation dependent. Some 
implementation may use X bytes per path while the other one Y bytes to 
store the same path. I think some separate BGP scaling document (even as 
BCP) may be equally useful for any technique to advertise more then best 
path. I would prefer to keep this outside of the solutions work on how 
to advertise and distribute those additional paths.

> Further scaling considerations occur in the core if the P routers
> must know about both the primary and secondary paths (and therefore
> peer/mesh with both R1 and R1'). You may be able to draw the
> conclusion that this is better than full mesh (5.1 #3) and therefore
> superior from a scaling perspective, but this at least needs to be
> discussed. It may be that the only way to avoid that particular issue
> is to operate with a BGP-free core...
 >
 > It may be appropriate to add a separate scaling considerations
 > discussion to your deployment considerations (section 6) to discuss
 > some of the above.

I agree 100% .. but as stated above I do not find this specific to 
diverse-path. It seems a general issue and I would highly encourage 
someone to take a stub to document this in IETF/IDR/GROW or maybe at 
Nanog community repository.

> There may be additional operational considerations from the
> perspective of route analysis - if you have either a homebuilt or off
> the shelf set of software that does route analysis for the purpose of
> event root-cause analysis, anomaly detection, capacity
> planning/failure analysis, etc, it has to be aware of these
> additional planes such that it returns the proper response when
> evaluating the routing table to determine what the expected behavior
> should be in the real network. This is especially important when it
> uses the table to determine how traffic will reroute during different
> failure scenarios. These tools may act like a participant in the mesh
> rather than a client in order to get a pure view of the table, and
> that may lead to undesired results if the multiple planes aren't
> taken into account. There may also be considerations for looking
> glass implementations and the actual information that is visible on
> the RRs and RR clients as the result of standard BGP show commands
> to aid in troubleshooting and verification.

Very good point. Two comments on this ..

- As to the impact to the tools I am less worried as presence of 
additional paths can be a fact today as already mentioned with full mesh 
or as used by some operator's by playing with adjusting different weight 
values of pair of RRs on a per net basis.

- The use of "planes" in the draft is more of a conceptual nature. In 
practice all paths are still kept in the single table where normal best 
path is calculated. That means that tools like looking glass should not 
observe any changes nor impact.

Once more time many thx for your review.
Robert.

---

From: "George, Wesley" <[email-masked]>
Date: Thu, 22 Sep 2011 12:42:16 -0400

-----Original Message-----
From: Robert Raszuk [mailto:[email-masked]]
Sent: Wednesday, September 21, 2011 5:25 PM
To: George, Wesley
Cc: [email-masked]
Subject: Re: [GROW] I-D Action: draft-ietf-grow-diverse-bgp-path-dist-05.txt

Hello Wes,

WEG] Understand, but I'm not sure that it's so easy to separate the two. You'll find me saying the same thing to anyone suggesting a change that has the net effect of significantly increasing the burn rate for memory and CPU resources, whether it's a configuration change or otherwise, because it still exacerbates the overall issue. (more on that in a moment)

WEG] Generally, I'm not sure that I'd make so much of a distinction. While yes, in theory changes of this type only impact the ASN that chooses to implement it, rather than what it announces to the outside world, the global scaling problem is due to the intersection between available resources, their growth curve, and the growth curve of the routing table. Saying that it only is a concern if it contributes to the size of the DFZ routing table is oversimplifying the root problem, because if internal scale problems exhaust the resources available for both internal and external routes, you still have the same end state - out of resources. In that case, the only difference between a local scaling problem and a global problem is the deployment penetration. If this is widely deployed, it has now steepened the growth curve noted in 4984, because it still is using some of the overall available resources. I've said on more than one occasion that the iBGP routes carried by an SP are as much or more of a problem than the growth of the global table because they don't have nearly as much of the aggregation and optimization to reduce their footprint. The only difference is the level of administrative control over growth, but that's a fairly limited knob to turn - for lots of reasons it may not be any more feasible to change things internally to reduce internal route growth than it is to change global route growth.
Besides, I think that your draft is trying to have it both ways - you malign Add Paths for having scaling problems, and then seem content to gloss over a very similar problem created by your solution simply because it appears to be slightly less severe and more localized.

WEG] I'm sorry, maybe I'm being thick, but I still don't understand how this would work in a way that would always avoid routing loops. Under normal state, you have a RR client reflecting its best path to the client based on the routes it receives from the rest of its neighbors, meaning that the clients don't have visibility to candidate alternatives that the RR does, so they're all making the same choice at least within the local cone of influence of that RR.
You add a second set of RRs (rr') that is announcing a second-best path as if it was the best path to restore 1 (or more) of the candidate alternatives to the client. The client receives the best and 2nd-best path and evaluates them using standard methods. If the thing that makes one route better than the other is something locally interesting like metric, and the client's particular place in the universe means that the metric is different as compared to other clients, the P routers, and the RRs, it may choose the 2nd-best path as best, and this may lead to routing loops if it tries to send the route to another router that has a different belief of what the best path is. This case is much more likely if the RR and RR' are not collocated with all of their clients and/or each other. I think that this may also be the case when the tiebreaker is router-id if you're not careful of the way that you address your route-reflectors and/or are not doing next-hop self at the edges.
Only in the case where the 2nd-best path is clearly worse to all members of the ASN (lower local pref, longer AS-path, etc) are you assured of no possibility for two routers each getting a different result when evaluating those two different routes. I think that 4.2 covers some part of this case, in the way that it documents its assumptions and what must be done to enable deployment, especially the references to ignoring IGP metric, but IMO it's not clear enough in the explanation why some of these things must be done - the failure case isn't discussed.

WEG] the document should explicitly state this. That's exactly what I was getting at when I mentioned analysis above. If nearly all applications only need one alternate to bring the total paths to two, and more would be diminishing returns, the document should recommend this, and note that more are possible if the operator's situation dictates by simply repeating the deployment more times. I will note that this guidance as well as the note at the end of 4.2 that "The additional planes of route reflectors do not need to be fully redundant as the primary one does" contradicts your example because it has both RR1' and RR2'.

WEG] I'm not looking for a level of detail that requires you to discuss the number of bytes per path. Simply noting that scaling issues exist and their general categories is enough. Make the logical leap for your reader that implementing this solution brings with it the scaling problems inherent with adding an additional route reflector (and therefore its additional routes and paths).

WEG] it may not be specific to diverse-path, but diverse-path is specifically advocating doing something that would otherwise not be done (adding additional RR<->client BGP peers w/full routes beyond what is necessary for simple RR redundancy). Therefore I still think that you need to discuss the specific scaling concerns that this implementation needs to consider, even if it's at a relatively high level and the document notes that these are not unique to this implementation. I agree that a general scaling considerations document may be appropriate, but since that does not exist and I don't want this document to be blocked awaiting completion of such, a brief discussion within this document would help a lot.

WEG] sure, but I don't think that it's valid to assume that all analysis tools have taken this into account in their implementation, so it's worth mentioning as an operational consideration. The comment may be helpful to characterize the level of potential impact.

WEG] a good clarification to add to the document.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 22 Sep 2011 18:55:14 +0200

Hi Wes,

Many thx for your comments. I will clarify the corresponding sections in
the draft.

As to your point of routing loop danger caused by advertising additional 
paths via IBGP I think if you could illustrate a topology example where 
such loop could form it would help to perhaps make the spec more clear 
to address such concern.

Many thx,
R.

> -----Original Message----- From: Robert Raszuk
> [mailto:[email-masked]] Sent: Wednesday, September 21, 2011 5:25
> PM To: George, Wesley Cc: [email-masked] Subject: Re: [GROW] I-D
> Action: draft-ietf-grow-diverse-bgp-path-dist-05.txt
>
> Hello Wes,
>
> I would prefer not to mix the growing internet scale concerns from
> some of the operational practices/configuration based based scale
> concerns.
>
> WEG] Understand, but I'm not sure that it's so easy to separate the
> two. You'll find me saying the same thing to anyone suggesting a
> change that has the net effect of significantly increasing the burn
> rate for memory and CPU resources, whether it's a configuration
> change or otherwise, because it still exacerbates the overall issue.
> (more on that in a moment)
>
>> I've made similar comments to the SIDR folks, and I think generally
>> anything that adds a non-trivial amount of impact to the growth
>> curve of the routing system needs to consider this.
>
> WEG] Generally, I'm not sure that I'd make so much of a distinction.
> While yes, in theory changes of this type only impact the ASN that
> chooses to implement it, rather than what it announces to the outside
> world, the global scaling problem is due to the intersection between
> available resources, their growth curve, and the growth curve of the
> routing table. Saying that it only is a concern if it contributes to
> the size of the DFZ routing table is oversimplifying the root
> problem, because if internal scale problems exhaust the resources
> available for both internal and external routes, you still have the
> same end state - out of resources. In that case, the only difference
> between a local scaling problem and a global problem is the
> deployment penetration. If this is widely deployed, it has now
> steepened the growth curve noted in 4984, because it still is using
> some of the overall available resources. I've said on more than one
> occasion that the iBGP routes carried by an SP are as much or more of
> a problem than the growth of the global table because they don't have
> nearly as much of the aggregation and optimization to reduce their
> footprint. The only difference is the level of administrative control
> over growth, but that's a fairly limited knob to turn - for lots of
> reasons it may not be any more feasible to change things internally
> to reduce internal route growth than it is to change global route
> growth. Besides, I think that your draft is trying to have it both
> ways - you malign Add Paths for having scaling problems, and then
> seem content to gloss over a very similar problem created by your
> solution simply because it appears to be slightly less severe and
> more localized.
>
>> 4. This asserts that no code changes are necessary to RR clients.
>> I'm not sure I totally agree with that... If the idea is to have a
>> primary (best) RR and then N additional paths, the general
>> assumption is that the N, N1, ... RRs are carrying routes that are
>> less and less preferred. How does this system avoid the same sort
>> of inconsistency of best path choice among different routers in the
>> network if there is no way to identify those paths as secondary? I
>> think you need some way to determine if the alternate routes are
>> intended to be ECMP routes or backup routes... You may be able to
>> cover this without code changes by using alternate configurations
>> of other BGP preference indicators (MED, Localpref, metric, etc),
>> perhaps with inbound route policy on the client or outbound on the
>> RR, but since things like metric may be different based on where
>> something is in the network, that may lead to inconsistency if used
>> by itself. Even then, the draft doesn't discuss how this should be
>> managed.
>
>
> I stand by the claim that no code change is needed on clients.
> Moreover no even additional policy change is required either.
>
> The best way to illustrate this is to compare presence of additional
> BGP paths on the clients in the scenario where clients are
> interconnected with full IBGP mesh or would get all paths with
> add-path. In neither case there is a notion of RR telling client
> which path is best or which is second best .. and there is number of
> good reasons for that (one is that for RR numbering paths can be
> different then for client, the other one is that when we would
> withdraw any path advertised and ordered we would need to re
> advertise with new order all remaining paths - that amount of churn
> is non negligible).
>
> WEG] I'm sorry, maybe I'm being thick, but I still don't understand
> how this would work in a way that would always avoid routing loops.
> Under normal state, you have a RR client reflecting its best path to
> the client based on the routes it receives from the rest of its
> neighbors, meaning that the clients don't have visibility to
> candidate alternatives that the RR does, so they're all making the
> same choice at least within the local cone of influence of that RR.
> You add a second set of RRs (rr') that is announcing a second-best
> path as if it was the best path to restore 1 (or more) of the
> candidate alternatives to the client. The client receives the best
> and 2nd-best path and evaluates them using standard methods. If the
> thing that makes one route better than the other is something locally
> interesting like metric, and the client's particular place in the
> universe means that the metric is different as compared to other
> clients, the P routers, and the RRs, it may choose the 2nd-best path
> as best, and this may lead to routing loops if it tries to send the
> route to another router that has a different belief of what the best
> path is. This case is much more likely if the RR and RR' are not
> collocated with all of their clients and/or each other. I think that
> this may also be the case when the tiebreaker is router-id if you're
> not careful of the way that you address your route-reflectors and/or
> are not doing next-hop self at the edges. Only in the case where the
> 2nd-best path is clearly worse to all members of the ASN (lower local
> pref, longer AS-path, etc) are you assured of no possibility for two
> routers each getting a different result when evaluating those two
> different routes. I think that 4.2 covers some part of this case, in
> the way that it documents its assumptions and what must be done to
> enable deployment, especially the references to ignoring IGP metric,
> but IMO it's not clear enough in the explanation why some of these
> things must be done - the failure case isn't discussed.
>
>> 4.1 Also, there's a definite scaling consideration on the RR
>> clients that isn't really discussed here - they are now going to be
>> storing some number of additional routes and paths that is linearly
>> related to the number of additional planes that are implemented.
>> The addition of more RR sessions that presumably carry a portion of
>> the full routing table now drives a non-trivial increase in memory
>> footprint and processing overhead (and potentially convergence time
>> for slower boxes). In the simplest case of 2 primary
>> route-reflectors (for diversity), and 1 2nd-best path RR, you've
>> added one session. If you want to carry a 3rd-best RR or have
>> redundant 2nd-best RRs, you've added 4 sessions. It's fair to say
>> that after a certain number of alternate paths, you start having
>> less routes because there are only so many alternative exits, but
>> otherwise there is a potentially large problem even if it's not
>> quite as bad as addpaths. I might recommend that you do some
>> analysis of the routing table to know where this threshold makes a
>> difference, based on how many alternate paths an average route
>> carries. In addition to being a scaling consideration, it also
>> helps to inform what value of N becomes diminishing returns because
>> most networks don't have that many backup paths. I envision this
>> being something like "80% of routes have 4 or less paths, so moving
>> beyond 4 planes may add overhead without much benefit..."
>
> It is absolutely correct to say that more paths client carries the
> more CPU cycles and memory will be used to process and store them.
>
> However there is one observation to be made ... in 99% of cases I
> have seen for distributing more then best path intra-domain the
> sufficient number of paths per net on each client is 2.
>
> WEG] the document should explicitly state this. That's exactly what I
> was getting at when I mentioned analysis above. If nearly all
> applications only need one alternate to bring the total paths to two,
> and more would be diminishing returns, the document should recommend
> this, and note that more are possible if the operator's situation
> dictates by simply repeating the deployment more times. I will note
> that this guidance as well as the note at the end of 4.2 that "The
> additional planes of route reflectors do not need to be fully
> redundant as the primary one does" contradicts your example because
> it has both RR1' and RR2'.
>
>
> IMHO cost of bringing additional paths for control plane is quite
> well understood today. Moreover it is quite implementation dependent.
> Some implementation may use X bytes per path while the other one Y
> bytes to store the same path. I think some separate BGP scaling
> document (even as BCP) may be equally useful for any technique to
> advertise more then best path. I would prefer to keep this outside of
> the solutions work on how to advertise and distribute those
> additional paths.
>
> WEG] I'm not looking for a level of detail that requires you to
> discuss the number of bytes per path. Simply noting that scaling
> issues exist and their general categories is enough. Make the logical
> leap for your reader that implementing this solution brings with it
> the scaling problems inherent with adding an additional route
> reflector (and therefore its additional routes and paths).
>>
>> discussion to your deployment considerations (section 6) to
>> discuss some of the above.
>
> WEG] it may not be specific to diverse-path, but diverse-path is
> specifically advocating doing something that would otherwise not be
> done (adding additional RR<->client BGP peers w/full routes beyond
> what is necessary for simple RR redundancy). Therefore I still think
> that you need to discuss the specific scaling concerns that this
> implementation needs to consider, even if it's at a relatively high
> level and the document notes that these are not unique to this
> implementation. I agree that a general scaling considerations
> document may be appropriate, but since that does not exist and I
> don't want this document to be blocked awaiting completion of such, a
> brief discussion within this document would help a lot.
>
>> perspective of route analysis - if you have either a homebuilt or
>> off the shelf set of software that does route analysis for the
>> purpose of event root-cause analysis, anomaly detection, capacity
>> evaluating the routing table to determine what the expected
>> behavior should be in the real network. This is especially
>> important when it uses the table to determine how traffic will
>> reroute during different failure scenarios. These tools may act
>> like a participant in the mesh rather than a client in order to get
>> a pure view of the table, and that may lead to undesired results if
>> the multiple planes aren't taken into account. There may also be
>> considerations for looking glass implementations and the actual
>> information that is visible on the RRs and RR clients as the result
>> of standard BGP show commands to aid in troubleshooting and
>> verification.
>
> additional paths can be a fact today as already mentioned with full
> mesh or as used by some operator's by playing with adjusting
> different weight values of pair of RRs on a per net basis.
>
> WEG] sure, but I don't think that it's valid to assume that all
> analysis tools have taken this into account in their implementation,
> so it's worth mentioning as an operational consideration. The comment
> may be helpful to characterize the level of potential impact.
>
> - The use of "planes" in the draft is more of a conceptual nature.
> In practice all paths are still kept in the single table where normal
> best path is calculated. That means that tools like looking glass
> should not observe any changes nor impact.
>
> This E-mail and any of its attachments may contain Time Warner Cable
> proprietary information, which is privileged, confidential, or
> subject to copyright belonging to Time Warner Cable. This E-mail is
> intended solely for the use of the individual or entity to which it
> is addressed. If you are not the intended recipient of this E-mail,
> you are hereby notified that any dissemination, distribution,
> copying, or action taken in relation to the contents of and
> attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify
> the sender immediately and permanently delete the original and any
> copy of this E-mail and any printout.
>
>

---

From: Jakob Heitz <[email-masked]>
Date: Thu, 22 Sep 2011 13:09:47 -0400

I pointed this out on March 27 and proposed a fix.
I believe Pradosh had forseen the same problem when
he proposed the "Edge_Discriminator attribute" in
draft-pmohapat-idr-fast-conn-restore.

To prevent clients in different clusters from choosing
different bestpaths based on the router ID tie breaker (f):

The RR Plane that advertises the best path MUST be configured
with a BGP Identifier higher than that of the RR Plane that
advertises the 2nd best. This must be higher than that of
the Plane that advertises the 3rd best and so on.

Similarly,

To prevent clients in different clusters from choosing
different bestpaths based on the peer address tie breaker (g):

The RR Plane that advertises the best path MUST be configured
with a peer address higher than that of the RR Plane that
advertises the 2nd best. This must be higher than that of
the Plane that advertises the 3rd best and so on.

On Thursday, September 22, 2011 9:55 AM, Robert Raszuk <> wrote:

> Hi Wes,
>
> Many thx for your comments. I will clarify the corresponding sections
> in the draft.
>
> As to your point of routing loop danger caused by advertising
> additional paths via IBGP I think if you could illustrate a topology
> example where such loop could form it would help to perhaps make the
> spec more clear to address such concern.
>
> Many thx,
> R.
>

--
Jakob Heitz.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 22 Sep 2011 20:54:30 +0200

Hello Jakob,

> draft-pmohapat-idr-fast-conn-restore.

Nope. This is only applicable for external paths when advertised into AS 
to make sure different external paths would be evaluated before the 
rtr_id tie break.

> the Plane that advertises the 3rd best and so on.

I do not believe this would in any case result in intra-domain loop. In 
IBGP if metric to the next hop is identical and if all attributes during 
best path comparison are equal it really does not matter which path is 
selected as best or second best or third best.

The packets are switched to their IP destinations which in turn are 
directed by each IP lookup router towards the next hop. And next hop 
metric comparison is performed much sooner then rtr_id check hence I see 
no loop problem.

Prove me wrong otherwise by an illustration please ;-)

Cheers,
R.

> On Thursday, September 22, 2011 9:55 AM, Robert Raszuk<>  wrote:
>
>> Hi Wes,
>>