Subject: [GROW] Discussion from today's WG meeting: ibgp scaling considerations

From: Christopher Morrow <[email-masked]>
Date: Thu, 26 Mar 2009 14:09:45 -0400

Today's WG meeting brought out some contentious discussion around this
presentation. The summary for a portion of the discussion was that
local optimizations in BGP mechanisms can often lead to system wide
systemic issues. One comment was that a particular feature
(advertise-own) ends up being used in 'internet' context where it's
inappropriate. In this instance though, often the VPN providers are
also running 'internet' as a VPN, to lower capex/opex and take
advantage of their larger 'internet' platforms for smaller 'vpn'
solutions.

The discussion seemed quite contentious and like it may prove
interesting to discuss here.as well...

-Chris

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 12:19:55 -0600

> advantage of their larger 'internet' platforms for smaller 'vpn'

So to clarify..  It's ACCEPT_OWN, and it's not what's
introducing the problem, it's "utilizing" it.

The issue I have *here* is the fact that a client (c)
advertises a prefix (p) to it's local RRs.  The RRs reflect
the route back to the client and the client is expected to
poison it based on the Originator ID matching the local
Router ID.

In RFC 1966 the RRs did not reflect the route back the client
because the client didn't know it was a client, and for
incremental deployment it was required that the RRs not reflect
it back or a routing information loop could occur.  RFC 2796
was updated and allowed an RR to reflect a route back to a
client and the onus was on the client to drop the update.

The problem is that if the client is advertising 100ks of
routes (which is perfectly reasonable for a peering edge
router) to the RRs, (100ks * num_rrs) are reflected back to
the client, and it now has to senselessly process and discard
all of those updates.

ACCEPT_OWN comes into the picture only because it employs
this fundamentally broken behavior.  IF ACCEPT_OWN was a
function of the RR, and the routes were only reflected back
to the client IF that community was presented, I'd be
perfectly fine with it.

-danny

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 26 Mar 2009 11:23:41 -0700

Hi Chris,

Danny's slides were not posted is there an internet draft documenting 
those BGP "local" optimizations leading to system wide systemic issues ?

What is the point of this discussion ?

*A* To change reflector spec back to prohibit reflection of paths 
previously received from the originating client ? Could such 
recommendation also take into the account the impact for the reflector 
itself ?

*B* To kill some other work (ADVERTISE_OWN) which while building on 
reflecting back to the originating client due it's design not having 
anything to do with plain IPv4/IPv6 routes - hence have no local PE box 
nor system wide impact ?

Can you or Danny clarify the point of the talk ?

Cheers,
R.

> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
> 
>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 26 Mar 2009 11:27:29 -0700

Danny,

Please note that tt is perfectly feasible not to reflect back to the 
client for one AFI/SAFI and still reflect back for other AFI/SAFIs.

If your main point complains about the change and permission to reflect 
back PATHs to the client introduced by RFC 2796 in the scope of the 
Internet let's drop even mentioning the ACCEPT_OWN as not relevant to 
the topic.

Cheers,
R.

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 12:30:53 -0600

On Mar 26, 2009, at 12:27 PM, Robert Raszuk wrote:

> If your main point complains about the change and permission to  
> reflect back PATHs to the client introduced by RFC 2796 in the scope  
> of the Internet let's drop even mentioning the ACCEPT_OWN as not  
> relevant to the topic.

But it is relevant because it employs that function.
As I said below:

"IF ACCEPT_OWN was a function of the RR, and the
routes were only reflected back to the client IF
that community was presented, I'd be perfectly
fine with it."

-danny

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 26 Mar 2009 11:37:59 -0700

Hi Danny,

AFI/SAFI 1/128 employs also many other functions some of them you may 
disagree with.

I don't think we need to blame all SAFI behavior. IMHO if you see a 
practical problem with reflecting back paths for the Internet IPv4/IPv6 
let's document those and quantify the significance of the problem.

Perhaps the conclusion would be reached that for IPv4 we should not 
reflect back to the src. Perhaps we will reach a contrary result that 
dropping routes is so cheap and intra-domain bandwith these days of no 
issue to worry about it.

But in all of this we could still if there is a valid need reflect back 
to the originating PE vpnv4 routes. Speaking of which there is already a 
mechanism in place called rt-constrain which does not allow to send all 
routes back yet does not break peer-group concept of single update 
generation and fast replication to the peers.

Cheers,
R.

---

From: Gargi Nalawade <[email-masked]>
Date: Thu, 26 Mar 2009 11:31:57 -0700

Disclaimer : Im not taking a stance on whether it is good or bad for an RR to advertise
a client's routes back to it. Want to make some observations on the pros and cons that
go into the advertise or not-to-advertise decision.

If the RRs were to stop reflecting back own routes to the clients, this would break
any peer-groups configured on the RRs. This would result in a lot more individual 
BGP updates being formatted and sent out. In the worst case 100 different sets of
updates for 100 clients. This would slow down convergence.

-Gargi

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Danny McPherson [[email-masked]]
Sent: Thursday, March 26, 2009 11:19 AM
To: Christopher Morrow
Cc: [email-masked]
Subject: [GROW] RR reflection to clients

-danny

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 26 Mar 2009 11:50:30 -0700

> the client, and it now has to senselessly process and discard

Do you have any data about how much processing this requires?

Recognizing the originator ID can happen very early when expensive  
actions such as applying prefix lists and route maps and inserting in  
the RIB haven't happened yet, so the impact of doing this could be  
fairly negligible.

(Then again, a Pentium200 can do hundreds of megabytes of MD5 hashing  
but somehow enabling MD5 on BGP sessions takes a good bit of toll on  
many route processors.)

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 26 Mar 2009 15:06:20 -0400

Danny,

> that community was presented, I'd be perfectly

I think the comment in quotes above demonstrates that at worst,  
ACCEPT_OWN is tangential to your RR beef.  Suppose the RR spec was  
changed to mandate suppression of own routes being sent back to their  
originators -- in that case I think we'd be happy to update the  
ACCEPT_OWN spec as you describe, to explicitly permit sending just the  
special routes back to their originators.  Of course it was  
unnecessary to spec this in the ACCEPT_OWN doc since as you point out,  
reality is that the routes are sent anyway.

So, I agree with Robert (and I think you do too) that this is a  
distraction from your primary point.

--John

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 26 Mar 2009 15:13:37 -0400

In the interest of historical accuracy:

On Mar 26, 2009, at 2:19 PM, Danny McPherson wrote:
> it back or a routing information loop could occur.

The fact is that some RFC 1966 (and pre RFC 1966) RR implementations  
*did* reflect routes back to the client, and relied on the client to  
suppress them, because of this:

     A BGP speaker SHALL NOT install a route with itself as the next  
hop.

(That's the RFC 4271 language but 1771 said the same or similar.)

--John

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 13:29:09 -0600

> updates for 100 clients. This would slow down convergence.

That's an implementation issue and practical reality
that folks have to deal with, I understand that.  It's
a matter of when the pain threshold exceeds the value
of this optimization.

Interestingly, this is how Juniper implemented it
initially - I'm not sure what the current behavior
is but I know that ACCEPT_OWN would require them to
change at least _some of this, if they haven't already.
I know this because a C router was reflecting routes
back to a J router and the J router was installing, and
an interesting discussion commenced at the time in a
network where I was an operator.  This is when J
implemented the poison-based-on-orig-ID on the client
side.  I hope it didn't drive them to change the route
reflectors 'reflection' behavior as well.

-danny

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 13:34:13 -0600

> client for one AFI/SAFI and still reflect back for other AFI/SAFIs.

Sure, one could do that, I don't think I said anything to
the contrary.

> of the Internet let's drop even mentioning the ACCEPT_OWN as not  

No, it's as much (more?) of an issue in VPNv4 or other SAFIs.
A client will receive every best route it sends to RRs * the
number of RRs.  Those updates have to be processed by the client
and discarded, and they're in the exact same processing path as
all other BGP updates from internal or external peers.  If I
were running a VPN network I'd prefer that the client spend it's
time and cycles chewing on useful updates, not useless updates.

Also note that MANY IP/VPN deployments use the same routers
for VPNs and Internet, and most implementations today don't
discriminate update processing per AF.

-danny

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 13:36:42 -0600

On Mar 26, 2009, at 12:37 PM, Robert Raszuk wrote:
>
> But in all of this we could still if there is a valid need reflect  
> back to the originating PE vpnv4 routes.

Yes, with the "if there is a valid reason" being the fulcrum.

-danny

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 26 Mar 2009 12:37:19 -0700

Danny,

I already pointed out that we have rt-constrain which based on pushing 
RTs precisely addresses that problem in the context of VPNv4.

Note is works equally well for customer VPNs routes or for the Internet 
routes in the VRFs.

Cheers,
R.

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 13:38:56 -0600

> Do you have any data about how much processing this requires?

No.

> actions such as applying prefix lists and route maps and inserting  
> in the RIB haven't happened yet, so the impact of doing this could  
> be fairly negligible.

Could be.  Then again, there's NO way that it can me more
efficient then having not received the updates - from a system
state perspective.  Further, those updates are multiplied by
the number of RRs and handled in the same processing path as
ALL other updates, so operators should be concerned about this.

> (Then again, a Pentium200 can do hundreds of megabytes of MD5  
> hashing but somehow enabling MD5 on BGP sessions takes a good bit of  
> toll on many route processors.)

Indeed!

-danny

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 13:40:14 -0600

On Mar 26, 2009, at 1:06 PM, John G. Scudder wrote:

> ACCEPT_OWN is tangential to your RR beef.

Unless implementations change their behavior to support this,
which is why I keep associating it.  But yes, in general, I
agree.

> Suppose the RR spec was changed to mandate suppression of own routes  
> being sent back to their originators -- in that case I think we'd be  
> happy to update the ACCEPT_OWN spec as you describe, to explicitly  
> permit sending just the special routes back to their originators.   
> Of course it was unnecessary to spec this in the ACCEPT_OWN doc  
> since as you point out, reality is that the routes are sent anyway.

Agreed..

> distraction from your primary point.

Noted.

-danny

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 26 Mar 2009 12:41:53 -0700

John,

 > A BGP speaker SHALL NOT install a route with itself as the next hop.

Isn't the below quote you cite talks about EBGP cases ?

For IBGP AFAIK no one was at that time doing next hop self on ASBRs 
hence just per the below sentence reflected routes would end up on the 
ASBRs.

In fact 1771 says just below your quote:

"  When a BGP speaker advertises the route to a BGP speaker located in
    its own autonomous system, the advertising speaker shall not modify
    the NEXT_HOP attribute associated with the route."

Cheers,
R.

>     A BGP speaker SHALL NOT install a route with itself as the next hop.
>

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 13:43:54 -0600

> (That's the RFC 4271 language but 1771 said the same or similar.)

I'm not sure about that, I know that in 1998 I had this problem
in a real network, as noted earlier.  From what I recall the spec
was the way it was because of this - what implementations did at
the time I do not know.

-danny

---

From: "John G. Scudder" <[email-masked]>
Date: Thu, 26 Mar 2009 15:51:18 -0400

> Isn't the below quote you cite talks about EBGP cases ?

No, it's not conditional.  (Section 5.1.3 if you want to check.)

Also, regardless of what the spec says, it would seem pretty clear  
that it never makes sense to install a route with yourself as the next  
hop, whether EBGP or IBGP.  (Modulo the issues discussed in the  
ACCEPT_OWN spec -- in that case the caveat is that the notion of  
"yourself" is a little tricky since multiple forwarding contexts are  
in use.)

> hence just per the below sentence reflected routes would end up on  
> the ASBRs.
>

Sure, but I don't see how that's material to the case under discussion.

--John

---

From: "UTTARO, JAMES, ATTLABS" <[email-masked]>
Date: Thu, 26 Mar 2009 15:56:20 -0400

It seems that trading off the work that a PE has to do to parse the
routes that have been sourced by itself needs to be weighed against the
added complexity and additional work a RR must do to ensure that it does
not send the routes back to said PE. I would guess that the RR would
need to maintain multiple data structures associated with each client
peer, have multiple update groups etc... The main point is which network
device should do the work? My approach has always been to minimize the
work on the RR as it is most important to the viability of the entire
service/network.. The RR infrastructure is more critical and important
than any single PE...

Jim Uttaro 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Danny McPherson
Sent: Thursday, March 26, 2009 3:40 PM
To: John G. Scudder
Cc: [email-masked]
Subject: Re: [GROW] RR reflection to clients

Agreed..

Noted.

-danny

---

From: Tony Li <[email-masked]>
Date: Thu, 26 Mar 2009 13:05:53 -0700

> service/network.. The RR infrastructure is more critical and important

There's nothing special about that in the RR.  It's the exact same 
mechanisms that any BGP speaker needs anyways.

Tony

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 26 Mar 2009 13:07:09 -0700

John,

 > Sure, but I don't see how that's material to the case under

Well it is to sort the history which you just started yourself to bring 
into this thread :-)

>> > A BGP speaker SHALL NOT install a route with itself as the next hop.

Clearly it was not the next hop. So my question was - what was it then 
in RFC1771-March1995 which made those clients dropping them ?

Or perhaps just like Danny says implementions of BGP based on 1771 did 
not know about ORIGINATOR check which must be coded since it was coming 
in the future in RFC1966-June1996 hence the reflected routes were 
originally accepted ?

R.

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 15:13:50 -0600

> mechanisms that any BGP speaker needs anyways.

Indeed.  I think one could argue that RRs are "overlay"
and most likely redundant, so they're not as critical as
a PE with a customer termination.

-danny

---

From: "Pradosh Mohapatra (pmohapat)" <[email-masked]>
Date: Thu, 26 Mar 2009 15:05:53 -0700

| > If the RRs were to stop reflecting back own routes to the clients, 
| > this would break any peer-groups configured on the RRs. This would 
| > result in a lot more individual BGP updates being formatted 
| and sent 
| > out. In the worst case 100 different sets of updates for 
| 100 clients. 
| > This would slow down convergence.
| 
| 
| That's an implementation issue and practical reality that 
| folks have to deal with, I understand that.  It's a matter of 
| when the pain threshold exceeds the value of this optimization.

Just a note: even with peer-group implementation, there are 
optimization tricks that one can do to not advertise the UPDATEs 
to the originator (e.g. checking originator ID at the time of 
TCP replication).

- Pradosh

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 26 Mar 2009 15:24:32 -0700

> ALL other updates, so operators should be concerned about this.

Then again, this is only an issue for the prefixes the router itself  
sends to the RRs and then are reflected back. You mentioned some  
really large numbers in your talk which seem atypical to me: in a  
network large enough to have RRs I would assume that most routers are  
only responsible for a relatively small number of best prefixes in the  
global table. Yes, this multiplies by the number of RRs but then again  
so do all the prefixes that the router isn't considered best for,  
which require significant more resources.

In the case where a router would indeed get back 100k if its own  
prefixes from RRs it would make more sense to make that router a non  
client peer of the RRs and make the problem go away in that way.

Considering all of the above and the fact that avoiding backreflection  
means breaking groupwise processing on the RRs which means extra  
processing for ALL prefixes times ALL clients for each RR I'd say that  
we shouldn't make changes here, but limit ourselves to documenting the  
issue so that vendors and operators can make sure the additional  
processing for backreflection is reduced.

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 26 Mar 2009 18:41:33 -0400

On Thu, Mar 26, 2009 at 2:23 PM, Robert Raszuk <[email-masked]> wrote:
> Hi Chris,
>
> Danny's slides were not posted is there an internet draft documenting those
> BGP "local" optimizations leading to system wide systemic issues ?
>
> *A* To change reflector spec back to prohibit reflection of paths previously
> received from the originating client ? Could such recommendation also take
> into the account the impact for the reflector itself ?
>
> anything to do with plain IPv4/IPv6 routes - hence have no local PE box nor
> system wide impact ?
>

I believe I can get the talk posted today... Hopefully also Danny can
clarify some (which I believe he did on another thread as well).

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 26 Mar 2009 20:31:32 -0400

> clarify some (which I believe he did on another thread as well).

note that Peter uploaded the slide decks...

-Chris

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 19:23:22 -0600

On Mar 26, 2009, at 4:24 PM, Iljitsch van Beijnum wrote:

> network large enough to have RRs I would assume that most routers  
> are only responsible for a relatively small number of best prefixes  
> in the global table. Yes, this multiplies by the number of RRs but  
> then again so do all the prefixes that the router isn't considered  
> best for, which require significant more resources.

Those numbers are likely to be reflective of the aggregate
set of unique prefixes (best routes) from ALL external peers
in an Internet table, as the local eBGP route is _usually
the best.

> client peer of the RRs and make the problem go away in that way.

I don't grok that?

> Considering all of the above and the fact that avoiding  
> backreflection means breaking groupwise processing on the RRs which  
> means extra processing for ALL prefixes times ALL clients for each  
> RR I'd say that we shouldn't make changes here, but limit ourselves  
> to documenting the issue so that vendors and operators can make sure  
> the additional processing for backreflection is reduced.

So I've taken a few minutes to illustrate the problem in
a couple slides in more detail.  The PDF is available here
at the link below, and I've attached (I hope this doesn't
cause problems for anyone):

http://www.tcb.net/rr-thing.pdf

To summarize some more thoughts on this, and best
illustrated in slide 3 in the deck above:

1) all best paths learned from the client are reflected
back to the client

2) this is multiplied by the number of RRs to which that
client peers

3) client processing of updates results in placement of ALL
updates in [serialized?] BGP input processing queue with ALL
other updates {m, 1..n}

*Input processing queue typically scheduled in round robin model

4) if n is sufficiently large, many of these updates may
very well still be in the process of being learned from
the external peer(s) (r) and initial convergence function,
and processed behind extraneous reflected routes

5) Serialized processing means all Address Families likely
effected.

6) Reflected route MAY be legitimate withdraw if alternative
best previously advertised from RR - therefore it must be
processed normally IF received - this is likely one of the
more interesting bits for the RR implementer.

I believe this is AN issue that implementers should
look at.  I understand that there may be efficiencies
in some implementations to copy and it's more complex
to not reflect to client IF Originator == client RID.

-danny

---

From: Danny McPherson <[email-masked]>
Date: Thu, 26 Mar 2009 19:27:48 -0600

>> BGP "local" optimizations leading to system wide systemic issues ?

I provided them to the chairs, they are available now.

>> What is the point of this discussion ?

There are many points.  Mainly that there is room for
improvements in three places: operations and network
architecture, implementations, and BGP itself.

>> previously

Sure.

>> anything to do with plain IPv4/IPv6 routes - hence have no local PE  

See email I sent a moment ago.  Like I said, I have
zero issue with a solution such as that, so long as
implementers aren't planning to change general BGP
behavior to make it easier.

I'll work with a few folks to document several of these
issues in a draft or two, apologies for not having them
available sooner.

-danny

---

From: Gargi Nalawade <[email-masked]>
Date: Thu, 26 Mar 2009 18:58:06 -0700

Hi Danny, All,

I agree that there are/could be efficiencies in some implementations or could be made.
We are still bound by the #BGP updates on the wire though.

Consider 3 RR-clients R1, R2, R3 advertising 3 prefixes P1, P2, P3 respectively to an 
RR. Further, R1 & R2 advertise P12, R2 & R3 advertise P23 and R3 & R1 advertise 
P31 to the RR. 

In the case you describe where the RR advertises the client routes back to the clients,
all the prefixes - P1, P2, P3, P12, P23 & P31 would be advertised in 1 packet which
gets replicated to all 3 clients (assuming all 3 are in a peer-group and the rest of the
attributes are same).

Now if we change it so that the RR does not advertise the client's own routes back to it,
the RR would end up formatting & sending - in the best case, 3 BGP updates instead of 1.

My point is - no matter what efficiencies we build into an implementation - mandating
that the RR not advertise client routes back to clients, would increase the #BGP updates 
processed by the RR and all the "n" RR-clients as well as on the wire by n-fold in the worst 
case.

-Gargi

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Danny McPherson [[email-masked]]
Sent: Thursday, March 26, 2009 6:23 PM
To: Iljitsch van Beijnum
Cc: [email-masked] [email-masked]
Subject: Re: [GROW] RR reflection to clients

I don't grok that?

-danny

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 26 Mar 2009 22:08:55 -0400

> all the prefixes - P1, P2, P3, P12, P23 & P31 would be advertised in 1 packet which

 'could be advertised in one packet' (if there aren't other things
associated with the routes which make it not packable in a single bgp
packet which may span more than 1 IP packet).

> processed by the RR and all the "n" RR-clients as well as on the wire by n-fold in the worst

only the RR is affected in this case, it makes 'more than one' packet,
one for each client... right?

-Chris

---

From: Shane Amante <[email-masked]>
Date: Thu, 26 Mar 2009 20:43:05 -0600

Iljitsch,

> best for, which require significant more resources.

I think it's important to point out that the concern is /routes/,  
i.e.: /paths/ + prefixes, not just prefixes alone that are the concern  
here.  In addition, your comment seems to be from the viewpoint of a  
"snapshot in time"; however, BGP (particularly on Edge routers) is  
very dynamic, because it's constantly receiving eBGP updates from  
peers and/or customers, performing path selection and when it arrives  
at a new best path forwarding it to RR's and then receiving it back.   
The larger point here is paths are much more dynamic than just  
prefixes, resulting in a more offered load on the overall system ...

> client peer of the RRs and make the problem go away in that way.

However, that would increase the size of the iBGP mesh, which was the  
point of putting RR's in in the first place.  Admittedly, you don't  
have to revert to a complete iBGP full-mesh, but administratively it's  
harder to plan & deploy what boxes will be part of the full-mesh (non- 
rr-clients) vs. rr-clients and then changing configs around when  
circuits toward peers or customers are groomed from box-to-box as well.

> the additional processing for backreflection is reduced.

Re: your comment on "breaking groupwise processing" ... don't RR's  
already have to account for not sending updates to a specific peer  
(or, set of peers) based on various ORF types they've received from  
that peer (assuming ORF's are enabled)?  Why can't that same code/ 
logic apply here (i.e.: it's already written, why can't we re-use it  
here)?

-shane

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 27 Mar 2009 02:59:56 -0700

Hi Danny,

> 5) Serialized processing means all Address Families likely

I think it is beneficial to as you recommend clarify the real network or 
system wide perspective. It sounds terrible to say in general that you 
are sending me something which I need to drop. But that would happen 
essentially only in 3 cases today (comment if I missed something):

Let's consider the PE1 point of view:

ISP1-
      PE1 ---- RR ---- PE2..N
ISP2-

*A* After original BGP session up provided that new paths are considered 
as best (as compared to other paths already on RR)

*B* After sending route refresh from PE1 to RR

*C* when PE1 switches his best path from ISP1 to ISP2

I think in both *A* & *C* there is no practical issue as the best paths 
RR would send you back are your own best paths which are already 
installed. If in the same time you would get some other paths from the 
same RR the processing of new paths may be behind a bit.

It will even further be of no issue when folks would start using more 
and more add-path feature where path to path switchover would be 
triggered by fast IGP flooded event and not by BGP message withdraw.

In the *B* case there could be some unnecessary processing delay 
introduced by receiving and dropping your own paths if you ask for all 
from RR. And how often for 1/1 you send route refresh may very much 
depend on how your implementation deals with paths received. If you 
store all of them always you may not need to worry about this case either.

For all other cases those would be just incremental updates - not 
something I would suppose we need to worry about at all.

> more interesting bits for the RR implementer.

If this is a withdraw of your own path coming back to you from RR it 
most likely was originally withdrawn by yourself to the RR isn't it ?

So I think suggestion made by Pradosh to just like we filter at the TCP 
replication for rt-constrain in 1/128 do the same based on the 
Originator still holds.

It is just a question to quantify how much this filtering would be of 
real practical significance.

Cheers,
R.

---

From: "UTTARO, JAMES, ATTLABS" <[email-masked]>
Date: Fri, 27 Mar 2009 22:28:47 -0400

The ACCEPT-OWN functionality is a direct result of a business model that
is based on per route granular stitching between different routing
contexts on PEs within an AS or spread across multiple ASes. This may be
value added centralised services as provided by the service provider or
a middleman service for creating long-standing or temporal communities
of interest. The push back I have heard in re ACCEPT-OWN is that it can
be done by simply configuring the "leaking" of the routes of interest on
the PE where the two VPNs intersect. That is where the source of Route A
from VPNA needs to be leaked into VPNB on the same PE. The issue with
the configuration approach is that it assumes that customers are static
as to where they may advertise routes that they may want stitched. The
fact is the ISP does not know a priori which routing contexts, on which
PEs,and in which AS the stitching can occur. The configuration approach
is a huge operational nightmare in large scale networks that span
multiple regions of the world. We had considered it..

>"So to clarify..  It's ACCEPT_OWN, and it's not what's
>introducing the problem, it's "utilizing" it."

The notion of sending the routes back to the originator is fairly
straightforward when the entire PE is viewed as one routing context. It
may be done as an efficient mechanism for a RR to advertise routes to
multiple peers but it is not the optimal behavior as it requires both
the RR and PEs to process additional routing state. In the VPN case the
PE is really homing multiple mutually exclusive routing contexts. The
ability to create simple operational models to stitch requires that we
can get the routes back to a PE and import them into a different routing
context that originated them in the first place. So "the problem" for
one routing context is actually a requirement for creating the above
value added services. How and which routes send back for importation is
really the discussion.

Jim Uttaro

-----Original Message-----
From: Danny McPherson [mailto:[email-masked]] 
Sent: Thursday, March 26, 2009 2:20 PM
To: Christopher Morrow
Cc: [email-masked]; UTTARO, JAMES, ATTLABS
Subject: RR reflection to clients

-danny

---

From: Danny McPherson <[email-masked]>
Date: Sun, 29 Mar 2009 19:21:11 -0600

On Mar 27, 2009, at 8:28 PM, UTTARO, JAMES, ATTLABS wrote:

> How and which routes send back for importation is
> really the discussion.

Indeed.  And to my point again, I hope vendors don't
blindly reflect all routes back to a client in order
to implement this particular capability, as it's effects
are far wider and systemic, and these extraneous updates
are simply not benign.

I think I've stated my point several times now, and even
illustrated some of the concerns with this particular
issue.  I trust operators and implementers that are
concerned with implementation and systemic scale and the
effects on convergence and stability in their environments
will have considered it.  Those that don't believe it's an
issue, I have no intentions to further concern you as such.

-danny

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 30 Mar 2009 15:39:45 +0200

On 27 mrt 2009, at 2:23, Danny McPherson wrote:

>> You mentioned some really large numbers in your talk which seem  
>> atypical to me: in a network large enough to have RRs I would  
>> assume that most routers are only responsible for a relatively  
>> small number of best prefixes in the global table.

> in an Internet table, as the local eBGP route is _usually

Yes, a router thinks its own eBGP route is best--but the RR wouldn't  
necessarily agree. Suppose 4 eBGP routers and an RR, the average  
backreflection would be 25%. Now maybe it's 85% for one and 5% for the  
three others, but the average can only be 25%.

>> prefixes from RRs it would make more sense to make that router a  
>> non client peer of the RRs and make the problem go away in that way.

> I don't grok that?

If a router is responsible for a very large number of best prefixes  
that are then reflected back to that router by the route reflectors,  
it would make sense to make that router NOT a route reflector client.  
If the router in question isn't an RR client, the RRs don't reflect  
any information back to the router. Of course this means that the  
router in question must have a full iBGP mesh with other other non-RR- 
client routers as well as with all the RRs.

> 1) all best paths learned from the client are reflected

Right. Note that the average number of those paths is (table size) /  
(# eBGP routers).

> 2) this is multiplied by the number of RRs to which that

Right again, but since EVERYTHING multiplies by the number of RRs this  
is not unexpected or worthy of special concern, especially as  
backreflected paths actually use up less resources than other paths.

> updates in [serialized?] BGP input processing queue with ALL

Depends on implementation. Again, other paths require more processing  
than backreflected ones so I don't see the issue.

> and processed behind extraneous reflected routes

See above.

> 5) Serialized processing means all Address Families likely

See above.

> more interesting bits for the RR implementer.

Yes, the whole notion that a BGP router must track which updates were  
sent to which peer is a big design flaw.

> to not reflect to client IF Originator == client RID.

Agree with that, but what action do you suggest?

I think the issue is sufficiently complex and unexpected that a good  
writeup would benefit the community, but I don't think imposing any  
new rules would be helpful, if only because implementations must  
expect the current behavior for many years to come anyway.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 30 Mar 2009 15:42:37 +0200

> logic apply here (i.e.: it's already written, why can't we re-use it  

Hm, aren't filters on iBGP sessions considered bad style?

I don't know what implementations do when peer group members send  
ORFs, but it wouldn't surprise me to learn that this would make the  
peer group processing advantages disappear.

---

From: Danny McPherson <[email-masked]>
Date: Mon, 30 Mar 2009 07:56:48 -0600

On Mar 30, 2009, at 7:39 AM, Iljitsch van Beijnum wrote:

> backreflection would be 25%. Now maybe it's 85% for one and 5% for  
> the three others, but the average can only be 25%.

Indeed, I noted already that there are likely dynamics
within a cluster that change this for a given route set,
and those likely vary widely based on interconnect
topology and network architecture.

> it would make sense to make that router NOT a route reflector  
> client. If the router in question isn't an RR client, the RRs don't  
> reflect any information back to the router. Of course this means  
> that the router in question must have a full iBGP mesh with other  
> other non-RR-client routers as well as with all the RRs.

In large networks it's extremely difficult to 'turn the steam
valves' in this manner.  Also, one might argue that you do get
the inter-cluster implicit aggregation benefits if other paths
are sourced from BGP speakers within the cluster.

> Right again, but since EVERYTHING multiplies by the number of RRs  
> this is not unexpected or worthy of special concern, especially as  

How's that the case, because they're not stored in a RIB-In?
All other resource consumption and processing would be the same,
in particular because a path reflected back may well be a
withdraw, and it has to be processed like any other update.

> Depends on implementation. Again, other paths require more  
> processing than backreflected ones so I don't see the issue.

How do you figure?  Again, because it can be an implicit
withdraw you can't simply discard it.

> expect the current behavior for many years to come anyway.

Well, to be clear, some implementations at least used to
not reflect routes back to clients (e.g., JUNOS) - I hope
they haven't blindly changed default behavior to enable
capabilities such as that of ACCEPT_OWN, and they they only
reflect necessary routes back to originating clients.

Bottom line - at some point systemic effects of this will
result in something breaking, or convergence delays, or scaling
issues.  Local optimizations that result in extraneous system
state aren't ideal.

-danny

---

From: "UTTARO, JAMES, ATTLABS" <[email-masked]>
Date: Mon, 30 Mar 2009 15:47:57 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Iljitsch van Beijnum
Sent: Monday, March 30, 2009 9:43 AM
To: Shane Amante
Cc: [email-masked] [email-masked]
Subject: Re: [GROW] RR reflection to clients

>>peer group processing advantages disappear.

One of the big concerns from an RR perspective when deciding whether to
deploy ORF.

Jim Uttaro