Subject: Re: [GROW] WGLC: draft-ietf-grow-ops-reqs-for-bgp-error-handling-04

From: Robert Raszuk <[email-masked]>
Date: Mon, 09 Jul 2012 15:02:57 +0200

Hi Russ,

Regarding your topology:

                     (   )
            /-------( AS2 )--------\
     (   ) /         (   )          \ (   )       (   )
    ( AS1 )                          ( AS4 )-----( AS5 )
     (   ) \         (   )          / (   )       (   )
            \-------( AS3 )--------/
                     (   )

    o  AS1 is advertising 192.0.2.128/25 to both AS2 and AS3.

    o  AS2 is advertising both 192.0.2.128/25 and 192.0.2.0/24 into AS4.

    o  AS3 is advertising 192.0.2.128/25 into AS4

I am afraid you are too optimistic :)

The most common scenario I see is that AS2 only advertises to AS4 the 
aggregate so 192.0.2.0/24. That means that your 192.0.2.128/25 would 
never get "BOUNDED" status hence the other path via AS3 will continue 
worldwide via AS4.

Contrary as you very well know the alternative proposal 
http://tools.ietf.org/id/draft-marques-idr-aggregate-00.txt seems to 
address the above case as well as all cases you are trying to cover. 
Could you comment and compare both solutions especially as a (co-)author 
of both ;).

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 10 Jul 2012 10:50:22 +0200

Hey Russ,

> I am afraid you are too optimistic :) ==
>
> Actually, this is really simple to fix in one of two ways.
>
> First, AS4 could see the overlap on the outbound side, note the
> origin AS is the same, and set the longer prefix to NO_EXPORT.

I don't think this is as simple as this ... Remember that when the less 
specific (/24) goes away the AS4 must readvertise the more specifics not 
to break routing. So therefor I do not think that any simple policy 
would work. The choice/decision what to advertise and what not must be 
done at best path run.

> Alternatively, the BGP implementation in AS4 could be modified so
> that when running bestpath if a losing route has a specific
> community, it is transferred to the winning route. This would allow
> the winning route to gain the BOUNDED community in processing.

Actually I am not that clear I understand this business reg communities 
marking. How about we consider even much simpler and ASBR local solution:

"If we are advertising on EBGP (so setting next hop self) and there are 
less specific covering more specific routes the more specific can be 
suppressed from advertisement."

Of course the implementation must back walk when the less specific goes 
away, but this is anyway required in all of the above.

> == The most common scenario I see is that AS2 only advertises to AS4
> the aggregate so 192.0.2.0/24. That means that your 192.0.2.128/25
> would never get "BOUNDED" status hence the other path via AS3 will
> continue worldwide via AS4. ==
>
> I rarely see this --because most of the time, folks want to "cover"
> the longer match with a shorter one, in case the longer prefix fails
> for some reason. With a covering route, both ISPs will be used for
> inbound traffic, and not just the one with the remaining longer
> prefix. This has been the recommended configuration for years,
> AFAIK.

If this is PI address given to a customer I actually do not know any ISP 
who would advertise his smaller blocks in addition to his aggregate 
those blocks are taken from. Only for backdoor/multihoming the smaller 
blocks get advertised via some other ISPs, but never directly connected.

> == Contrary as you very well know the alternative proposal
> Could you comment and compare both solutions especially as a
> (co-)author of both ;). ==
>
> I prefer the simpler solution presented in the draft we just
> published. Draft-marquis and the virtual aggregation drafts both add

Just to be very clear .. VA (any of the VA drafts) does not talk about 
the same problem. For one it does not stop BGP from advertising anything 
and second it is intra-domain solution.

> a lot of complexity to operations, while the BOUNDED community is
> very simple, and can (mostly) be implemented today with no changes to
> BGP, no changes to current traffic engineering, and no increased
> stretch within the SP network. Complexity can always be added, but a
> simple system on which future modifications can be made to cover
> corner cases seems like a better idea to me... :-)

Simple but not simpler ....

Best,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 10 Jul 2012 17:00:08 +0200

>> I don't think this is as simple as this ... Remember that when the
>> less specific (/24) goes away the AS4 must readvertise the more
>> specifics not to break routing. So therefor I do not think that any
>> simple policy would work. The choice/decision what to advertise and
>> what not must be done at best path run.
>
> Sure --but won't best path be run when the best path towards a
> specific destination falls out of the table?

Best path today does not compare /16 and /24. In Pedro's proposal it does.

>> Actually I am not that clear I understand this business reg
>> communities marking. How about we consider even much simpler and
>> ASBR local solution:
>
> A local ASBR only solution won't work for various reasons, so let's
> not. :-)

I think it works. The example is comparing all best path eligible
attributes of less and more specific during best path and if the less
specific would win suppress the more specific.

I am only not sure if we really need to compare all. For example I am
not sure that like in your proposal it really matters if the more
specific was received over the same session as less specific (provided
that actually happens that often as you say :-)

> So what you are saying is that if a provider owns 10.1.0.0/16, and
> gives out 10.1.1.0/24 to a customer to advertise through some other
> provider the customer is connected to, the provider would never
> advertise 10.1.1.0/24 as well as 10.1.0.0/16? Why not? Since the
> longest match always wins, what you're saying is that the provider
> that handed out the address space would end up routing through the
> competing provider (the customer's other provider) to reach the
> customer, because there is a longer match being advertised by the
> customer through the competing provider.
>
> That doesn't make any sense to me.

That has been for years common recommendation of any RIR ... That's the 
value of PA addresses. Multihomers were told to get PI which indeed does 
not make the BGP table smaller.

Examples:

"Announcement of initial allocation as a single entity"
http://meetings.ripe.net/ripe-53/presentations/rou-ps-rou.pdf

"Advertisement of more specific prefixes should not be used unless
absolutely necessary and, where sensible, a covering aggregate
should also be advertised.  Further, LIRs should use BGP methods
such as NO_EXPORT [RFC-1997] and NOPEER [RFC-3765] or provider-specific
communities, as described in [RIPE-399] to limit the propagation of
more specific prefixes in the routing table."
SRC: ripe-532.txt

>> Just to be very clear .. VA (any of the VA drafts) does not talk
>> about the same problem. For one it does not stop BGP from
>> advertising anything and second it is intra-domain solution.
>
> Right --the VA drafts increase stretch and don't reduce the total
> table size for anyone.

It never intended to. It reduces the local RIB/FIB size. I am not even 
sure why you are comparing apple to oranges here :)

Best,
R.

---

From: Rob Shakir <[email-masked]>
Date: Tue, 10 Jul 2012 17:27:03 +0100

Hi Shane,

Thanks very much for the detailed review of this document, and apologies it has taken a short while for me to reply.

I've responded to your comments in-line as [rjs].

On 25 Jun 2012, at 22:15, Shane Amante wrote:

> I've read this draft and support it being published as an Informational RFC.
> 
> I do have some comments, which are not blocking, but that I do hope improve the clarity & readability of the draft.
> 
> 1)  In Section 1.1, the following paragraph hints at the problems with current network architectures, but doesn't really come out and say what they are more directly.
> ---snip---
> Traditional network architectures would deploy an Interior Gateway
> Protocol (IGP) to carry infrastructure and customer prefixes, with an
> Exterior Gateway Protocol (EGP) such as BGP being utilised to
> propagate these prefixes to other Autonomous Systems.  [...]
> ---snip---
> IMO, it would be better if the following section more directly said that scale, in terms of the sheer amount of routing information in BGP, has increased substantially, particularly in places like the core of a Route Reflection hierarchy with order _several_ million paths, and rising.  I wish I could point you at a link to a talk Danny & I gave to GROW and/or IDR at IETF 71 (?), but unfortunately I can't find a copy of the slides on the IETF's Web site.  You can see that it's in the GROW WG Agenda at IETF 71, but the slides link to a different presentation: <http://tools.ietf.org/wg/grow/agenda?item=agenda71.html>
> 
> Anyway, IMO, this is (from my vantage point) the single factor that makes outages not only much more visible, but /also/ (just as importantly) take longer to recover from ... (at the end of the day, CPU is finite, there are no "easy" answers wrt using multi-core processors to speed-up convergence of single AFI's, DRAM speeds are not keeping pace, etc.)  This may be useful to mention here to "set the stage", as it were, since throughout the draft there's a continuous (and, good) dialogue of not making convergence times worse or CPU load increase as a result of incorporating better recovery capabilities.  IOW, another benefit you don't really mention anywhere, but probably should, is that /theoretically/ some of these mechanisms may help further promote scalability of BGP.  Obviously, we won't know that until we are evaluating solutions and researchers have time to study them, etc.

[rjs]: I tried to add something to cover this that fits in with Section 1.1:

                        <t>
                            The combination of the increased number of deployments of BGP-4 as an intra-AS routing protocol, its use for the propagation of additional types of routing and service information, and the growth of IP services has resulted in a substantial increase in the volume of information carried within BGP-4. In numerous networks, RIB sizes of the order of millions of entries exist, with particular high-scale points existing at BGP speakers performing aggregation or functionality designed improve utilisation of network resources (e.g., route reflector hierarchies). Whilst clearly an increase in the amount routing information carried in BGP results in greater impact to services during failures, it is also critical to their recovery time. The increased time to compute new paths following a failures and subsequently re-learn them following recoveries results in greater impact of failures within the protocol, and hence adds further weight to the requirement to avoid failures affecting all routing, or service, information carried via a particular adjacency. Whilst an argument could be made the convergence time of BGP-4 can be reduced through additional computational resource being deployed, it is notable that significant challenges continue to exist for operators of scaling BGP-4, and hence mechanisms which improve the scalability of the protocol are of particular note.
                        </t>

[rjs]: Do you think this captures the point that was missing? I tried to avoid making any point that asserts that the requirements defined are going to make BGP scale better, but rather just call out that operationally, throwing resources at the problem isn't something that's always possible, so anything that reduces the amount of "work" BGP needs to do is interesting.

> 
> Related to the above, there is also the following paragraph:
> ---snip---
> Along with this change in role, the nature of the IP routing
> information that is carried has changed.  BGP has become a ubiquitous
> means by which service information can be propagated between devices.
> ---snip---
> ... it may be better to say that the original scope of BGP wrt *just* exchanging reachability information has expanded to include functions that *were* typically in the domain of 'traditional' provisioning activities, and leave it at that.  (You are obviously familiar with things like BGP A-D + VPLS-LDP vs. VPLS-BGP vs. RFC 4364 ... where the former divorces service discovery [provisioning] from signaling/network reachability, whereas the latter two tightly couple service discovery & network reachability).

[rjs]: I'm not quite clear on whether this gets the point across completely - do we think that it is just that things have become in the realm of provisioning activities, or rather is it that there are more and more functions that are overloading onto BGP. I agree that this sentence doesn't necessarily capture that - but do you think that it's the generic information transfer protocol between PEs, as well as replacing provisioning mechanisms?

> 
> 2)  In Section 3, wrt the discussion on draft-ietf-idr-optional-transitive, it may be much more compelling to state that practically this solution has (currently) only been defined to be applicable to 3 out of the 29 total BGP path attributes that have been assigned by IANA, at this time.  And, even if you *could* to go farther, then only 10 out of the total 29 BGP path attributes are optional transitive.  So, the short story is: a lot more work in the area of session preservation
> of iBGP sessions remains to be done, in particular wrt the other types of mandatory, discretionary, etc. attributes ... (assuming that such a thing is even technically possible with BGPv4, as we know it).

[rjs]: Agreed, it is definitely worth highlighting this. I amended the paragraph mentioned to cover this:

                    <t>
                        At the time of writing, error handling mechanisms related to optional, transitive attributes - such as <xref target='I-D.ietf-idr-optional-transitive' /> are restricted to handling only a subset of attribute errors - whereas the operational requirement is to expand this coverage to the widest set of errors possible (i.e., all semantic errors within UPDATE messages). Additionally, where approaches applicable to a greater number of attributes are proposed (e.g., <xref target='I-D.chen-ebgp-error-handing'>), these are limited to deployment in eBGP applications only, where requirements also exist in intra-domain cases. As such, it is envisaged that if extended to cover these expanded cases, these mechanisms provide a means to avoid the transmission of a NOTIFICATION message to a remote BGP speaker, based on a single erroneous message, where at all possible, and hence meet this requirement. Critical errors, including those whereby the NLRI cannot be extracted from the UPDATE message, represent cases whereby the receiving system cannot handle the error gracefully based on this mechanism.
                    </t>

> 
> [ snipped ]
> 
> 4)  In Section 4, I can't really discern a strong (?) preference for one of the associated recommendations to recovering RIB consistency.  I *think* what you should say, more clearly, is there is a strong preference toward narrow, targeted refresh mechanisms covering specific routes over those broader solutions (recovering an entire Adj-RIB-In), to reduce the
> impact on CPU, DRAM, etc.  Perhaps you could add a sentence at the end of the 4th paragraph, where you discuss draft-zeng-one-time-prefix-orf, along the lines of what I just mentioned above?  The good news is you state this very clearly, but much later, in Section 7.  Perhaps you can pull this sentence from Section 7 up to Section 4, as well?
> ---snip---
> [...] It is recommended that where available, any automatic (or
> manual) triggered recovery mechanism behaviour utilises such targeted
> means in preference to any whole RIB refresh mechanism (such as
> ROUTE-REFRESH).
> ---snip--- 

[rjs]: Agreed, yes. I think that there is a strong preference for the targeted mechanism in terms of scale. I rephrased the Section 3 text to be stronger, and left in the Section 7 text:

	<t>
		Whilst re-advertisement of the whole BGP RIB provides a means by which withdrawn NLRI can be re-advertised, there are some scaling implications that must be considered. In the case that a ROUTE-REFRESH is generated, all NLRI must be re-packed into UPDATE messages and advertised by one speaker on the BGP session, whilst the other must receive all UPDATE messages, and validate the RIB's consistency. In order to avoid the control-plane load, it is therefore a requirement to utilise targeted mechanisms where possible, rather than incurring the additional load on both the advertising and receiving speaker of building and processing UPDATEs for the entire contents of the RIB.
	</t>

> 5)  Section 5 seems targeted solely at recreating the data structures at one side of a BGP session, (specifically a receiver of a BGP session).  However, who's to say the corruption is not actually in the Adj-RIB-Out of the BGP transmitter?  Thus, transmitting you an Adj-RIB-Out all over again, even "gracefully", is unlikely to do much good.  Perhaps it would be good to consider whether the "Receiving Speaker", in Graceful Restart terminology, should consider 'gracefully' purging and recreating his Adj-RIB-Out data structures, as well, before re-xmit'ing them to the "Restarting Speaker"?

[rjs]: Absolutely, this definitely wasn't handled in the draft -- thanks for pointing it out. I added the following text - do you think this is sufficient?

                    <t>
                        In some cases, the erroneous condition may be due to corruption of the Adj-RIB-Out on the advertising BGP speaker - rather than caused by the receiving speaker's state. In these cases, where existing structures are replayed whilst performing graceful restart functionality, the error condition is not necessarily resolved. Therefore, it is recommended that during a session restart event, as described within this section, the advertising speaker purge and rebuild RIB structures, in order to resolve any corruption within these structures.
                    </t> 

> 
> 
> 
> 
> Nits:
> 1)  From <http://tools.ietf.org/html/rfc4271#section-3.1>:
> ---snip---
> For the purpose of this protocol, a route is defined as a unit of
> information that pairs a set of destinations with the attributes of a
> path to those destinations.  The set of destinations are systems
> whose IP addresses are contained in one IP address prefix that is
> carried in the Network Layer Reachability Information (NLRI) field of
> an UPDATE message, and the path is the information reported in the
> path attributes field of the same UPDATE message.
> ---snip---
> ... thus, to be more correct, I think you should scrub the document to say "route" in places where you're saying "prefix".  Some examples include:
> 
> http://tools.ietf.org/html/draft-ietf-grow-ops-reqs-for-bgp-error-handling-04#section-1.2
> ---snip---
> o  It is unacceptable within modern deployments of the BGP-4 protocol
>   that a single erroneous UPDATE packet affects prefixes that it
>   does not carry.  This requirement therefore requires some
> ---snip---
> s/affects prefixes/affects routes/
> 
> ---snip---
> this reset of the BGP-4 session results in interruption to
> forwarding packets (by means of withdrawing prefixes installed by
> BGP-4 into a device's RIB, and subsequently FIB).  To this end,
> ---snip---
> s/withdrawing prefixes/withdrawing routes/

[rjs]: Thanks, I've done this to reflect this definition.

> 
> 2)  In Section 2, this may be just wording, but I'm unclear on what you mean in the below where you say: "... handle the error in a manner focused on the NLRI contained within the message".  Are you trying to say something like "constrain the logging and/or ignoring of malformed UPDATE messages specific to the narrowest subset of routes for which malformed path attributes have been received?"  I'm not suggesting my wording is necessarily better, in this case, but perhaps you could look at being a bit more precise below?
> ---snip---
> Where an UPDATE message is considered invalid by a BGP speaker due to
> an error within a path attribute that is not the NLRI (where the
> definition of NLRI includes reachability information encoded in the
> MP_REACH_NLRI and MP_UNREACH_NLRI attributes as specified in
> [RFC4760]) it is a requirement of any enhanced error handling
> mechanism to handle the error in a manner focused on the NLRI
> contained within the message.  
> ---snip---

[rjs]: Yes - the intention is to define this based on the narrowest set possible, the reason that I used this wording is that (in my view) this is defined by the NLRI actually in the message (if there were differing path attributes for NLRI, then we expect that this is packed into a second UPDATE message). Perhaps a hybrid of our wording would clarify this (unless you think the assertion above is erroneous?).

> 
> 3)  Section 2:
> ---snip---
> contained within the message.  Since in this case, the message
> received from the remote peer is syntactically valid, it is
> considered that such an UPDATE is indicative of erroneous data within
> a path attribute.  [...]
> ---snip---
> s/path attribute/path attributes/

[rjs]: Is the point here "one or more path attributes"? I'm not sure I quite understand the nit? :-)

> 
> 4)  In Section 4, there do you really mean "transitive" here or do you instead mean "transient"?
> ---snip---
> It is of particular note for both means of recovering RIB consistency
> described that these are effective only when considering transitive
> errors within an implementation - for instance, should an RFC ...
> ---snip---

[rjs]: Transient, thanks.

> ... also, in the following, do you mean "dynamic" instead of "transitive" in the below?
> ---snip---
> [...] It is not advisable that a transitive
> filter and advertisement mechanism is triggered by all error handling
> events due to the load this is likely to place on the neighbour
> receiving such a request.  Where this BGP speaker is a relatively
> ---snip---

[rjs]: Dynamic is definitely what is meant here.

> 
> 5)  In Section 7.1, I think you probably want to s/NLRI/route/ in several places, e.g.:
> ---snip---
> ... upstream speaker to perform a best-path selection, and re-advertise a
> new set of __NLRI__ before the downstream system is able to converge to a
> new path.
> ---snip---

[rjs]: Thanks, nit corrected.

Many thanks again for your comments - if you could cast your eyes over the above corrections, and let me know if you feel they're sufficient, that'd be fantastic.

In response to all the comments received during this review period, I'll push an -05 of the draft.

Kind regards,
r.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 11 Jul 2012 00:03:30 +0200

/* subject fixed */

Hi Russ,

> We could compare the less specific with the more specific, as you
> say. But it's important that the comparison be done on the inbound
> side, not the outbound side, if you want the option of reducing
> internal table size.

Ok here I completely disagree. Let me try to explain why ...

The tools to solve RIB and FIB scaling do not require to prevent BGP 
from limiting it's table size.

The tools which may help with Internet BGP table grow are much more 
efficient outbound from any AS not inbound. The reason is that there are 
many inbound points which may receive overlapping prefixes.

> I don't think we say it has to come in on the same session, or even
> from the same entry point --only that the overlapping routes have to
> exist in the same table at some point for comparison. You could
> always do this at the outbound ASBR, but that's not the most
> efficient place to do it. The most efficient place is the inbound
> ASBR, I think.

As above I think we clearly disagree on that point ;)

> You said PI space first,

Sorry yes I meant PA in the first place as well.

> not PA... With PA, if a provider is stupid
> enough to give the inbound traffic to one of their customers to a
> competing provider, then go ahead.

It will not ! Because wise provider do not suppress anything on inbound 
within his network he carries his more specific prefixes just fine. It 
is just outbound to his upstream he advertises his single PA space. 4

> I can't imagine doing that as a
> provider. If I'm providing the PA space, I want to be the primary
> entry point into the customer's network.

And you still are .. of course provided that customer will not convince 
some other ISP with sufficient $$$ to advertise part of your PA space.

That's why RIRs were recognizing this as an important enough issue that 
such multimedia customer were getting Provider Independent address. Of 
course if we finally converge on how to do multihoming with PA addresses 
it would be much better.

> Why bother with "prefer local routes to my customers over routes from
> peers," if I'm going to give out a set of longer prefixes that
> customer can advertise through another provider, drawing traffic from
> within my own network across a transit link into a peer to reach that
> same customer?
>
> But note:
>
>> should also be advertised.
>
> A covering aggregate should also be advertised. In other words, if
> you're advertising the longer prefix, you _should_ advertise a
> shorter covering route --specifically what we're counting on in this
> draft. So it looks like the draft is already in line with what best
> common practice is. :-)

You looked at the exception not the rule :)

r.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 11 Jul 2012 01:13:06 +0200

Hi Russ,

> 1. A provider who is giving a customer PA space in parts, but that PA
> space is not being advertised anyplace other than to that provider.
> In this case, pulling traffic overlapping routes doesn't matter,
> because the provider who allows the use of PA space is going to take
> them out by not advertising the longer prefixes from their own space.
> In this case, the draft we've proposed would work, as well.

This is done today without the need for your draft. That's common 
practice :)

> 2. A provider who is giving a customer PA space and allowing the
> customer to advertise that same PA space through another provider. In
> this case, if the provider is not advertising the PA space at the
> same length as the customer is to an alternate provider, the
> alternate provider will consume all the inbound traffic to the
> customer. This is dumb on the part of the provider giving out the PA
> space.

What is the provider supposed to do ? Give ticket to his customer ? Get 
rid of his customer ?

See the PA owner can not do much not to break the business relation with 
his customer. However his (and another providers) upstreams can suppress 
the more specifics.

This is where the aim should be. Moreover the more specific should be 
suppressed only where it brings no more value.

> 3. A customer who has PI space and is advertising different bits in
> different places. Again, best practice would say to advertise a
> shorter covering route as well, to guard against failures. If the end
> AS is advertising longer and shorter prefixes that overlap, the draft
> we've proposed works --even if they're advertising them in different
> places, what we've proposed will take the longer prefix out once it
> no longer draws traffic along a different path than the shorter
> prefix.

I don't think we can do much with PI addresses.

> So, I'm a bit confused as to what case you think isn't covered... I
> thought you were saying it was normal operation for a provider to
> give out PA space, allow the customer to break that space up, and

I don not understand what do you mean by "allow" If I peer with you and 
you give me /24 out of your /16 then I pay Alvaro $$$ then he will be 
happy to advertise this /24. What can you do about that ? Be mad at both 
of us ?

> then only advertise the shorter block covering the space given to the
> customer. I don't care how you jiggle local pref if you do this,
> longer prefix wins, and the alternate provider consumes the traffic
> --even from your AS.

Nope. Not from your AS as as I said in the last email you will carry 
within your AS my /24 and prefer customer routes rather then peer routes 
(assuming Alvaro is even peering with you)

;-)
R.

---

From: "White, Russell" <[email-masked]>
Date: Mon, 9 Jul 2012 23:57:19 +0000

==
                     (   )
            /-------( AS2 )--------\
     (   ) /         (   )          \ (   )       (   )

==

First, AS4 could see the overlap on the outbound side, note the origin AS is the same, and set the longer prefix to NO_EXPORT.

Alternatively, the BGP implementation in AS4 could be modified so that when running bestpath if a losing route has a specific community, it is transferred to the winning route. This would allow the winning route to gain the BOUNDED community in processing.

==
==

I rarely see this --because most of the time, folks want to "cover" the longer match with a shorter one, in case the longer prefix fails for some reason. With a covering route, both ISPs will be used for inbound traffic, and not just the one with the remaining longer prefix. This has been the recommended configuration for years, AFAIK.

==
==

I prefer the simpler solution presented in the draft we just published. Draft-marquis and the virtual aggregation drafts both add a lot of complexity to operations, while the BOUNDED community is very simple, and can (mostly) be implemented today with no changes to BGP, no changes to current traffic engineering, and no increased stretch within the SP network. Complexity can always be added, but a simple system on which future modifications can be made to cover corner cases seems like a better idea to me... :-)

:-)

Russ

---

From: "White, Russell" <[email-masked]>
Date: Tue, 10 Jul 2012 14:24:23 +0000

> would work. The choice/decision what to advertise and what not must be

Sure --but won't best path be run when the best path towards a specific destination falls out of the table?

> marking. How about we consider even much simpler and ASBR local solution:

A local ASBR only solution won't work for various reasons, so let's not. :-)

> blocks get advertised via some other ISPs, but never directly connected.

So what you are saying is that if a provider owns 10.1.0.0/16, and gives out 10.1.1.0/24 to a customer to advertise through some other provider the customer is connected to, the provider would never advertise 10.1.1.0/24 as well as 10.1.0.0/16? Why not? Since the longest match always wins, what you're saying is that the provider that handed out the address space would end up routing through the competing provider (the customer's other provider) to reach the customer, because there is a longer match being advertised by the customer through the competing provider.

> and second it is intra-domain solution.

Right --the VA drafts increase stretch and don't reduce the total table size for anyone. In return, you must inject a good bit of complexity, and consider the various problems involved with routing traffic on a smaller number of links through your network. 

I prefer the simpler solution. And yes, it is simpler, even if it doesn't solve (outright) every possible problem in the space.

:-)

Russ

---

From: "White, Russell" <[email-masked]>
Date: Tue, 10 Jul 2012 21:44:35 +0000

> Best path today does not compare /16 and /24. In Pedro's proposal it does.

We could compare the less specific with the more specific, as you say. But it's important that the comparison be done on the inbound side, not the outbound side, if you want the option of reducing internal table size.

> that actually happens that often as you say :-)

I don't think we say it has to come in on the same session, or even from the same entry point --only that the overlapping routes have to exist in the same table at some point for comparison. You could always do this at the outbound ASBR, but that's not the most efficient place to do it. The most efficient place is the inbound ASBR, I think.

> not make the BGP table smaller.

You said PI space first, not PA... With PA, if a provider is stupid enough to give the inbound traffic to one of their customers to a competing provider, then go ahead. I can't imagine doing that as a provider. If I'm providing the PA space, I want to be the primary entry point into the customer's network.

Why bother with "prefer local routes to my customers over routes from peers," if I'm going to give out a set of longer prefixes that customer can advertise through another provider, drawing traffic from within my own network across a transit link into a peer to reach that same customer? 

> should also be advertised.

A covering aggregate should also be advertised. In other words, if you're advertising the longer prefix, you _should_ advertise a shorter covering route --specifically what we're counting on in this draft. So it looks like the draft is already in line with what best common practice is. :-)

> sure why you are comparing apple to oranges here :)

I didn't bring it up. :-)

:-)

Russ

---

From: "White, Russell" <[email-masked]>
Date: Tue, 10 Jul 2012 22:54:34 +0000

> some other ISP with sufficient $$$ to advertise part of your PA space.

Okay, there are three cases I can see:

1. A provider who is giving a customer PA space in parts, but that PA space is not being advertised anyplace other than to that provider. In this case, pulling traffic overlapping routes doesn't matter, because the provider who allows the use of PA space is going to take them out by not advertising the longer prefixes from their own space. In this case, the draft we've proposed would work, as well.

2. A provider who is giving a customer PA space and allowing the customer to advertise that same PA space through another provider. In this case, if the provider is not advertising the PA space at the same length as the customer is to an alternate provider, the alternate provider will consume all the inbound traffic to the customer. This is dumb on the part of the provider giving out the PA space.

3. A customer who has PI space and is advertising different bits in different places. Again, best practice would say to advertise a shorter covering route as well, to guard against failures. If the end AS is advertising longer and shorter prefixes that overlap, the draft we've proposed works --even if they're advertising them in different places, what we've proposed will take the longer prefix out once it no longer draws traffic along a different path than the shorter prefix.

So, I'm a bit confused as to what case you think isn't covered... I thought you were saying it was normal operation for a provider to give out PA space, allow the customer to break that space up, and then only advertise the shorter block covering the space given to the customer. I don't care how you jiggle local pref if you do this, longer prefix wins, and the alternate provider consumes the traffic --even from your AS.

:-)

Russ

---

From: "White, Russell" <[email-masked]>
Date: Wed, 11 Jul 2012 00:07:01 +0000

> What is the provider supposed to do ? Give ticket to his customer ? Get

Advertise the same length route as the other provider is... That's what I would do.

However, if we're blocking the longer prefix route when it no longer impacts traffic flow, then the alternate provider will block the longer prefix when he receives the shorter from the owner of the PA space.

Again, this is all about blocking longer prefixes when they are no longer useful in the routing system as a whole. What we've proposed says that seeing two overlapping routes is the best indication of this condition, and attempts to strike a balance between traffic engineering and reducing table size.

> I don't think we can do much with PI addresses.

You can remove unneeded longer prefix traffic engineering routes with PI space just like you can PA space.

:-)

Russ

---

From: Shane Amante <[email-masked]>
Date: Wed, 11 Jul 2012 10:50:26 -0600

Hi Rob,

> Thanks very much for the detailed review of this document, and apologies it has taken a short while for me to reply.

No problem, on both counts.  See below.

>                            The combination of the increased number of deployments of BGP-4 as an intra-AS routing protocol, its use for the propagation of additional types of routing and service in…

The above looks good, but I've made some minor modifications.  See below.
---snip---
The combination of the increased number of deployments of BGP-4 as an intra-AS routing protocol, its use for the propagation of additional types of routing and service information, and the growth of IP services has resulted in a substantial increase in the volume of information carried within BGP-4. In numerous networks, RIB sizes of the order of millions of entries exist within individual BGP speakers, with particularly high-scale points exhibited at BGP speakers performing aggregation or functionality designed improve utilisation of network resources (e.g., route reflector hierarchies). Whilst clearly an increase in the amount routing information carried in BGP results in greater impact to services during failures, which is only amplified by a corresponding increase in recovery times. Following a failure, there is a substantial recovery time to learn, compute and distribute new paths, which results in a greater observed impact to services affected, and hence adds further weight to the requirement to avoid failures altogether or, at least, mitigate their impact to the narrowest scope possible, (e.g.: a specific NLRI). Whilst an argument could be made that convergence time of BGP-4 could potentially be reduced through deployment of additional computational resource, it is notable that solution is not necessarily straightforward from an implementation or deployment point-of-view, (e.g.: scaling computation resources within a single address-family is difficult).  Thus, significant challenges continue to exist for operators when scaling BGP-4 deployments, and hence mechanisms which improve the scalability of BGP-4 are very important.
---snip---

> [rjs]: Do you think this captures the point that was missing? I tried to avoid making any point that asserts that the requirements defined are going to make BGP scale better, but rather just call o…

I definitely agree.

> [rjs]: I'm not quite clear on whether this gets the point across completely - do we think that it is just that things have become in the realm of provisioning activities, or rather is it that there…

I believe that you are correct, and better off, in stating "more and more functions that are overloaded (sic) onto BGP".  Although, I'm not sure that "overloaded" is an appropriate adjective.  

The point I was trying to get at is as follows.  I think there's a continuum of information exchanged within BGP from real-time information (reachability) to less dynamic (perhaps, even static) information, with _examples_ of the latter being auto-discovery/provisioning use cases.  While traditional applications, such as vanilla Internet service for which BGP was originally designed, only fall into the "real-time information" category ... there are a lot of new(er) applications that do not fit "neatly" in a single category and, in fact, span the range of real-time to less dynamic categories depending on which facet of a particular protocol you look at, (examples being: IPVPN, MVPN, VPLS-BGP, etc.).  Regardless, I don't think it's prudent to make value judgements (particularly at this point in time when these protocols are already widely deployed and successful) as to the "correctness" of these functions/services being in BGP, since that's bound to be very subjective.  Rather, we need to recognize the world for what it is today, which is why I think use of the word "overloaded" may be inappropriate.  Furthermore, I think that talking about this in such a context is only recognizing a symptom (the more complex the system, the higher the probability is to introduce errors), when in reality we should be trying to focus in on the root problem: since we've put so many eggs in one basket, we need unnoticeable (or, faster) recovery from errors that affect real-time, reachability information.

>                        At the time of writing, error handling mechanisms related to optional, transitive attributes - such as <xref target='I-D.ietf-idr-optional-transitive' /> are restricted to ha…

OK.

> 		Whilst re-advertisement of the whole BGP RIB provides a means by which withdrawn NLRI can be re-advertised, there are some scaling implications that must be considered. In the case that a ROUTE-R…

OK.

>                        In some cases, the erroneous condition may be due to corruption of the Adj-RIB-Out on the advertising BGP speaker - rather than caused by the receiving speaker's state. In th…

OK.

> [rjs]: Yes - the intention is to define this based on the narrowest set possible, the reason that I used this wording is that (in my view) this is defined by the NLRI actually in the message (if th…

I see your point now.  How about the following hybrid text?
---snip---
... it is a requirement of any enhanced error handling mechanism to constrain the error handling so that it is narrowly focused on the NLRI contained within the bad UPDATE message.
---snip---

> [rjs]: Is the point here "one or more path attributes"? I'm not sure I quite understand the nit? :-)

Yes, sorry: "one or more path attributes".  (My point was you can't predict, here anyway, that it will only a single path attribute that is a problem.  Ideally, a more robust error-handling solution would not make such assumptions :-).

> Many thanks again for your comments - if you could cast your eyes over the above corrections, and let me know if you feel they're sufficient, that'd be fantastic.

And, thank you Rob for your excellent work on this.

-shane

---

From: Rob Shakir <[email-masked]>
Date: Wed, 18 Jul 2012 18:54:43 +0100

Hi Shane,

Thanks for the comments again, and apologies (again!) for the delay in responding.

Please find my responses in-line as [rjs].

On 11 Jul 2012, at 17:50, Shane Amante wrote:

>>> [...snip...]

[rjs]: Thanks, other than some minor editorial changes I adopted this paragraph -- it seems like a good hybrid.

>>> [...snip...]

[rjs]: I guess there may be negative connotations of 'overloaded', I guess what I really mean is maybe "layered" onto BGP -- poor wording perhaps.

> The point I was trying to get at is as follows.  I think there's a continuum of information exchanged within BGP from real-time information (reachability) to less dynamic (perhaps, even static) inf…

[rjs]: Completely agree with this. I think my poor choice of wording perhaps portrayed my view as negative -- rather, the key point for me is that the robustness and error handling that we are discussing here is designed with the vanilla Internet service as the baseline - and as we extend the protocol to different deployment cases (no judgement about the value of which is made), then some of the initial assumptions perhaps don't hold true. I think this is in agreement with yourself, insofar that I think we would both assert that for the real-time information, potentially the behaviour required in a number of areas of the protocol is not the same as the behaviour required for relatively static information.

> ... it is a requirement of any enhanced error handling mechanism to constrain the error handling so that it is narrowly focused on the NLRI contained within the bad UPDATE message.

[rjs]: Sure, this sounds good.

> Yes, sorry: "one or more path attributes".  (My point was you can't predict, here anyway, that it will only a single path attribute that is a problem.  Ideally, a more robust error-handling solutio…

[rjs]: ACK, updated this to 'one or more' :-)

> And, thank you Rob for your excellent work on this.

[rjs]: No worries - I'll take a read through and submit an -05 of the draft that merges the edits we've discussed in this thread.

Thanks again for the comments,
r.

---

From: Christopher Morrow <[email-masked]>
Date: Sun, 22 Jul 2012 22:17:15 -0400

Rob,
Did you want to spin a new version of the draft and get final comments
from Shane? then move this along to IESG-land?

Or are there still comments/issues to deal with from other folk? (the
russ/robert discussion seemed to peter out as well)

-chris

On Wed, Jul 18, 2012 at 1:54 PM, Rob Shakir <[email-masked]> wrote:
>>>                           The combination of the increased number of deployments of BGP-4 as an intra-AS routing protocol, its use for the propagation of additional types of routing and service information, and the growth of IP services has resulted in a substantial increase in the volume of information carried within BGP-4. In numerous networks, RIB sizes of the order of millions of entries exist, with particular high-scale points existing at BGP speakers performing aggregation or functionality designed improve utilisation of network resources (e.g., route reflector hierarchies). Whilst clearly an increase in the amount routing information carried in BGP results in greater impact to services during failures, it is also critical to their recovery time. The increased time to compute new paths following a failures and subsequently re-learn them following recoveries results in greater impact of failures within the protocol, and hence adds further weight to the requirement to
>  avoid failures affecting all routing, or service, information carried via a particular adjacency. Whilst an argument could be made the convergence time of BGP-4 can be reduced through additional computational resource being deployed, it is notable that significant challenges continue to exist for operators of scaling BGP-4, and hence mechanisms which improve the scalability of the protocol are of particular note.
>>>                       </t>
>>
>>
>> The combination of the increased number of deployments of BGP-4 as an intra-AS routing protocol, its use for the propagation of additional types of routing and service information, and the growth of IP services has resulted in a substantial increase in the volume of information carried within BGP-4. In numerous networks, RIB sizes of the order of millions of entries exist within individual BGP speakers, with particularly high-scale points exhibited at BGP speakers performing aggregation or functionality designed improve utilisation of network resources (e.g., route reflector hierarchies). Whilst clearly an increase in the amount routing information carried in BGP results in greater impact to services during failures, which is only amplified by a corresponding increase in recovery times. Following a failure, there is a substantial recovery time to learn, compute and distribute new paths, which results in a greater observed impact to services affected, and hence adds further
>  weight to the requirement to avoid failures altogether or, at least, mitigate their impact to the narrowest scope possible, (e.g.: a specific NLRI). Whilst an argument could be made that convergence time of BGP-4 could potentially be reduced through deployment of additional computational resource, it is notable that solution is not necessarily straightforward from an implementation or deployment point-of-view, (e.g.: scaling computation resources within a single address-family is difficult).  Thus, significant challenges continue to exist for operators when scaling BGP-4 deployments, and hence mechanisms which improve the scalability of BGP-4 are very important.
>> ---snip---
>
>> The point I was trying to get at is as follows.  I think there's a continuum of information exchanged within BGP from real-time information (reachability) to less dynamic (perhaps, even static) information, with _examples_ of the latter being auto-discovery/provisioning use cases.  While traditional applications, such as vanilla Internet service for which BGP was originally designed, only fall into the "real-time information" category ... there are a lot of new(er) applications that do not fit "neatly" in a single category and, in fact, span the range of real-time to less dynamic categories depending on which facet of a particular protocol you look at, (examples being: IPVPN, MVPN, VPLS-BGP, etc.).  Regardless, I don't think it's prudent to make value judgements (particularly at this point in time when these protocols are already widely deployed and successful) as to the "correctness" of these functions/services being in BGP, since that's bound to be very subjective.  Rathe
>  r, we need to recognize the world for what it is today, which is why I think use of the word "overloaded" may be inappropriate.  Furthermore, I think that talking about this in such a context is only recognizing a symptom (the more complex the system, the higher the probability is to introduce errors), when in reality we should be trying to focus in on the root problem: since we've put so many eggs in one basket, we need unnoticeable (or, faster) recovery from errors that affect real-time, reachability information.
>
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

---

From: <[email-masked]>
Date: Mon, 23 Jul 2012 16:08:53 +0100

Hi Chris,

Sure - I will push an -05 version with the changes that were highlighted
this week (including integrating the changes that Shane and I discussed in
this thread). It would be great to progress this draft to the IESG
following this.

I don't believe that there are other comments to deal with. The discussion
between Russ and Robert did not relate to this draft (but was rather a
discussion of draft-white-grow-overlapping-routes-00 with the wrong
subject line).

Many thanks,
r.

On 23/07/2012 03:17, "Christopher Morrow" <[email-masked]>
wrote:

>>>>                           The combination of the increased number of
>>>>deployments of BGP-4 as an intra-AS routing protocol, its use for the
>>>>propagation of additional types of routing and service information,
>>>>and the growth of IP services has resulted in a substantial increase
>>>>in the volume of information carried within BGP-4. In numerous
>>>>networks, RIB sizes of the order of millions of entries exist, with
>>>>particular high-scale points existing at BGP speakers performing
>>>>aggregation or functionality designed improve utilisation of network
>>>>resources (e.g., route reflector hierarchies). Whilst clearly an
>>>>increase in the amount routing information carried in BGP results in
>>>>greater impact to services during failures, it is also critical to
>>>>their recovery time. The increased time to compute new paths following
>>>>a failures and subsequently re-learn them following recoveries results
>>>>in greater impact of failures within the protocol, and hence adds
>>>>further weight to the requirement to
>>  avoid failures affecting all routing, or service, information carried
>>via a particular adjacency. Whilst an argument could be made the
>>convergence time of BGP-4 can be reduced through additional
>>computational resource being deployed, it is notable that significant
>>challenges continue to exist for operators of scaling BGP-4, and hence
>>mechanisms which improve the scalability of the protocol are of
>>particular note.
>>>>                       </t>
>>>
>>>
>>> The combination of the increased number of deployments of BGP-4 as an
>>>intra-AS routing protocol, its use for the propagation of additional
>>>types of routing and service information, and the growth of IP services
>>>has resulted in a substantial increase in the volume of information
>>>carried within BGP-4. In numerous networks, RIB sizes of the order of
>>>millions of entries exist within individual BGP speakers, with
>>>particularly high-scale points exhibited at BGP speakers performing
>>>aggregation or functionality designed improve utilisation of network
>>>resources (e.g., route reflector hierarchies). Whilst clearly an
>>>increase in the amount routing information carried in BGP results in
>>>greater impact to services during failures, which is only amplified by
>>>a corresponding increase in recovery times. Following a failure, there
>>>is a substantial recovery time to learn, compute and distribute new
>>>paths, which results in a greater observed impact to services affected,
>>>and hence adds further
>>  weight to the requirement to avoid failures altogether or, at least,
>>mitigate their impact to the narrowest scope possible, (e.g.: a specific
>>NLRI). Whilst an argument could be made that convergence time of BGP-4
>>could potentially be reduced through deployment of additional
>>computational resource, it is notable that solution is not necessarily
>>straightforward from an implementation or deployment point-of-view,
>>(e.g.: scaling computation resources within a single address-family is
>>difficult).  Thus, significant challenges continue to exist for
>>operators when scaling BGP-4 deployments, and hence mechanisms which
>>improve the scalability of BGP-4 are very important.
>>> ---snip---
>>
>> [rjs]: Thanks, other than some minor editorial changes I adopted this
>>paragraph -- it seems like a good hybrid.
>>
>>
>>>>> [...snip...]
>>>>
>>>> [rjs]: I'm not quite clear on whether this gets the point across
>>>>completely - do we think that it is just that things have become in
>>>>the realm of provisioning activities, or rather is it that there are
>>>>more and more functions that are overloading onto BGP. I agree that
>>>>this sentence doesn't necessarily capture that - but do you think that
>>>>it's the generic information transfer protocol between PEs, as well as
>>>>replacing provisioning mechanisms?
>>>
>>> I believe that you are correct, and better off, in stating "more and
>>>more functions that are overloaded (sic) onto BGP".  Although, I'm not
>>>sure that "overloaded" is an appropriate adjective.
>>
>> [rjs]: I guess there may be negative connotations of 'overloaded', I
>>guess what I really mean is maybe "layered" onto BGP -- poor wording
>>perhaps.
>>
>>> The point I was trying to get at is as follows.  I think there's a
>>>continuum of information exchanged within BGP from real-time
>>>information (reachability) to less dynamic (perhaps, even static)
>>>information, with _examples_ of the latter being
>>>auto-discovery/provisioning use cases.  While traditional applications,
>>>such as vanilla Internet service for which BGP was originally designed,
>>>only fall into the "real-time information" category ... there are a lot
>>>of new(er) applications that do not fit "neatly" in a single category
>>>and, in fact, span the range of real-time to less dynamic categories
>>>depending on which facet of a particular protocol you look at,
>>>(examples being: IPVPN, MVPN, VPLS-BGP, etc.).  Regardless, I don't
>>>think it's prudent to make value judgements (particularly at this point
>>>in time when these protocols are already widely deployed and
>>>successful) as to the "correctness" of these functions/services being
>>>in BGP, since that's bound to be very subjective.  Rath
> e
>>  r, we need to recognize the world for what it is today, which is why I
>>think use of the word "overloaded" may be inappropriate.  Furthermore, I
>>think that talking about this in such a context is only recognizing a
>>symptom (the more complex the system, the higher the probability is to
>>introduce errors), when in reality we should be trying to focus in on
>>the root problem: since we've put so many eggs in one basket, we need
>>unnoticeable (or, faster) recovery from errors that affect real-time,
>>reachability information.
>>
>> [rjs]: Completely agree with this. I think my poor choice of wording
>>perhaps portrayed my view as negative -- rather, the key point for me is
>>that the robustness and error handling that we are discussing here is
>>designed with the vanilla Internet service as the baseline - and as we
>>extend the protocol to different deployment cases (no judgement about
>>the value of which is made), then some of the initial assumptions
>>perhaps don't hold true. I think this is in agreement with yourself,
>>insofar that I think we would both assert that for the real-time
>>information, potentially the behaviour required in a number of areas of
>>the protocol is not the same as the behaviour required for relatively
>>static information.
>>
>>>>
>>>> [rjs]: Yes - the intention is to define this based on the narrowest
>>>>set possible, the reason that I used this wording is that (in my view)
>>>>this is defined by the NLRI actually in the message (if there were
>>>>differing path attributes for NLRI, then we expect that this is packed
>>>>into a second UPDATE message). Perhaps a hybrid of our wording would
>>>>clarify this (unless you think the assertion above is erroneous?).
>>>
>>> ... it is a requirement of any enhanced error handling mechanism to
>>>constrain the error handling so that it is narrowly focused on the NLRI
>>>contained within the bad UPDATE message.
>>> ---snip---
>>
>>> Yes, sorry: "one or more path attributes".  (My point was you can't
>>>predict, here anyway, that it will only a single path attribute that is
>>>a problem.  Ideally, a more robust error-handling solution would not
>>>make such assumptions :-).
>>
>>>> Many thanks again for your comments - if you could cast your eyes
>>>>over the above corrections, and let me know if you feel they're
>>>>sufficient, that'd be fantastic.
>>>
>> [rjs]: No worries - I'll take a read through and submit an -05 of the
>>draft that merges the edits we've discussed in this thread.
>>

---

From: Christopher Morrow <[email-masked]>
Date: Mon, 23 Jul 2012 14:15:45 -0400

> this thread). It would be great to progress this draft to the IESG

sounds good to me, I think submissions may be blocked for a bit while
ietf-meeting-start happens... but once the new version goes in I'll
send (please ping me to remind me when it submits) iesg publication
requests out.

> discussion of draft-white-grow-overlapping-routes-00 with the wrong

ah, that was my feeling as well... go thread overlap!

-chris