Subject: Re: [Idr] Possible usecases of one-time selective route refresh

From: Rob Shakir <[email-masked]>
Date: Sun, 14 Nov 2010 22:49:22 +0000

[...Split from the draft-chen-ebgp-error-handling thread for readability...]

On 11 Nov 2010, at 14:47, Jie Dong wrote:

> Also there can be other usecases of these lightweight maintenance tool. More thoughts on the usecases would be appreciated.

Hi Jie,

From my point of view, there's a set of problems that aren't considered in these use cases, that is probably quite useful from an operator point of view.

In VPNv4 routing implementations, we have a set of local VRFs configured on a PE, which have specific import criteria based on route-targets. To keep each VRF RIB consistent is not a problem based on receiving an initial copy of the VPNv4 RIB, however, on a policy change, a problem exists in ensuring that the local VRF's tables are consistent with the new import policy (and as such, from a customer's perspective, that their local L3VPN actually reflects the changes they requested on all PE devices they connect to). This problem cannot be solved locally, in general, due to the fact that most PE implementations do not hold the Adj-RIB-In in memory, due to resource constraints - inbound filtering is performed based on imported RTs.

At the moment, to apply such a policy change, one can either stipulate that VRF tables are only consistent based on newly received UPDATEs (i.e. as each remote PE re-advertises prefixes), or initiate a Route Refresh. At the current time, this requires the remote neighbour to re-advertise the complete RIB. Since the remote speaker is typically a route reflector in many SP deployments, this can result in 600K+ prefixes being re-sadvertised to the PE (most of which will be re-discarded due to inbound filtering based on imported route-targets). Therefore, from our perspective, if the local device can perform a route refresh based on the specific import criteria for the VRF that has changed policy, this constrains the number of prefixes re-advertised to the PE from the RR to the number that really require refresh to ensure that the local RIB is consistent. This is a significant advantage over the current behaviour.

Whilst this problem could be solved with ORF at the current time - this does not achieve the same bound on the problem - specific PEs, for example those supporting NNI, may import a large subset of the VPNv4 RIB for the AS - and hence the general ORF policy from the RR to the PE may still result in a refresh that results in a large number of prefixes being sent.

From a resource consumption perspective, since the PE has generated the ORF, then the RR needs only to process this filter, and apply it to egress prefixes. If we assume that building the UPDATEs is the most taxing activity in the re-advertisement of routes to the PE, then this behaviour should result in a net CPU reduction for the devices.

For this reason, I would see this draft as a useful addition to the protocol.

Kind regards,
Rob

---

From: John Scudder <[email-masked]>
Date: Sun, 14 Nov 2010 19:38:04 -0800

Rob,

This particular case can be addressed -- rather more straightforwardly IMO -- using RFC 4684, Constrained Route Distribution (so called "RT-Constrain").  

FWIW,

--John

---

From: Jie Dong <[email-masked]>
Date: Mon, 15 Nov 2010 12:00:42 +0800

Hi Rob,

Thanks a lot for providing this use case. We also heard from some other
operators about similar requirement -- refresh routes of specific VPN.

We would consider adding this into next version draft. Thanks.

Best regards,
Jie

> -----Original Message-----
> From: Rob Shakir [mailto:[email-masked]]
> Sent: Monday, November 15, 2010 6:49 AM
> To: [email-masked]; Jie Dong
> Subject: Re: [Idr] Possible usecases of one-time selective route refresh
> 
> > Also there can be other usecases of these lightweight maintenance tool.
> More thoughts on the usecases would be appreciated.
> 
> Hi Jie,
> 
> From my point of view, there's a set of problems that aren't considered in
> these use cases, that is probably quite useful from an operator point of
view.
> 
> In VPNv4 routing implementations, we have a set of local VRFs configured
on
> a PE, which have specific import criteria based on route-targets. To keep
each
> VRF RIB consistent is not a problem based on receiving an initial copy of
the
> VPNv4 RIB, however, on a policy change, a problem exists in ensuring that
> the local VRF's tables are consistent with the new import policy (and as
such,
> from a customer's perspective, that their local L3VPN actually reflects
the
> changes they requested on all PE devices they connect to). This problem
> cannot be solved locally, in general, due to the fact that most PE
> implementations do not hold the Adj-RIB-In in memory, due to resource
> constraints - inbound filtering is performed based on imported RTs.
> 
> At the moment, to apply such a policy change, one can either stipulate
that
> VRF tables are only consistent based on newly received UPDATEs (i.e. as
each
> remote PE re-advertises prefixes), or initiate a Route Refresh. At the
current
> time, this requires the remote neighbour to re-advertise the complete RIB.
> Since the remote speaker is typically a route reflector in many SP
> deployments, this can result in 600K+ prefixes being re-sadvertised to the
PE
> (most of which will be re-discarded due to inbound filtering based on
> imported route-targets). Therefore, from our perspective, if the local
device
> can perform a route refresh based on the specific import criteria for the
VRF
> that has changed policy, this constrains the number of prefixes
re-advertised
> to the PE from the RR to the number that really require refresh to ensure
> that the local RIB is consistent. This is a significant advantage over the
> current behaviour.
> 
> Whilst this problem could be solved with ORF at the current time - this
does
> not achieve the same bound on the problem - specific PEs, for example
those
> supporting NNI, may import a large subset of the VPNv4 RIB for the AS -
and
> hence the general ORF policy from the RR to the PE may still result in a
> refresh that results in a large number of prefixes being sent.
> 
> From a resource consumption perspective, since the PE has generated the
> ORF, then the RR needs only to process this filter, and apply it to egress
> prefixes. If we assume that building the UPDATEs is the most taxing
activity in
> the re-advertisement of routes to the PE, then this behaviour should
result in
> a net CPU reduction for the devices.
> 
> Rob=

---

From: zengqing <[email-masked]>
Date: Mon, 15 Nov 2010 12:06:09 +0800

Hi John and Rob,

As my understanding, the particular case Rob mentioned may be not changing imported RTs but changing inbound filtering for some local VRF. If it's true, this problem might be resolved by One-time RT based ORF.

Regards
Qing

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of John Scudder
Sent: Monday, November 15, 2010 11:38 AM
To: Rob Shakir
Cc: [email-masked]
Subject: Re: [Idr] Possible usecases of one-time selective route refresh

Rob,

FWIW,

--John

> For this reason, I would see this draft as a useful addition to the protocol.

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: John Scudder <[email-masked]>
Date: Sun, 14 Nov 2010 20:31:21 -0800

Qing,

My impression was different, based on Rob's comment that "inbound filtering is performed based on imported RTs".  Anyway, he can clarify as needed.

Regards,

--John

---

From: Rob Shakir <[email-masked]>
Date: Wed, 17 Nov 2010 13:42:02 +0000

> This particular case can be addressed -- rather more straightforwardly IMO -- using RFC 4684, Constrained Route Distribution (so called "RT-Constrain").  

Hi John, All,

I agree that RFC 4684 can solve this problem to some extent, however, I think there are perhaps two advantages to having a single-use ORF for this case:

- We handle the case whereby there is some inbound filtering for particular VRFs - at the moment there is an arbitrary benchmark set that we can discard based on RT - however, I wouldn't see it as unreasonable if a prefix learnt from an RR was discarded should it not be imported into a VRF (after all, this is what we appear to be implying by discarding based on RT). This allows us to reduce the memory footprint on the PE by always discarding routes that are not imported as a candidate path into any VRF, since we would have a clear mechanism to refresh a specific VRF's table on a policy change - I'm not sure I see how this is solved by RT-Constrain, unless there's a change in the RTs that we now advertise we are importing, RT constrain does not result in a set of UPDATEs. Perhaps you can correct me here?

- We remove the requirement that the RR holds state for this information - at the moment, as is mentioned in Section 7 of the draft, there is now a requirement for our RR to scale above the number of routes in the network, and maintain sufficient CPU and memory resources to store an RT-based topology map. Whilst the suggestion that one could introduce smaller clusters within an AS - I'm not sure that this scales sensibly to relatively large, diverse networks. It certainly introduces operational complexity to determining which device in a AS was responsible for erroneous advertisements, or route selection. If we use a single-time ORF field, this would mean that the RR has a transitive state that it holds for the duration of one, relatively small, set of UPDATEs. Since we'd expect that the end PE is able to maintain some form of admission control of these requests (as it must with Route Refresh currently), this results in a much lower load for the centralised RR resource.

Hopefully these points make sense, if not, please let me know, or correct my understanding of 4684!

Many thanks,
Rob

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 17 Nov 2010 15:09:03 +0100

Hi Rob,

You are incorrect on first point, but very good observation reg the 
second point you are bringing.

In first point RT-Constrain update when new RTs are added to PEs will 
result in RR sending only those VPN routes where such RTs are attached. 
That is as a matter of fact one of the main benefits of rt-constrain.

So one time refresh would be completely overlapping with RT-Constrain 
update message from PE to RR. And once you are using RT-Constrain new 
RTs must be advertised by RT-Constrain.

Your second point is very correct. If network does not support 
RT-Constrain use of selective one time refresh may be much better 
solution to get an incremental routes from RR rather then get full damp 
of perhaps million VPN routes with full refresh request. I would sign 
under such proposal with both hands.

So to summarize I do support one time refresh draft.

Many thx,
R.

>> This particular case can be addressed -- rather more
>> straightforwardly IMO -- using RFC 4684, Constrained Route
>> Distribution (so called "RT-Constrain").
>
> Hi John, All,
>
> I agree that RFC 4684 can solve this problem to some extent, however,
> I think there are perhaps two advantages to having a single-use ORF
> for this case:
>
> - We handle the case whereby there is some inbound filtering for
> particular VRFs - at the moment there is an arbitrary benchmark set
> that we can discard based on RT - however, I wouldn't see it as
> unreasonable if a prefix learnt from an RR was discarded should it
> not be imported into a VRF (after all, this is what we appear to be
> implying by discarding based on RT). This allows us to reduce the
> memory footprint on the PE by always discarding routes that are not
> imported as a candidate path into any VRF, since we would have a
> clear mechanism to refresh a specific VRF's table on a policy change
> - I'm not sure I see how this is solved by RT-Constrain, unless
> there's a change in the RTs that we now advertise we are importing,
> RT constrain does not result in a set of UPDATEs. Perhaps you can
> correct me here?
>
> - We remove the requirement that the RR holds state for this
> information - at the moment, as is mentioned in Section 7 of the
> draft, there is now a requirement for our RR to scale above the
> number of routes in the network, and maintain sufficient CPU and
> memory resources to store an RT-based topology map. Whilst the
> suggestion that one could introduce smaller clusters within an AS -
> I'm not sure that this scales sensibly to relatively large, diverse
> networks. It certainly introduces operational complexity to
> determining which device in a AS was responsible for erroneous
> advertisements, or route selection. If we use a single-time ORF
> field, this would mean that the RR has a transitive state that it
> holds for the duration of one, relatively small, set of UPDATEs.
> Since we'd expect that the end PE is able to maintain some form of
> admission control of these requests (as it must with Route Refresh
> currently), this results in a much lower load for the centralised RR
> resource.
>
> Many thanks, Rob
> _______________________________________________ Idr mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/idr
>

---

From: John Scudder <[email-masked]>
Date: Wed, 17 Nov 2010 08:03:52 -0800

[Lest it not be clear, this is me participating as a contributor to the group and not in my capacity as co-chair.]

(Terminology: by "RTC" I mean "RT-Constrain", i.e. RFC 4684.)

Rob,

> - We handle the case whereby there is some inbound filtering for particular VRFs - at the moment there is an arbitrary benchmark set that we can discard based on RT - however, I wouldn't see it as…

Having read it a couple of times I'm not entirely clear on the scenario you're positing.  With RTC there are basically two cases:

- A PE imports a given RT into some VRF.  In that case, the PE should advertise the RT, and necessarily must store the routes received with that RT.  If an import for that RT is added for a second VRF, the routes will already be available on the PE, no refresh from the RR is required.
- A PE doesn't import a given RT into any VRF.  In that case, the PE should not advertise the RT.  If an import for that RT is added for some VRF, the RT will be advertised and the relevant routes will be sent from the RR to the PE, both as an immediate set of updates and any incrementals required thereafter.

I have a number of questions and comments about your second, which I'll break down point-by-point:

> - We remove the requirement that the RR holds state for this information -

I'm not sure what the referent of "this information" is?  From context, perhaps you mean the RTC routes.  In any realistic use case, the overhead of the RT routes is small compared to the VPN route overhead that is potentially saved.

> at the moment, as is mentioned in Section 7 of the draft,

I assume by "the draft" you mean RFC 4684.

> there is now a requirement for our RR to scale above the number of routes in the network,

Section 7 points out that by introducing RTC, one can scale one's RR mesh more effectively.  I don't think there is intended to be any implication in that section that RTC imposes any meaningful additional overhead -- quite the contrary.

> and maintain sufficient CPU and memory resources to store an RT-based topology map.

See above.  (It's not really a "topology map" any more than any other Bellman-Ford table is a "topology map" but that's probably what you meant.)

> Whilst the suggestion that one could introduce smaller clusters within an AS - I'm not sure that this scales sensibly to relatively large, diverse networks. It certainly introduces operational complexity to determining which device in a AS was responsible for erroneous advertisements, or route selection.

My two observations are first, that the approach of subdividing reflection clusters is not specific to RTC, and second, that it's often if not universally true that optimizations can lead to greater complexity.  One has to consider each case on its own merits.  I personally don't see greater operational transparency from the use of one-time refresh for this particular application, but YMMV.  (Much more evaluation of relative merits below.)

> If we use a single-time ORF field, this would mean that the RR has a transitive state that it holds for the duration of one, relatively small, set of UPDATEs.

Again, you are referring to the filter state and contrasting it to the RTC state?  I'll reiterate that the state overhead for these is minor in contrast to the elephant in the room -- the VPN route state itself.  Again, see below.

> Since we'd expect that the end PE is able to maintain some form of admission control of these requests

"These requests" meaning transmission of one-time ORFs?

> (as it must with Route Refresh currently), this results in a much lower load for the centralised RR resource.

If I've understood you correctly, I think the summary is that you're expressing concern about the overhead imposed on RRs to maintain RTC state.  I'm not sure if you're not concerned about the overhead imposed by the VPN routes themselves, or if you just haven't considered this in your analysis.  Here is a thumbnail sketch of the tradeoffs as I see them with several different approaches.  In all cases, I assume that PEs store only routes whose RTs they import, and thus must somehow get new routes from the RR when they add an RT import:

- Plain old VPN, no optimizations: every RR stores every VPN route.  When an RT import is added on the PE, a full route refresh is required.  Costs: state to store all VPN routes on RRs.  CPU and bandwidth on RRs to generate and transmit unwanted updates to PEs, and memory to store their state.  CPU and bandwidth on PEs to receive and filter out unwanted updates.  CPU and bandwidth on both sides for full route refresh.

- VPN with ORF: every RR stores every VPN route.  When an RT import is added on the PE, just the relevant routes are fetched.  Costs: state to store all VPN routes on RRs.  State to store ORFs on RRs.

- VPN with one-time ORF (but no regular ORF): every RR stores every VPN route.  I presume that when an RT import is added on the PE, a one-time ORF is used to fetch just the relevant routes.  Costs: state to store all VPN routes on RRs.  CPU and bandwidth on RRs to generate and transmit unwanted updates to PEs, and memory to store their state.  (This is implicit in "no regular ORF".  If we did have regular ORF, this wouldn't be the case but then I don't see where the need for one-time ORF would come in.)  CPU and bandwidth on PEs to receive and filter out unwanted updates.

- VPN with RTC: Any given RR stores a subset of VPN routes, defined by the union of RTs imported by its clients.  When an RT import is added on the PE, a new RTC route is advertised, resulting in just the relevant routes being fetched.  Costs: State to store RTC routes on RRs.

To anticipate one common question, the cost to perform filtering (either ORF or especially RTC type filtering) is much less than the cost of building and transmitting an update.  So this is a clear win in the automated filtering column.

Summarizing:

Approach		Costs
-----------		-------
Plain old VPN	State to store all VPN routes on RRs.   
				CPU and bandwidth on RRs/PEs to generate/filter unwanted updates.  
				Full route refresh when RT import added.
VPN + ORF		State to store all VPN routes on RRs.  
				State to store ORFs on RRs.
VPN + OT-ORF	State to store all VPN routes on RRs.   
				CPU and bandwidth on RRs/PEs to generate/filter unwanted updates
VPN + RTC		State to store RTC routes on RRs.

Based on this, my own take on minimizing resource cost of the various approaches ranks them as RTC, followed by ORF, followed by the other two.

Regards,

--John

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 17 Nov 2010 13:13:20 -0500

To clarify one point I think Rob was making:

Suppose we filter on RT to import into a VRF.
We can use RT constrain to do that.
Suppose, in addition, we use another policy to further filter these routes.
This additional policy is not RT based.
Now, if we remove this additional policy, we want a route-refresh
to restore the routes that were discarded by that policy, just for
this one VRF.
This is where RT constrain does not help.
An RT based one time refresh will work well.

Apologies to Rob if that wasn't your point :)

--
Jakob Heitz.

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Robert Raszuk
> Sent: Wednesday, November 17, 2010 6:09 AM
> To: Rob Shakir
> Cc: [email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 17 Nov 2010 23:45:03 +0100

Jakob,

 > Suppose, in addition, we use another policy to further
 > filter these routes. This additional policy is not RT based.

Such additional policy *exampled import map over basic import RT* limits 
the routes from being inserted into the vrf, but in all implementations 
I am familiar with it does not remove them from the main vpn BGP table 
of a PE.

Hence no need to route refresh upon removal of this additional policy or 
upon configuration of the same RT import without the policy to another 
vrf table.

But if there are implementations which do remove those routes from the 
PE then I do agree one time refresh would be helpful there too.

 > This is where RT constrain does not help.

I am not sure about that .. would need to test with few RTC 
implementations to verify it. If you re-advertise the same RT over 
RT-Constrain session RR should be able to re-advertise to you VPN routes 
with such RT attached. In such case it would be semantically identical 
to the result of sending one-time ORF.

Many thx,
R.