Subject: Re: [Idr] RTC [was: Re: Possible usecases of one-time selective route refresh]

From: "Robert Raszuk (raszuk)" <[email-masked]>
Date: Thu, 18 Nov 2010 08:59:16 +0900

Hi John,

I do not see Rob's point as a debate of state of RTC on the RRs. RTC requires PE upgrade and some PEs may just for various reasons not be capable of  RTC (neither RTC lite :). 

So for those one time ORF vs full refresh when addding an RT seems like a good choice.

Thx,
R.

On Nov 18, 2010, at 8:26 AM, "Robert Raszuk (raszuk)" <[email-masked]> wrote:

> 
> 
> -------- Original Message --------
> Subject: [Idr] RTC [was: Re: Possible usecases of one-time selective
> route refresh]
> Date: Wed, 17 Nov 2010 08:03:52 -0800
> From: John Scudder <[email-masked]>
> To: Rob Shakir <[email-masked]>
> CC: [email-masked] <[email-masked]>
> 
> [Lest it not be clear, this is me participating as a contributor to the
> group and not in my capacity as co-chair.]
> 
> (Terminology: by "RTC" I mean "RT-Constrain", i.e. RFC 4684.)
> 
> Rob,
> 
> On Nov 17, 2010, at 8:42 AM, Rob Shakir wrote:
> 
> > I agree that RFC 4684 can solve this problem to some extent, however, I think there are perhaps two advantages to having a single-use ORF for this case:
> >
> > - We handle the case whereby there is some inbound filtering for particular VRFs - at the moment there is an arbitrary benchmark set that we can discard based on RT - however, I wouldn't see it as unreasonable if a prefix learnt from an RR was discarded should it not be imported into a VRF (after all, this is what we appear to be implying by discarding based on RT). This allows us to reduce the memory footprint on the PE by always discarding routes that are not imported as a candidate path into any VRF, since we would have a clear mechanism to refresh a specific VRF's table on a policy change - I'm not sure I see how this is solved by RT-Constrain, unless there's a change in the RTs that we now advertise we are importing, RT constrain does not result in a set of UPDATEs. Perhaps you can correct me here?
> 
> Having read it a couple of times I'm not entirely clear on the scenario
> you're positing.  With RTC there are basically two cases:
> 
> - A PE imports a given RT into some VRF.  In that case, the PE should
> advertise the RT, and necessarily must store the routes received with
> that RT.  If an import for that RT is added for a second VRF, the routes
> will already be available on the PE, no refresh from the RR is required.
> - A PE doesn't import a given RT into any VRF.  In that case, the PE
> should not advertise the RT.  If an import for that RT is added for some
> VRF, the RT will be advertised and the relevant routes will be sent from
> the RR to the PE, both as an immediate set of updates and any
> incrementals required thereafter.
> 
> I have a number of questions and comments about your second, which I'll
> break down point-by-point:
> 
> > - We remove the requirement that the RR holds state for this information -
> 
> I'm not sure what the referent of "this information" is?  From context,
> perhaps you mean the RTC routes.  In any realistic use case, the
> overhead of the RT routes is small compared to the VPN route overhead
> that is potentially saved.
> 
> > at the moment, as is mentioned in Section 7 of the draft,
> 
> I assume by "the draft" you mean RFC 4684.
> 
> > there is now a requirement for our RR to scale above the number of routes in the network,
> 
> Section 7 points out that by introducing RTC, one can scale one's RR
> mesh more effectively.  I don't think there is intended to be any
> implication in that section that RTC imposes any meaningful additional
> overhead -- quite the contrary.
> 
> > and maintain sufficient CPU and memory resources to store an RT-based topology map.
> 
> See above.  (It's not really a "topology map" any more than any other
> Bellman-Ford table is a "topology map" but that's probably what you meant.)
> 
> > Whilst the suggestion that one could introduce smaller clusters within an AS - I'm not sure that this scales sensibly to relatively large, diverse networks. It certainly introduces operational complexity to determining which device in a AS was responsible for erroneous advertisements, or route selection.
> 
> My two observations are first, that the approach of subdividing
> reflection clusters is not specific to RTC, and second, that it's often
> if not universally true that optimizations can lead to greater
> complexity.  One has to consider each case on its own merits.  I
> personally don't see greater operational transparency from the use of
> one-time refresh for this particular application, but YMMV.  (Much more
> evaluation of relative merits below.)
> 
> > If we use a single-time ORF field, this would mean that the RR has a transitive state that it holds for the duration of one, relatively small, set of UPDATEs.
> 
> Again, you are referring to the filter state and contrasting it to the
> RTC state?  I'll reiterate that the state overhead for these is minor in
> contrast to the elephant in the room -- the VPN route state itself.
> Again, see below.
> 
> > Since we'd expect that the end PE is able to maintain some form of admission control of these requests
> 
> "These requests" meaning transmission of one-time ORFs?
> 
> > (as it must with Route Refresh currently), this results in a much lower load for the centralised RR resource.
> 
> If I've understood you correctly, I think the summary is that you're
> expressing concern about the overhead imposed on RRs to maintain RTC
> state.  I'm not sure if you're not concerned about the overhead imposed
> by the VPN routes themselves, or if you just haven't considered this in
> your analysis.  Here is a thumbnail sketch of the tradeoffs as I see
> them with several different approaches.  In all cases, I assume that PEs
> store only routes whose RTs they import, and thus must somehow get new
> routes from the RR when they add an RT import:
> 
> - Plain old VPN, no optimizations: every RR stores every VPN route.
> When an RT import is added on the PE, a full route refresh is required.
>   Costs: state to store all VPN routes on RRs.  CPU and bandwidth on RRs
> to generate and transmit unwanted updates to PEs, and memory to store
> their state.  CPU and bandwidth on PEs to receive and filter out
> unwanted updates.  CPU and bandwidth on both sides for full route refresh.
> 
> - VPN with ORF: every RR stores every VPN route.  When an RT import is
> added on the PE, just the relevant routes are fetched.  Costs: state to
> store all VPN routes on RRs.  State to store ORFs on RRs.
> 
> - VPN with one-time ORF (but no regular ORF): every RR stores every VPN
> route.  I presume that when an RT import is added on the PE, a one-time
> ORF is used to fetch just the relevant routes.  Costs: state to store
> all VPN routes on RRs.  CPU and bandwidth on RRs to generate and
> transmit unwanted updates to PEs, and memory to store their state.
> (This is implicit in "no regular ORF".  If we did have regular ORF, this
> wouldn't be the case but then I don't see where the need for one-time
> ORF would come in.)  CPU and bandwidth on PEs to receive and filter out
> unwanted updates.
> 
> - VPN with RTC: Any given RR stores a subset of VPN routes, defined by
> the union of RTs imported by its clients.  When an RT import is added on
> the PE, a new RTC route is advertised, resulting in just the relevant
> routes being fetched.  Costs: State to store RTC routes on RRs.
> 
> To anticipate one common question, the cost to perform filtering (either
> ORF or especially RTC type filtering) is much less than the cost of
> building and transmitting an update.  So this is a clear win in the
> automated filtering column.
> 
> Summarizing:
> 
> Approach                Costs
> -----------             -------
> Plain old VPN   State to store all VPN routes on RRs.
>                                 CPU and bandwidth on RRs/PEs to generate/filter unwanted updates.
>                                 Full route refresh when RT import added.
> VPN + ORF               State to store all VPN routes on RRs.
>                                 State to store ORFs on RRs.
> VPN + OT-ORF    State to store all VPN routes on RRs.
>                                 CPU and bandwidth on RRs/PEs to generate/filter unwanted updates
> VPN + RTC               State to store RTC routes on RRs.
> 
> Based on this, my own take on minimizing resource cost of the various
> approaches ranks them as RTC, followed by ORF, followed by the other two.
> 
> Regards,
> 
> --John
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: John Scudder <[email-masked]>
Date: Thu, 18 Nov 2010 12:46:43 -0800

Robert,

One-time ORF would also require a PE upgrade (as well as an RR upgrade). 

My back-of-the-envelope analysis also suggests that plain old ORF provides more value than one-time ORF in this situation, though less than RTC.

--John

On Nov 17, 2010, at 6:59 PM, Robert Raszuk (raszuk) wrote:

> <ATT00001..txt>

---

From: Jie Dong <[email-masked]>
Date: Fri, 19 Nov 2010 10:01:03 +0800

Hi John,

IMO the major difference between plain ORF and one-time ORF is the lifetime
of the filters, and for one-time ORF the peer does not need to save the
filters locally and perform such filtering for every subsequent update. Thus
one-time ORF can be quite lightweight than plain ORF and can be used
with/without plain ORF.

Another possible use case is the import RTs are unchanged, but some policies
on the main VPN table are changed, i.e. policies which would modify
attributes of the routes. Thus the routes in the main VPN table are
different from the original routes on peers. When such policies are removed,
it would be needed to retrieve original routes from the peer. With one-time
ORF containing the import RTs, peer could avoid advertising routes which
would be discarded based on these import RTs. Prefixed based one-time ORF
could also work in similar way.

Though one-time ORF could be triggered automatically, IMO another common use
case would be manually request some specific refresh, such as refresh routes
of specific prefixes, or routes with some specific RTs.

Best regards,
Jie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of John
> Scudder
> Sent: Friday, November 19, 2010 4:47 AM
> To: Robert Raszuk (raszuk)
> Cc: [email-masked]
> Subject: Re: [Idr] RTC [was: Re: Possible usecases of one-time selective
route
> refresh]
> My back-of-the-envelope analysis also suggests that plain old ORF provides
> more value than one-time ORF in this situation, though less than RTC.
> 
> --John
> 
> > I do not see Rob's point as a debate of state of RTC on the RRs. RTC
requires
> PE upgrade and some PEs may just for various reasons not be capable of
> RTC (neither RTC lite :).
> >
> > So for those one time ORF vs full refresh when addding an RT seems like
a
> good choice.
> >> > I agree that RFC 4684 can solve this problem to some extent, however,
I
> think there are perhaps two advantages to having a single-use ORF for this
> case:
> >> >
> >> > - We handle the case whereby there is some inbound filtering for
> particular VRFs - at the moment there is an arbitrary benchmark set that
we
> can discard based on RT - however, I wouldn't see it as unreasonable if a
> prefix learnt from an RR was discarded should it not be imported into a
VRF
> (after all, this is what we appear to be implying by discarding based on
RT).
> This allows us to reduce the memory footprint on the PE by always
> discarding routes that are not imported as a candidate path into any VRF,
> since we would have a clear mechanism to refresh a specific VRF's table on
a
> policy change - I'm not sure I see how this is solved by RT-Constrain,
unless
> there's a change in the RTs that we now advertise we are importing, RT
> constrain does not result in a set of UPDATEs. Perhaps you can correct me
> here?
> >>
> >> > Whilst the suggestion that one could introduce smaller clusters
within
> an AS - I'm not sure that this scales sensibly to relatively large,
diverse
> networks. It certainly introduces operational complexity to determining
> which device in a AS was responsible for erroneous advertisements, or
route
> selection.
> >> > If we use a single-time ORF field, this would mean that the RR has a
> transitive state that it holds for the duration of one, relatively small,
set of
> UPDATEs.
> >> > (as it must with Route Refresh currently), this results in a much
lower
> load for the centralised RR resource.
> >>

---

From: John Scudder <[email-masked]>
Date: Thu, 18 Nov 2010 18:58:15 -0800

> one-time ORF can be quite lightweight than plain ORF and can be used

In evaluating "lightweight" one must consider two costs that are being balanced -- one is the cost of saving and evaluating ORFs, as you point out.  But the other is the cost to generate/filter unwanted updates during the steady state (i.e. after the OT-ORF has been evaluated and discarded).  With regular ORF, a filtered route will be discarded without ever being transmitted.  If OT-ORF is the only thing in use (and is not currently active) filtering would occur on the PE side.  That amounts to a significant cost, compared to minor savings.  This suggests that in this particular scenario, OT-ORF does not make sense as the sole filtering technology in use.  And it is debatable what it adds if some other filtering technology (RTC, regular ORF) is in use.

> different from the original routes on peers.

This is true in some implementations, not so in others.

> When such policies are removed,
> it would be needed to retrieve original routes from the peer.

Again, this is implementation specific.

> With one-time
> would be discarded based on these import RTs.

I thought you were going to say that OT-ORF could provide a way to retrieve only the routes that (potentially) match the policy that has been changed.  I would agree with that, although again the need for this is implementation specific.  What you wrote is different though.

> Prefixed based one-time ORF

Why would you need to do this other than to work around a buggy implementation?

Why would you operationally feel confident that you had requested a sufficient subset, if your implementation is already not trustworthy?  Wouldn't it be safer in such a broken circumstance to request a refresh of the entire routing table?  Optimization of a situation that is broken to begin with seems questionable.  (I believe this comment was made by someone at the mic at our meeting as well.)

Regards,

--John

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 19 Nov 2010 10:42:55 +0100

Hi John,

 > you point out.  But the other is the cost to generate/filter unwanted
 > updates during the steady state (i.e. after the OT-ORF has been
 > evaluated and discarded).

But why would you need to send full table from RR to PE in a steady 
state ? In a steady state we would only send flapping routes or added or 
deleted ones. Are you saying that discarding those on the PEs ifr they 
carry non intersecting RT will be a problem ?

 > Wouldn't it be safer in such a broken circumstance to request a
 > refresh of the entire routing table?

How would you do that if you are recommending use of regular ORF ? 
Wouldn't you first need to remove all ORF entries then issue a refresh 
request then add again all ORF entries ?

Contrary to the above with OT-ORF you can always issue a full table 
refresh request without any additional RT tables manipulation on the RRs.

Thx,
R.

>> IMO the major difference between plain ORF and one-time ORF is the
>> lifetime of the filters, and for one-time ORF the peer does not
>> need to save the filters locally and perform such filtering for
>> every subsequent update. Thus one-time ORF can be quite lightweight
>> than plain ORF and can be used with/without plain ORF.
>
> In evaluating "lightweight" one must consider two costs that are
> being balanced -- one is the cost of saving and evaluating ORFs, as
> you point out.  But the other is the cost to generate/filter unwanted
> updates during the steady state (i.e. after the OT-ORF has been
> evaluated and discarded).  With regular ORF, a filtered route will be
> discarded without ever being transmitted.  If OT-ORF is the only
> thing in use (and is not currently active) filtering would occur on
> the PE side.  That amounts to a significant cost, compared to minor
> savings.  This suggests that in this particular scenario, OT-ORF does
> not make sense as the sole filtering technology in use.  And it is
> debatable what it adds if some other filtering technology (RTC,
> regular ORF) is in use.
>
>> Another possible use case is the import RTs are unchanged, but some
>> policies on the main VPN table are changed, i.e. policies which
>> would modify attributes of the routes. Thus the routes in the main
>> VPN table are different from the original routes on peers.
>
>> When such policies are removed, it would be needed to retrieve
>> original routes from the peer.
>
>> With one-time ORF containing the import RTs, peer could avoid
>> advertising routes which would be discarded based on these import
>> RTs.
>
> I thought you were going to say that OT-ORF could provide a way to
> retrieve only the routes that (potentially) match the policy that has
> been changed.  I would agree with that, although again the need for
> this is implementation specific.  What you wrote is different
> though.
>
>> Prefixed based one-time ORF could also work in similar way.
>>
>> Though one-time ORF could be triggered automatically, IMO another
>> common use case would be manually request some specific refresh,
>> such as refresh routes of specific prefixes, or routes with some
>> specific RTs.
>
> Why would you operationally feel confident that you had requested a
> sufficient subset, if your implementation is already not trustworthy?
> Wouldn't it be safer in such a broken circumstance to request a
> refresh of the entire routing table?  Optimization of a situation
> that is broken to begin with seems questionable.  (I believe this
> comment was made by someone at the mic at our meeting as well.)
>
> Regards,
>
> --John

---

From: John Scudder <[email-masked]>
Date: Fri, 19 Nov 2010 08:46:57 -0800

> carry non intersecting RT will be a problem ?

A significant topology change (for example, but not limited to, a major EBGP peering going down) could lead to a large table exchange.  Although initial convergence and route refresh are the most common bulk exchange cases, they aren't the only ones.  

> How would you do that if you are recommending use of regular ORF ? 

I'm not so much recommending the use of regular ORF, as contrasting what resources are consumed by the various solutions.  We report, you decide. ;-)  I don't think it's possible to respond to your question (and the following ones) without having a more detailed description of what exact problem OT-ORF is supposed to be solving.  The high-order bit from my end is that once you're in a position where you are (virtually speaking) pounding on the side of the router trying to shake loose some routes that have gotten stuck, you're already in a world of hurt.  Using a mechanical router-pounder to shake loose the routes may not be the preferred approach. :-)

--John

---

From: Jakob Heitz <[email-masked]>
Date: Fri, 19 Nov 2010 15:23:27 -0500

The value of one-time refresh:

Suppose we import received routes into a VRF based on RT.
We filter some routes out with an additional policy.
This additional policy can not be represented in an ORF.

Now, we change this additional policy.
We need to recover the routes that this policy previously discarded.

To recover those routes, implementation choices are
1. Keep the received routes in local cache and recover them from there.
2. Ask for total refresh.
3. Ask for one-time refresh selected by RT.

I think the best choice is (3).

The RT constraints didn't change, therefore RT constrain can
not be used to recover these routes.

--
Jakob Heitz.

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of John Scudder
> Sent: Thursday, November 18, 2010 6:58 PM
> To: Jie Dong
> Cc: Robert Raszuk (raszuk); [email-masked]
> Subject: Re: [Idr] RTC [was: Re: Possible usecases of
> one-time selective route refresh]
>
> In evaluating "lightweight" one must consider two costs that
> are being balanced -- one is the cost of saving and
> evaluating ORFs, as you point out.  But the other is the cost
> to generate/filter unwanted updates during the steady state
> (i.e. after the OT-ORF has been evaluated and discarded).
> With regular ORF, a filtered route will be discarded without
> ever being transmitted.  If OT-ORF is the only thing in use
> (and is not currently active) filtering would occur on the PE
> side.  That amounts to a significant cost, compared to minor
> savings.  This suggests that in this particular scenario,
> OT-ORF does not make sense as the sole filtering technology
> in use.  And it is debatable what it adds if some other
> filtering technology (RTC, regular ORF) is in use.
>
> I thought you were going to say that OT-ORF could provide a
> way to retrieve only the routes that (potentially) match the
> policy that has been changed.  I would agree with that,
> although again the need for this is implementation specific.
> What you wrote is different though.
>
> Why would you operationally feel confident that you had
> requested a sufficient subset, if your implementation is
> already not trustworthy?  Wouldn't it be safer in such a
> broken circumstance to request a refresh of the entire
> routing table?  Optimization of a situation that is broken to
> begin with seems questionable.  (I believe this comment was
> made by someone at the mic at our meeting as well.)
>
> Regards,
>
> --John
>
> >
> > Best regards,
> > Jie
> >
> >>>> should not advertise the RT.  If an import for that RT
> is added for
> > some
> >>>> VRF, the RT will be advertised and the relevant routes
> will be sent
> > from
> >>>>> there is now a requirement for our RR to scale above
> the number of
> >> routes in the network,
> >>>>
> >>>> Bellman-Ford table is a "topology map" but that's
> probably what you
> >> meant.)
> >>>>
> >>>> RTC state?  I'll reiterate that the state overhead for
> these is minor
> > in
> >>>>> Since we'd expect that the end PE is able to maintain
> some form of
> >> admission control of these requests
> >>>>
> >>>> them with several different approaches.  In all cases, I
> assume that
> > PEs
> >>>> store only routes whose RTs they import, and thus must
> somehow get
> >> new
> >>>>  Costs: state to store all VPN routes on RRs.  CPU and
> bandwidth on
> >> RRs
> >>>> (This is implicit in "no regular ORF".  If we did have
> regular ORF,
> > this
> >>>> the union of RTs imported by its clients.  When an RT
> import is added
> > on
> >>>> approaches ranks them as RTC, followed by ORF, followed
> by the other
> >> two.
> >>>>
> >>>> Regards,
> >>>>
> >>>> --John

---

From: John Scudder <[email-masked]>
Date: Fri, 19 Nov 2010 13:31:25 -0800

Jakob,

Yes, I can see that scenario.  Some implementations already do #1, but as you say it is an implementation choice.  As to #3, the question is whether the scenario is significant enough to warrant implementing a new protocol extension, also keeping in mind that option #1 exists and can be done unilaterally.  This is a subjective question.

Regards,

--John

On Nov 19, 2010, at 3:23 PM, Jakob Heitz wrote:

>> one-time selective