Subject: [Idr] Fwd: I-D Action:draft-keyur-bgp-enhanced-route-refresh-00.txt

From: Enke Chen <[email-masked]>
Date: Wed, 06 Oct 2010 11:42:50 -0700

FYI.  Will appreciate comments.

-- Enke

-------- Original Message --------
Subject: 	I-D Action:draft-keyur-bgp-enhanced-route-refresh-00.txt
Date: 	Tue, 5 Oct 2010 22:00:02 -0700 (PDT)
From: 	[email-masked]
Reply-To: 	[email-masked]
To: 	[email-masked]

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : Enhanced Route Refresh Capability for BGP-4
	Author(s)       : K. Patel, et al.
	Filename        : draft-keyur-bgp-enhanced-route-refresh-00.txt
	Pages           : 6
	Date            : 2010-10-05

In this document we enhance the existing BGP route refresh mechanisms
to provide for the demarcation of the beginning and the ending of a
route refresh.  The enhancement can be used to facilitate on-line,
non-disruptive consistency validations of BGP routing updates.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-keyur-bgp-enhanced-route-refresh-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Qing Zeng <[email-masked]>
Date: Sat, 09 Oct 2010 16:03:33 +0800

Hi All:

I have some questions:
1. This is the method to checking for possible missing withdraws between BGP speakers, 
what's the most likely to occur application scenario of  possible missing withdraws?  

2. Why don't consider that using End-of-RIB(EOR) or extension-UPDATE message to 
mark the beginning and the ending of a route refresh? After all, the EOR is more often
used to mark completion of the whole Rib update for routing convergence.

3. How to operate when BGP speaker is sending Adj-Rib-Out due to refresh with route oscillation?
For correctly identifing the beginning and the ending of a route refresh , should route selection be hold on the speaker?

Best Regards!
Qing

----- 原始邮件 ----- 
发件人: "Enke Chen" <[email-masked]>
收件人: <[email-masked]>
抄送: "Keyur Patel" <[email-masked]>
发送时间: 2010年10月7日 2:42
主题: [Idr] Fwd: I-D Action:draft-keyur-bgp-enhanced-route-refresh-00.txt

> implementation to automatically retrieve the ASCII version of the

--------------------------------------------------------------------------------

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Enke Chen <[email-masked]>
Date: Mon, 11 Oct 2010 01:06:15 -0700

Hi, Qing:

Please see my comments inline.

> what's the most likely to occur application scenario of  possible missing withdraws?

It provides a way to detect inconsistencies and to force consistency 
over a established session.  Dealing with missing withdraws (caused by 
software bugs) is one of the applications.  Yes, there have been a few 
occurrences in the past.

> used to mark completion of the whole Rib update for routing convergence.

The End-of-Rib as currently defined and used for GR is not adequate for 
the current purpose as:

      - there is no beginning.
      - in the case of continuous churn, we do not want the End-of-Rib 
to be delayed like the end-of-refresh.

Also, extending the route refresh seems more logical than extending the 
UPDATE message in this case.

> For correctly identifing the beginning and the ending of a route refresh , should route selection be hold on the speaker?

In case of continuous churn,  an implementation should support a locally 
configurable upper bound for generating the end-of-refresh.  One way is 
to suspend the inbound processing briefly.  That should be optional 
(will revise in the next version).   Another way is to just generate the 
end-of-refresh.   There is little practical difference between the 
approaches for a route that keeps churning.

Thanks.   -- Enke

> Best Regards!
> Qing
>
>
>
>
> ----- 原始邮件 -----
> 发件人: "Enke Chen"<[email-masked]>
> 收件人:<[email-masked]>
> 抄送: "Keyur Patel"<[email-masked]>
> 发送时间: 2010年10月7日 2:42

---

From: Qing Zeng <[email-masked]>
Date: Mon, 11 Oct 2010 18:56:28 +0800

Hi Enke:
Thanks for your comments and please see my further replies inline with [Qing1].

Best Regards!
 Qing

----- 原始邮件 ----- 
发件人: "Enke Chen" <[email-masked]>
收件人: "Qing Zeng" <[email-masked]>
抄送: <[email-masked]>; "Keyur Patel" <[email-masked]>; "Enke Chen" <[email-masked]>
发送时间: 2010年10月11日 16:06
主题: Re: [Idr] Fwd: I-D Action:draft-keyur-bgp-enhanced-route-refresh-00.txt

> software bugs) is one of the applications.  Yes, there have been a few 

[Qing1]
OK, I think it is a valuable requirements for maintenance.
But,if it's caused by software bugs or some other reason, there are also possible missing updates.
For the possible missing updates, using route refresh will recovery the failure directly without detecting.
So is that only with "force consistency" more appropriate than with "detect inconsistencies"?
As an aside,
" All the routes that were not re-advertised in the route refresh MUST be
purged, and SHOULD be logged for further analysis." 
How to further analysis? Manual deleting or on-line?

> The End-of-Rib as currently defined and used for GR is not adequate for 
[Qing1]
   The End-of-Rib should not be used only for GR.
   From RFC4724
   "   Although the End-of-RIB marker is specified for the purpose of BGP
   graceful restart, it is noted that the generation of such a marker
   upon completion of the initial update would be useful for routing
   convergence in general, and thus the practice is recommended."
>      - there is no beginning.
[Qing1]
      Agree. 
      But could we consider extending the one special attribute for UPDATE message? 
      at the same time, some routes will be packing with this flag in one UPDATE.

> to be delayed like the end-of-refresh.
[Qing1]
      How to understand the "delayed"?  
      In the case of continuous churn, is there difference between the End-of-Rib and the end-of-refresh?

> approaches for a route that keeps churning.
[Qing1]
OK,I see. And if I had to choose one between these two, I choose the former.
I think the latter will make the results of detecting inconsistencies more inaccurate.
In my understanding, The goal of suspending the inbound processing briefly 
after the upper bound is reached is to speed up refresh process, 
but why not consider suspending the inbound processing when begining of the refresh?

---

From: Paul Jakma <[email-masked]>
Date: Mon, 11 Oct 2010 13:17:56 +0100 (BST)

> FYI.  Will appreciate comments.

Something like this would be really useful. Two questions though:

1. Would it be possible to re-use the existing End-of-RIB signaling
    functionality specified in draft-ietf-idr-restart, and already
    present in several implementations?

2. What use do you envisage for the "start of refresh" message? Is it
    perhaps superfluous?

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
I have a map of the United States.  It's actual size.  I spent last summer
folding it.  People ask me where I live, and I say, "E6".
 		-- Steven Wright

---

From: Jie Dong <[email-masked]>
Date: Mon, 11 Oct 2010 20:35:55 +0800

Paul, 

>     present in several implementations?

Agreed. End-of-RIB could be used for completion of route refresh.

> 2. What use do you envisage for the "start of refresh" message? Is it

In my understanding, this is for the demarcation of the whole set of
refreshed routes, which can be used for removing routes which are not
re-advertised during the refresh.
This is described in the last paragraph of Section 3.

Regards,
Jie

---

From: "iLya" <[email-masked]>
Date: Mon, 11 Oct 2010 14:59:10 +0200

--------------------------------------------------

>From operator's PoV I'm concerned about impact of suspending inbound 
processing. With current IPv4 and L3VPN RIB sizes it may take few minutes 
for some routers to exchange full tables. If inbound processing is delayed 
for more than few seconds the routes being sent to peer may result in 
blackholing traffic for significant time. I don't have instant solution to 
the problem, but some optimisation could be made by verifying whether it's 
necessary to send whole RIB out - if route-refresh requester could assert 
whether it has up-to-date info from peer (e.g. by using some RIB 
digest/fingerprint) then full-blown exchange could be avoided in some (or 
even most?) cases.

/iLya

---

From: Paul Jakma <[email-masked]>
Date: Mon, 11 Oct 2010 14:19:21 +0100 (BST)

> Agreed. End-of-RIB could be used for completion of route refresh.

Yeah, sorry, I had missed the rest of the thread, with your comments 
and Enke's replies. ;)

> In my understanding, this is for the demarcation of the whole set 
> of refreshed routes, which can be used for removing routes which 
> are not re-advertised during the refresh.

Why do you need to know when the RR-generated UPDATEs start? You 
control sending of the RR request, so you can mark that peers routes 
as stale /before/ you send the RR request. ??

regards,
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
I never failed to convince an audience that the best thing they
could do was to go away.

---

From: Keyur Patel <[email-masked]>
Date: Mon, 11 Oct 2010 10:00:39 -0700

Paul,

>     present in several implementations?

As Enke had mentioned in his earlier email reply to Qing, we do NOT want to
delay sending of End-of-Rib like End-of-Refresh in case of continuous route
churn. 

> 2. What use do you envisage for the "start of refresh" message? Is it

SOR is sent to identify the beginning of the table exchange. It should be
used for purging of the stale routes after the table exchange.

Best Regards,
Keyur

---

From: Keyur Patel <[email-masked]>
Date: Mon, 11 Oct 2010 10:04:25 -0700

Paul,

> as stale /before/ you send the RR request. ??

There are cases where table announcements can happen without being initiated
by the Route Refresh. SOR is used in those cases as well.

Best Regards,
-Keyur

---

From: Paul Jakma <[email-masked]>
Date: Mon, 11 Oct 2010 19:00:48 +0100 (BST)

On Mon, 11 Oct 2010, Keyur Patel wrote:

> There are cases where table announcements can happen without being 
> initiated by the Route Refresh. SOR is used in those cases as well.

For unsolicited 'pushed' refreshes (e.g. for policy changes) the 
sender should just generate the required withdrawals, as you would 
want to do anyway already / for peers that don't support this 
extension. Plus validation is likely to be expensive on the receiver 
side in terms of having to scan the RIB to mark the entries as stale. 
Peers shouldn't be able to 'push' such work onto others unsolicited.

I think at least some implementations will just ignore SoR. It seems 
a ripe accidental-DoS vector in the one use case, and superfluous in 
the other.

Anyway...
-- 
Paul Jakma	[email-masked]	Key ID: 64A2FF6A
Fortune:
If you fool around with something long enough, it will eventually break.

---

From: Qing Zeng <[email-masked]>
Date: Tue, 12 Oct 2010 10:11:09 +0800

iLya and All,

For BGP routes,  maybe the most appropriate RIB digest is the prefixs.
Thus if only using prefixs, the inconsistent number of routes could  be detected, 
and the inconsistent of attributes of routes could not be detected.
This issue should be further discussed.

And at the other hand, we have some ideas of Precise Route Refresh Capability to related wtih the requirement iLya mentioned
---Sending precise route refresh request when it is unnecessary to send whole RIB out.
We will soon submit the draft and appreciate your comments.

Best Regards!
Qing

---

From: Rob Shakir <[email-masked]>
Date: Tue, 12 Oct 2010 09:47:12 +0100

Hi Qing, IDR,

> This issue should be further discussed.

It seems to me that Ilya's concerns are very valid here -- any approach that
would require inbound processing to be suspended would appear harmful, even
if only for the group of peers to which the BGP speaker requesting a route
refresh is a member of is suspended. In some cases (especially where things
are congested), this may result in very slow convergence for a relatively
large number of speakers (for instance a route reflector in an L3VPN network
is likely to pool all speakers within a single "update-group" or equivalent
structure). Whilst I don't have hard data for this, I do not think route
refresh events that last >10 minutes are impossible (consider large L3VPN
tables, with no co-operative route filtering).

If there is a locally configurable upper bound for the generation of the end
of refresh, what value does this have over the router requesting a refresh
tracking the time since it was sent, and utilising a similar timer to remove
stale routes? 

> And at the other hand, we have some ideas of Precise Route Refresh Capability
> to related wtih the requirement iLya mentioned

If this is suggested, then I would question whether one can actually
validate RIB consistency (which is what the draft we are discussing appears
to aim to do) by selecting the parts of the RIB that we want to receive? How
do we know which elements of the RIB are inconsistent?

Can we not solve any requirement to have a "Precise Route Refresh" by
utilising the existing ORF mechanism in RFC5291, with some possible
modifications to allow a set of filters with a limited scope of a single
route refresh?

Kind regards,
Rob

This e-mail has been scanned for viruses by the Cable & Wireless Worldwide e-mail security system - powered by MessageLabs. For more information on a proactive managed e-mail security service, visit http://www.cw.com/managed-exchange

The information contained in this e-mail is confidential and may also be subject to legal privilege. It is intended only for the recipient(s) named above. If you are not named above as a recipient, you must not read, copy, disclose, forward or otherwise use the information contained in this email. If you have received this e-mail in error, please notify the sender (whose contact details are above) immediately by reply e-mail and delete the message and any attachments without retaining any copies.

Cable & Wireless Worldwide plc 
Registered in England and Wales. Company Number 07029206
Registered office: Liberty House, 76 Hammersmith Road, London W14 8UD, England

---

From: Jie Dong <[email-masked]>
Date: Tue, 12 Oct 2010 18:05:52 +0800

Hi Rob, 

> tables, with no co-operative route filtering).

IMO that is why precise route refresh is needed, since in most cases not all
the routes in adj-rib-out need to be re-advertised. If the operator could
locate the problematic routes in some scope, with a precise refresh request
the duration of refresh can be significantly reduced.

> do we know which elements of the RIB are inconsistent?

This precise refresh is not for RIB consistency validation (which is the
goal of enhanced refresh draft). The purpose of precise refresh is to reduce
the time, bandwidth and processing of route refresh in some maintenance
scenarios. There will be a new draft soon describing this precise refresh in
detail.

> modifications to allow a set of filters with a limited scope of a single

we have similar thought here. 
But in my opinion, the use case of precise refresh is asking a peer to
re-advertise some specific routes in peer's current rib-out, without
modifying the peer's outbound filter, which is not what ORF can achieve. 

Though the filters can be similar to those in ORF, logically this mechanism
should be considered as an extension to refresh.

Best regards,
Jie

---

From: Rob Shakir <[email-masked]>
Date: Tue, 12 Oct 2010 16:49:41 +0100

Hi Jie,

> the duration of refresh can be significantly reduced.

It seems this assumes that the operator initiates a Route Refresh - we
should probably bear in mind that there are a number of cases where this
might be initiated by an automatic mechanism (e.g. due to a policy change),
in this case, is there a way that one can identify which prefixes one would
expect to request? It would seem to me that there is a requirement for
relatively complex filtering to be able to make specific requests (e.g.
where there is a requirement to refresh only prefixes with a list of 10 RT
extcomms?)

> scenarios. There will be a new draft soon describing this precise refresh in

This implies that my previous question (regarding the utility of Enhanced
Route Refresh when compared to an requester-side timer) is still relatively
valid.

> should be considered as an extension to refresh.

Do we not therefore risk duplicating work in ORF, rather than merely
defining some mechanism to make a specific outbound filter temporary for a
single Route Refresh?

I will keep further questions until I've read your draft, and can go from
there, however, I'm still interested in the initial queries around the
Enhanced Route Refresh behaviour, and the utility of such a mechanism.

Many thanks,
Rob

This e-mail has been scanned for viruses by the Cable & Wireless Worldwide e-mail security system - powered by MessageLabs. For more information on a proactive managed e-mail secure service, visit http://www.cw.com/managed-exchange

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 13 Oct 2010 03:55:36 +0000

Enke (and other authors),

I like the general concept of doing a "refresh for consistency".

I agree that we need a signaling mechanism to indicate that we've started
UPDATEs related to that refresh.

I also agree that we need some sort of end-of-rib marker to indicate that
the refresh is done.

What this mechanism resembles most to me is a form of "triggered graceful
restart".  While I think the encoding of the status of the refresh in the
existing refresh message type a little weird, it's an adequate encoding.

I think if the draft was updated slightly to imply or at least reference the
pre-existing graceful restart semantics for the AFI/SAFI that is being
refreshed, I think I would be happier with this document.  The GR mechanism
also implies our upper bound for the refresh time.  Tying this mechanism
back to our GR semantics suggests to implementors that existing machinery
for marking routes as stale and expiring them if not refreshed would help
future implementors realize the features are rather similar.

Finally, while I'm not a huge fan of the complexity of the BGP FSM as
published in RFC 4271, such a "soft graceful restart" mechanism has FSM
implications.  Those should be documented.

-- Jeff

---

From: Keyur Patel <[email-masked]>
Date: Tue, 12 Oct 2010 21:48:25 -0700

Hi Rob, Ilya,

Comment inlined #Keyur

> tables, with no co-operative route filtering).

#Keyur: The above mentioned approach is suggested in the draft as *a
possible solution* and by no means it is *a mandatory approach*.
Furthermore, this approach is suggested in cases where a persistent
(possibly huge) churn results in peers servicing refresh to lag for a
prolong period of time. Otherwise EOR would be generated after the table
announcement.

We can try and clarify the text in the next draft revision.

Best Regards,
Keyur 
> This e-mail has been scanned for viruses by the Cable & Wireless Worldwide
> e-mail security system - powered by MessageLabs. For more information on a
> proactive managed e-mail security service, visit
> http://www.cw.com/managed-exchange
> 
> 
> 
> The information contained in this e-mail is confidential and may also be
> subject to legal privilege. It is intended only for the recipient(s) named
> above. If you are not named above as a recipient, you must not read, copy,
> disclose, forward or otherwise use the information contained in this email. If
> you have received this e-mail in error, please notify the sender (whose
> contact details are above) immediately by reply e-mail and delete the message
> and any attachments without retaining any copies.
>

---

From: Paul Jakma <[email-masked]>
Date: Wed, 13 Oct 2010 16:54:23 +0100 (BST)

On Tue, 12 Oct 2010, Keyur Patel wrote:

> #Keyur: The above mentioned approach is suggested in the draft as 
> *a possible solution* and by no means it is *a mandatory approach*. 
> (possibly huge) churn results in peers servicing refresh to lag for 
> a prolong period of time. Otherwise EOR would be generated after 
> the table announcement.

A bounded solution would be to make use of the peer's Adj-Out, rather 
than trying to place the EoR based on the main RIB:

a) lock the Adj-Out for the peer in some way, to prevent the
   insertions/deletions (updates of data is fine).

b) send each route to the peer (this can be latest copy from the RIB,
   obviously, if the implementation wishes)

c) send the EoR

d) unlock the Adj-Out

Inserts/deletes to the AdjOut can block or be buffered somehow, 
without blocking the main LocalRIB per se.

Remember, the important bit here is to allow the receiver to detect 
certain inconsistencies (missed withdraws particularly). It doesn't 
per se have to get a copy of the peer's RIB before the EoR - indeed, 
that's likely impossible to do given that it is not unreasonable for 
a LocalRIB to change as fast as it can be sent, on short but 
noticeable grain time scales.

Further, there will be certain inconsistencies that are just too 
costly to try detect with this scheme. E.g. if the route refresh 
sends a route with different attributes to the one in the RIB - is 
that cause it was missed, or cause it was updated in the sender just 
before the RR? You generally won't be able to say, methinks.

It might be an idea to define some kind of reversible pseudo-checksum 
(i.e. that can be updated with every insertion/deletion of a route 
alone; CRC based thing might work..) and include that with BGP 
messages as a kind of RIB-generation-identifier to allow 
inconsistencies to be flagged.

A simpler message number might also be considered, but it'd be far 
weaker.

regards,
Fortune:
>From 0 to "what seems to be the problem officer" in 8.3 seconds.
 		-- Ad for the new VW Corrado

---

From: Martin Djernaes <[email-masked]>
Date: Wed, 13 Oct 2010 21:18:27 +0100

Hi Enke, Keyur et al,

The concept of a new route fresh message with clear demarcation markers is a useful concept. 

I do have a few things I think we should consider changing.

In section 3 you write:

   In the case of continuous churn of the BGP table, an implementation
   SHALL impose an upper bound on how long it would take to complete the
   route refresh.  Once the upper bound is reached, the implementation
   SHOULD suspend the processing of the received UPDATE messages briefly
   until the route refresh to the peer is complete.

I think adding any information into an official document stating that it is ok to suspend processing of incoming updates for even a small bit of time. This gives an uncontrollable delay in the propagation of updates and won't be good for the networks.

I think it would be beneficial to clarify how "begin" and "end" marker are send and processed is required. Especially if subsequent refresh requests, received in the time period from the first "begin" marker is send and until the corresponding "end" marker is send. What is the proper handling.

Also, along the same line, I think it should be said that the side sending a "begin" marker may end up dropping the "end" marker (or delaying it so long that it becomes mood). How this is handled on the requesting side is probably left up to the implementation to handle.

In section 2.2 you write:

   The "Reserved" field of the ROUTE-REFRESH message specified in
   [RFC2918] is re-defined as the "Message Subtype" with the following
   values:

          0 - Normal route refresh request [RFC2918], or ORF [RFC5291]
          1 - Demarcation of the beginning of a route refresh
          2 - Demarcation of the ending of a route refresh

I would suggest that instead of overloading the empty octet that instead an extended route refresh message is expanded with a TLV field where either a "type start" TLV or a "type end" TLV  can be placed. This, imho, gives more flexibility in the long run.

Martin

From: [email-masked] [mailto:[email-masked]] On Behalf Of Enke Chen
Sent: 06 October 2010 20:43
To: [email-masked]
Cc: Keyur Patel
Subject: [Idr] Fwd: I-D Action:draft-keyur-bgp-enhanced-route-refresh-00.txt

-- Enke

-------- Original Message -------- 
Subject: 
I-D Action:draft-keyur-bgp-enhanced-route-refresh-00.txt
Date: 
Tue, 5 Oct 2010 22:00:02 -0700 (PDT)
From: 
[email-masked]
Reply-To: 
[email-masked]
To: 
[email-masked]