Subject: [Idr] draft-litkowski-idr-bgp-timestamp

From: <[email-masked]>
Date: Tue, 22 Jul 2014 19:19:57 +0000

[Renamed topic]

Hi Robert,

> Your results may actually in the above case casue more problems then gains unfortunately.

I don’t really understand why … even if RR are not in the dataplane (which would be the case for VPN environment), RR behavior is really important to track.

Consider a MPLS VPN scenario, some PEs connected to RR clusters and meshing between clusters.
Consider that one of the RR is getting stuck because of Route-refresh processing of 2M of routes or just busy to do something else due to bad scheduling in the implemention … if a PE lose a customer connection, and customer has a backup connection and backup PE has to send a new BGP update to propagate the new path. The time for RR to propagate the BGP update is critical to track because this will condition convergence time for the customer …

Thoughts ?

Stephane

From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Tuesday, July 22, 2014 11:15
To: LITKOWSKI Stephane SCE/IBNF; Jeffrey Haas
Cc: idr wg
Subject: draft-litkowski-idr-rtc-interas

Stephane,

I have one fundamental question on this ...

You have focused on making sure that BGP control plane works fast and solid etc. That is great and timestamp idea at each BGP hop would help.

However let's observe that BGP control plane is NOT congruent with data plane. In other words you may have a case where someone may have slow and overloaded RR but uses 40 GB pipes for data plane while other AS installed super-hyper x86 RRs and has 1 GB links for data plane.

Your results may actually in the above case casue more problems then gains unfortunately.

The possible simple fix is to mandate that timestamp is ONLY added when your change next hop to self or that you can have a new flag "N" - Next Hop Set which will help a lot to make more use cases for this proposal.

Best,
R.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Jul 2014 21:29:42 +0200

Stephane,

If you build your service counting on control plane convergence I think you
are doing bad thing :) Sorry.

If this draft is to further help with such designs I would clearly switch
to "no support" immediately. I had other use cases in mind which did not
sound negative.

Specifically in your case if one PE of multihomed site looses PE-CE you
locally protect the site (10s ms max) and take all time it needs for BGP to
converge regardless how long it takes. Of course I am assuming basic rules
like unique RD per VRF in place.

The most important is to maintain data plane connectivity.

Besides I am not sure how timestamp will help you in this case. Leave alone
that timestamp may never catch the refresh max CPU load peak so you may
incorrectly consider RR as sound when it is really not.

Cheers,
R.

On Tue, Jul 22, 2014 at 9:19 PM, <[email-masked]> wrote:

> I don’t really understand why … even if RR are not in the dataplane (which
> would be the case for VPN environment), RR behavior is really important to
> track.
>
>
>
> Consider that one of the RR is getting stuck because of Route-refresh
> processing of 2M of routes or just busy to do something else due to bad
> scheduling in the implemention … if a PE lose a customer connection, and
> customer has a backup connection and backup PE has to send a new BGP update
> to propagate the new path. The time for RR to propagate the BGP update is
> critical to track because this will condition convergence time for the
> customer …
>
>
>
> Thoughts ?
>
>
>
>
>
> Stephane
>
>
>
>
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of *Robert
> Raszuk
> *Sent:* Tuesday, July 22, 2014 11:15
> *To:* LITKOWSKI Stephane SCE/IBNF; Jeffrey Haas
> *Cc:* idr wg
> *Subject:* draft-litkowski-idr-rtc-interas
>
>
>
> Stephane,
>
>
>
> You have focused on making sure that BGP control plane works fast and
> solid etc. That is great and timestamp idea at each BGP hop would help.
>
>
>
> However let's observe that BGP control plane is NOT congruent with data
> plane. In other words you may have a case where someone may have slow and
> overloaded RR but uses 40 GB pipes for data plane while other AS installed
> super-hyper x86 RRs and has 1 GB links for data plane.
>
>
>
> The possible simple fix is to mandate that timestamp is ONLY added when
> your change next hop to self or that you can have a new flag "N" - Next Hop
> Set which will help a lot to make more use cases for this proposal.
>
>
>
> Best,
>
> R.
>
>
>
>
>
>
>

---

From: <[email-masked]>
Date: Tue, 22 Jul 2014 19:38:30 +0000

I think we are inline …
But not all of the customers requires 50msec or < 1sec convergence time. For most of the traffic, some seconds is acceptable and there is no need to deploy dataplane protection for such purpose. But anyway, we have to track abnormal behaviors of the controlplane (race conditions, corner cases …) where it does not do its job correctly …

I was also thinking about propagation time of C-multicast route (at least type 5/6/7) which is critical … and customer cannot wait 30/50 seconds (or some hours … don’t laugh, it’s not a joke …) to receive the group membership …

Our proposal does not prone in anyway to use controlplane or dataplane to manage any redundancy …

There is a need to monitor BGP controlplane behavior and especially update propagation time : the basic use case is really to ensure that everything goes well … compared to the base line we evaluated in labs or theory.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Tuesday, July 22, 2014 15:30
To: LITKOWSKI Stephane SCE/IBNF
Cc: Jeffrey Haas; idr wg
Subject: Re: draft-litkowski-idr-bgp-timestamp

Stephane,

If you build your service counting on control plane convergence I think you are doing bad thing :) Sorry.

If this draft is to further help with such designs I would clearly switch to "no support" immediately. I had other use cases in mind which did not sound negative.

Specifically in your case if one PE of multihomed site looses PE-CE you locally protect the site (10s ms max) and take all time it needs for BGP to converge regardless how long it takes. Of course I am assuming basic rules like unique RD per VRF in place.

Besides I am not sure how timestamp will help you in this case. Leave alone that timestamp may never catch the refresh max CPU load peak so you may incorrectly consider RR as sound when it is really not.

Cheers,
R.

On Tue, Jul 22, 2014 at 9:19 PM, <[email-masked]<mailto:[email-masked]>> wrote:
[Renamed topic]

Hi Robert,

Thoughts ?

Stephane

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Robert Raszuk
Sent: Tuesday, July 22, 2014 11:15
To: LITKOWSKI Stephane SCE/IBNF; Jeffrey Haas
Cc: idr wg
Subject: draft-litkowski-idr-rtc-interas

Stephane,

Best,
R.

Thank you.

---

From: <[email-masked]>
Date: Tue, 22 Jul 2014 19:43:18 +0000

And just to add something else linked to dataplane protection …
Even if we implement a dataplane protection (PE-CE, or even CORE), we are using OAMs to verify that it works fine and does its job correctly … because maybe in 95% of the cases it would work and you will have 50msec of traffic loss, but in some conditions you may fall into a bug or corner case that will put you at some seconds or more.

Here it’s the same, we have some mechanic in place (BGP), we want an OAM (simple probing) to ensure that it does it’s job correctly …

Hope the comparison clarifies my position.

From: Idr [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Tuesday, July 22, 2014 15:39
To: Robert Raszuk
Cc: Jeffrey Haas; idr wg
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

I think we are inline …

Stephane,

Cheers,
R.

Hi Robert,

Thoughts ?

Stephane

Stephane,

Best,
R.

Thank you.

Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Jul 2014 21:52:56 +0200

Ahh ok .. then we are slowly getting more on the same page. Well at least
we are on the same bookshelf now :)

If this is your application why not use BGP Diagnostic Message for it now
part of BGP Operational Message draft which is an IDR WG doc ?

http://tools.ietf.org/html/draft-ietf-idr-operational-message-00

- -

Someone at the IDR I think asked in the room that this should be at least a
separate SAFI. But till we go there let's see if you see problems with
adding this functionality to Operational Message.

Thx,
R.

On Tue, Jul 22, 2014 at 9:38 PM, <[email-masked]> wrote:

>  I think we are inline …
>
> But not all of the customers requires 50msec or < 1sec convergence time.
> For most of the traffic, some seconds is acceptable and there is no need to
> deploy dataplane protection for such purpose. But anyway, we have to track
> abnormal behaviors of the controlplane (race conditions, corner cases …)
> where it does not do its job correctly …
>
>
>
> I was also thinking about propagation time of C-multicast route (at least
> type 5/6/7) which is critical … and customer cannot wait 30/50 seconds (or
> some hours … don’t laugh, it’s not a joke …) to receive the group
> membership …
>
>
>
> There is a need to monitor BGP controlplane behavior and especially update
> propagation time : the basic use case is really to ensure that everything
> goes well … compared to the base line we evaluated in labs or theory.
>
>
>
>
>
> *Sent:* Tuesday, July 22, 2014 15:30
> *To:* LITKOWSKI Stephane SCE/IBNF
> *Cc:* Jeffrey Haas; idr wg
> *Subject:* Re: draft-litkowski-idr-bgp-timestamp
>
>
>
> Stephane,
>
>
>
> Besides I am not sure how timestamp will help you in this case. Leave
> alone that timestamp may never catch the refresh max CPU load peak so you
> may incorrectly consider RR as sound when it is really not.
>
>
>
> Cheers,
> R.
>
>
>
>
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Jul 2014 21:59:10 +0200

I do see very crisp what you are trying to accomplish.

I am concerned that you are proposing wrong tool for it. At least even if
there is some merit to timestamp as an indicator (which I am not that
convinced) I think just new attribute to be added when operator seems fit
to current UPDATEs seems not perfect.

To make the decision of which paths to choose I would consider other
information like CPU usage of RR ? Or free RAM to be able to easily tell
that one RR may have some memory leak etc ... and those information are (I
hope) already being collected by your monitoring stations.

How timestamps are going to ignore all of those and replace to hit best
path selection ?

​r.​

On Tue, Jul 22, 2014 at 9:43 PM, <[email-masked]> wrote:

> Even if we implement a dataplane protection (PE-CE, or even CORE), we are
> using OAMs to verify that it works fine and does its job correctly …
> because maybe in 95% of the cases it would work and you will have 50msec of
> traffic loss, but in some conditions you may fall into a bug or corner case
> that will put you at some seconds or more.
>
>
>
> Here it’s the same, we have some mechanic in place (BGP), we want an OAM
> (simple probing) to ensure that it does it’s job correctly …
>
>
>
> *From:* Idr [mailto:[email-masked]] *On Behalf Of *
> [email-masked]
> *Sent:* Tuesday, July 22, 2014 15:39
> *To:* Robert Raszuk
>
> *Subject:* Re: [Idr] draft-litkowski-idr-bgp-timestamp
>
>
>
> I think we are inline …
>

---

From: <[email-masked]>
Date: Tue, 22 Jul 2014 20:06:32 +0000

No no … this have nothing to deal with best path selection … timestamp are just informations to be used by external tool.
TS attribute is not part of path comparison.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Tuesday, July 22, 2014 15:59
To: LITKOWSKI Stephane SCE/IBNF
Cc: Jeffrey Haas; idr wg
Subject: Re: draft-litkowski-idr-bgp-timestamp

I am concerned that you are proposing wrong tool for it. At least even if there is some merit to timestamp as an indicator (which I am not that convinced) I think just new attribute to be added when operator seems fit to current UPDATEs seems not perfect.

To make the decision of which paths to choose I would consider other information like CPU usage of RR ? Or free RAM to be able to easily tell that one RR may have some memory leak etc ... and those information are (I hope) already being collected by your monitoring stations.

How timestamps are going to ignore all of those and replace to hit best path selection ?

​r.​

On Tue, Jul 22, 2014 at 9:43 PM, <[email-masked]<mailto:[email-masked]>> wrote:
And just to add something else linked to dataplane protection …
Even if we implement a dataplane protection (PE-CE, or even CORE), we are using OAMs to verify that it works fine and does its job correctly … because maybe in 95% of the cases it would work and you will have 50msec of traffic loss, but in some conditions you may fall into a bug or corner case that will put you at some seconds or more.

From: Idr [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Tuesday, July 22, 2014 15:39
To: Robert Raszuk

Cc: Jeffrey Haas; idr wg

I think we are inline …

Stephane,

Cheers,
R.

Hi Robert,

Thoughts ?

Stephane

Stephane,

Best,
R.

Thank you.

Thank you.

Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Jul 2014 22:09:51 +0200

Then why not use BGP Operational Message then ?

Seems like perfect fit to me if this is just a BGP control plane network
propagation speedometer.

Cheers,
R.

On Tue, Jul 22, 2014 at 10:06 PM, <[email-masked]> wrote:

>  No no … this have nothing to deal with best path selection … timestamp
> are just informations to be used by external tool.
>
> *Sent:* Tuesday, July 22, 2014 15:59
>
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>
>

---

From: <[email-masked]>
Date: Tue, 22 Jul 2014 20:10:00 +0000

I don’t really see how bgp ops message can be used for that purpose …
I may see BGP ops message usable as an information collector between the tool and a BGP Speaker, but I don’t see how it could help to retrieve timestamps from all hops on the path vector for a specific NLRI.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Tuesday, July 22, 2014 15:53
To: LITKOWSKI Stephane SCE/IBNF
Cc: Jeffrey Haas; idr wg
Subject: Re: draft-litkowski-idr-bgp-timestamp

Ahh ok .. then we are slowly getting more on the same page. Well at least we are on the same bookshelf now :)

If this is your application why not use BGP Diagnostic Message for it now part of BGP Operational Message draft which is an IDR WG doc ?

- -

Someone at the IDR I think asked in the room that this should be at least a separate SAFI. But till we go there let's see if you see problems with adding this functionality to Operational Message.

Thx,
R.

On Tue, Jul 22, 2014 at 9:38 PM, <[email-masked]<mailto:[email-masked]>> wrote:
I think we are inline …
But not all of the customers requires 50msec or < 1sec convergence time. For most of the traffic, some seconds is acceptable and there is no need to deploy dataplane protection for such purpose. But anyway, we have to track abnormal behaviors of the controlplane (race conditions, corner cases …) where it does not do its job correctly …

Stephane,

Cheers,
R.

Hi Robert,

Thoughts ?

Stephane

Stephane,

Best,
R.

Thank you.

Thank you.

---

From: Jon Mitchell <[email-masked]>
Date: Tue, 22 Jul 2014 17:27:35 -0400

> Consider that one of the RR is getting stuck because of Route-refresh processing of 2M of routes or just busy to do something else due to bad scheduling in the implemention … if a PE lose a custome…

Stephane -

I'm supportive of your goal, however after the presentation I'm still a bit unclear on the need for the BGP extension to propogate the timestamp information.  As was brought up by Saikat today in the WG meeting, if update packing is different for this prefix due to the timestamp attribute then it loses some of it's value for providing an accurate view of your end to end BGP propogation time.  Someone suggested BMP, it appears it already supports this option in paragraph 3 of Section 5 of that draft.  The justification given that instrumenting sessions (BMP/BGP/or other export mechanism for the data) to all routers versus getting the information from egress PE only does not seem to be a large concern if monitoring this route propogation is of importance to the operator.  After all presumably there are a much larger number of PE's in most networks than RR's so instrumenting monitoring all devices is not much more work than just the PE's.  Specifying or identifying which prefixes are monitored (the timestamp is collected and exported for) seems also like something that can be handled by the local implementation based on a community or other attributes.

Jon

---

From: <[email-masked]>
Date: Tue, 22 Jul 2014 22:00:53 +0000

Hi Jon,

Thanks for your comment.
Regarding packing, depending of the address family you are trying to monitor, current packing may be low or high. In case where existing packing is low, it does not change anything. In case packing is high, yes I agree that the processing of the update is a bit changed and my beacon would be sent may be faster. Packing is introducing slight delay but I don't think delay introduced by packing is a major component expect if an implementation get stucks in packing (it may happen but I never seen it). IMHO, I don't think this is a critical point but I'm open to other opinions.

Now for BMP, we are not targeting to monitor our beacons on all PEs, just a subset of representative.
Moreover it's not only a question about number of sessions, it's a question of ordering the information retrieved. If you consider using BMP , the tool will peer with selected PEs, but also "transit BGP Speakers" (ASBRs, RRs ...). When the tool will receive the timestamp information (if implementation of BMP supports timestamp), it requires to sort and reorganize the received information based on the knowledge of the topology : you cannot just sort by timestamp, you have to find relationship between BGP Speakers (information from BMP and topology) and combine them to recreate our timestamp vector. BGP transport permits to create automatically this timestamp vector without having to implement a complex machinery in the tool. But BMP can be used at the selected PE to the tool to retrieve timestamp vectors.

Hope it clarifies.

Stephane

-----Original Message-----
From: Jon Mitchell [mailto:[email-masked]] 
Sent: Tuesday, July 22, 2014 17:28
To: LITKOWSKI Stephane SCE/IBNF
Cc: Robert Raszuk; Jeffrey Haas; idr wg
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

Stephane -

Jon

---

From: Jon Mitchell <[email-masked]>
Date: Tue, 22 Jul 2014 18:22:46 -0400

> Moreover it's not only a question about number of sessions, it's a question of ordering the information retrieved. If you consider using BMP , the tool will peer with selected PEs, but also "transi…

Stephane - yes, I think this second point is the real object of the draft (correct me if I'm wrong).  Offline correlation engines are complicated to implement and/or expensive, although commercially available.  Getting the timestamp data seems not to be the real issue here at all (via BMP or other mechanisms a local implementation can provide).  Maybe this point should be more clear in the intro and text.

Jon

---

From: Thomas Mangin <[email-masked]>
Date: Wed, 23 Jul 2014 08:56:24 +0100

Bonjour Stephane,

I am not at the meeting, so could only work from your draft ... Your proposal seems sensitive and useful to me and I believe it would be a useful addition to the protocol.

IMHO, adding some extra message processing would take some CPU time, taking this CPU from the propagation of UPDATE, so the format should be as simple as possible and require as little parsing as possible. It is not clear to me that a speaker can ignore the validity of the data before appending its TS. Forcing the TS to be validated at every hop would seems to be self-defeating. The format is as well quite complex to parse and I would be in favour of a much more simple one with less flags and/or options.

Regarding format proposed :

Type 1 and Type 2 and later in peer : 16 bits ASN can be coded using 32 bits - why make a distinction - we do not need to know if the speaker considered itself or was configured as a ASN4 speaker. Why not include ASN and IPs ( prepended with AFI/SAFI so it is possible to include other Family later )
Why not always include all possible information ASN, IP, IPv6 ( knowing that only some may be present ). At debugging time, it is not clear what would be most useful therefore having both the AS and IP should be better.

Receiving timestamp from January 1900, I am under the impression that most system have built-in Epoch function from 1970 and with 64 bits would not suffer from any issue with 2038, I wonder why you picked 1990 (which I realise is another option).

Regarding the Peer Flags,I would rather see all the information being included (ASN and IP). Debugging information should try to provide as much information as possible.

4.2, I assume this apply for prefixes which match the selection in 4.1

4.4, I would rather see TS being transmitted by default on IBGP sessions and explicit on EBGP ( easier to configure )

4.5.2 does not say how the new TS should be set. It is not clear if the summary option must be implemented or can be left out ( IMHO, this should be left out the draft and be left to be done by the external tool displaying the information )

4.6 does not indicate what must be done if two TS entries are in the both synchronised but inconsistent, like a time sequence of 1, 3, 2, 5 where a router found a way to go back in time :-) .... Should it be ignored or should this TS sequence be seen as invalid and discarded ( assuming each router is performing some validation of the attribute ).

Only skimmed over 5+ 

Thomas

---

From: <[email-masked]>
Date: Wed, 23 Jul 2014 12:32:46 +0000

Hi Thomas,

Thanks for your interest on our proposal.
Your comments are valuable and for sure, there is some work to do on the encoding which is not optimal for parsing today.

Pls find inline comments.

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Thomas Mangin
Sent: Wednesday, July 23, 2014 03:56
To: [email-masked]
Cc: Jeffrey Haas; idr wg; Robert Raszuk
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

Bonjour Stephane,

Type 1 and Type 2 and later in peer : 16 bits ASN can be coded using 32 bits - why make a distinction - we do not need to know if the speaker considered itself or was configured as a ASN4 speaker. Why not include ASN and IPs ( prepended with AFI/SAFI so it is possible to include other Family later ) Why not always include all possible information ASN, IP, IPv6 ( knowing that only some may be present ). At debugging time, it is not clear what would be most useful therefore having both the AS and IP should be better.
[SLI] Regarding 2B/4B ASN, I would agree ... regarding parsing facility , I was more thinking about a typed timestamp entry for the next version with fixed size format (AS type, IPv4 type, IPv6 type).

Receiving timestamp from January 1900, I am under the impression that most system have built-in Epoch function from 1970 and with 64 bits would not suffer from any issue with 2038, I wonder why you picked 1990 (which I realise is another option).
[SLI] Agree, in fact, as I'm not an expert at all on timing, I just copied/pasted NTPv4 timestamp format :) 

4.4, I would rather see TS being transmitted by default on IBGP sessions and explicit on EBGP ( easier to configure )
[SLI] That makes sense to me

4.5.2 does not say how the new TS should be set. It is not clear if the summary option must be implemented or can be left out ( IMHO, this should be left out the draft and be left to be done by the external tool displaying the information )
[SLI] Agree that this is not clear, will fix it ... 

4.6 does not indicate what must be done if two TS entries are in the both synchronised but inconsistent, like a time sequence of 1, 3, 2, 5 where a router found a way to go back in time :-) .... Should it be ignored or should this TS sequence be seen as invalid and discarded ( assuming each router is performing some validation of the attribute ).
[SLI] No this must not be considered as a format error, because this may happen if router becomes unsynchronized. It would be the job of the tool to understand that there is an issue in timestamp informations in the vector. I can add some words on this.

Only skimmed over 5+ 

Thomas

On 22 Jul 2014, at 23:00, <[email-masked]> <[email-masked]> wrote:

>> I don't really understand why ... even if RR are not in the dataplane (which would be the case for VPN environment), RR behavior is really important to track.
>> 
>> processing of 2M of routes or just busy to do something else due to 
>> bad scheduling in the implemention ... if a PE lose a customer 
>> connection, and customer has a backup connection and backup PE has to 
>> send a new BGP update to propagate the new path. The time for RR to 
>> propagate the BGP update is critical to track because this will 
>> condition convergence time for the customer ...
>> 
>> Thoughts ?
> 
> Stephane -
> 
> ______________________________________________________________________
> ___________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
> exploites ou copies sans autorisation. Si vous avez recu ce message 
> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or 
> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.

---

From: <[email-masked]>
Date: Wed, 23 Jul 2014 16:40:46 +0000

Sure we can update the text.

Thanks,

Stephane

-----Original Message-----
From: Jon Mitchell [mailto:[email-masked]] 
Sent: Tuesday, July 22, 2014 18:23
To: LITKOWSKI Stephane SCE/IBNF
Cc: Robert Raszuk; Jeffrey Haas; idr wg
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

Jon

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 24 Jul 2014 14:50:06 -0400

Jon and Stephan: 

Based on work with vendor boxes for the BMWG BGP work - I would recommend
you timestamp the BGP trace NLRI as you leave the BGP Peer and as you
receive it. Otherwise, you run into the data flow patterns the trace NLRI is
included in causing variation in your flows.

Sue 

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Jon Mitchell
Sent: Tuesday, July 22, 2014 6:23 PM
To: [email-masked]
Cc: Jeffrey Haas; idr wg; Robert Raszuk
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

On 22/07/14 22:00 +0000, [email-masked] wrote:
> Hi Jon,
> 
> Thanks for your comment.
> Regarding packing, depending of the address family you are trying to
monitor, current packing may be low or high. In case where existing packing
is low, it does not change anything. In case packing is high, yes I agree
that the processing of the update is a bit changed and my beacon would be
sent may be faster. Packing is introducing slight delay but I don't think
delay introduced by packing is a major component expect if an implementation
get stucks in packing (it may happen but I never seen it). IMHO, I don't
think this is a critical point but I'm open to other opinions.
> Moreover it's not only a question about number of sessions, it's a
question of ordering the information retrieved. If you consider using BMP ,
the tool will peer with selected PEs, but also "transit BGP Speakers"
(ASBRs, RRs ...). When the tool will receive the timestamp information (if
implementation of BMP supports timestamp), it requires to sort and
reorganize the received information based on the knowledge of the topology :
you cannot just sort by timestamp, you have to find relationship between BGP
Speakers (information from BMP and topology) and combine them to recreate
our timestamp vector. BGP transport permits to create automatically this
timestamp vector without having to implement a complex machinery in the
tool. But BMP can be used at the selected PE to the tool to retrieve
timestamp vectors.

Stephane - yes, I think this second point is the real object of the draft
(correct me if I'm wrong).  Offline correlation engines are complicated to
implement and/or expensive, although commercially available.  Getting the
timestamp data seems not to be the real issue here at all (via BMP or other
mechanisms a local implementation can provide).  Maybe this point should be
more clear in the intro and text.

Jon

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 24 Jul 2014 22:26:55 +0200

Sue your observation is valid however I have two comments to it:

*A* it does not apply to specifically crafted beacons as by such "special
crafting" you eliminate by design the expected variations.

*B* in general case you are right, but the general case is still broken if
timestamp is a property of the BGP path. Path may sit idle in BGP speaker
for minutes or hours and longer before at some event it is promoted as best
and send with say 30 min in/out delta falsely indicating potential problems
with given BGP speaker. And the way to eliminate that is *A* hence perhaps
just sticking to A is sufficient.

best regards,
R.

---

From: <[email-masked]>
Date: Fri, 25 Jul 2014 16:48:10 +0000

Susan,

Thanks for pointing this. We will investigate the possibility to do it from an implementation point of view. Putting timestamp at receive side is easy, at transmit side it may be more complex and really implementation dependant as we would need to try to have a consensus at which part of the sending process, we would need to put the timestamp (Update formatting, Route Queueing for the peer group ...)

Best Regards,

Stephane

-----Original Message-----
From: Susan Hares [mailto:[email-masked]] 
Sent: Thursday, July 24, 2014 14:50
To: 'Jon Mitchell'; LITKOWSKI Stephane SCE/IBNF
Cc: 'Jeffrey Haas'; 'idr wg'; 'Robert Raszuk'
Subject: RE: [Idr] draft-litkowski-idr-bgp-timestamp

Jon and Stephan: 

Based on work with vendor boxes for the BMWG BGP work - I would recommend you timestamp the BGP trace NLRI as you leave the BGP Peer and as you receive it. Otherwise, you run into the data flow patterns the trace NLRI is included in causing variation in your flows.

Sue 

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Jon Mitchell
Sent: Tuesday, July 22, 2014 6:23 PM
To: [email-masked]
Cc: Jeffrey Haas; idr wg; Robert Raszuk
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

> Regarding packing, depending of the address family you are trying to
monitor, current packing may be low or high. In case where existing packing is low, it does not change anything. In case packing is high, yes I agree that the processing of the update is a bit changed and my beacon would be sent may be faster. Packing is introducing slight delay but I don't think delay introduced by packing is a major component expect if an implementation get stucks in packing (it may happen but I never seen it). IMHO, I don't think this is a critical point but I'm open to other opinions.
> 
> Now for BMP, we are not targeting to monitor our beacons on all PEs, 
> just
a subset of representative.
> Moreover it's not only a question about number of sessions, it's a
question of ordering the information retrieved. If you consider using BMP , the tool will peer with selected PEs, but also "transit BGP Speakers"
(ASBRs, RRs ...). When the tool will receive the timestamp information (if implementation of BMP supports timestamp), it requires to sort and reorganize the received information based on the knowledge of the topology :
you cannot just sort by timestamp, you have to find relationship between BGP Speakers (information from BMP and topology) and combine them to recreate our timestamp vector. BGP transport permits to create automatically this timestamp vector without having to implement a complex machinery in the tool. But BMP can be used at the selected PE to the tool to retrieve timestamp vectors.

Jon

---

From: "Susan Hares" <[email-masked]>
Date: Sat, 26 Jul 2014 08:22:12 -0400

Stephane: 

Try looking into time-stamp when BGP packet gets put on the transmit queue.
The purpose is that you avoid the costly inbound policy and outbound policy
cycles.  Let me know if it is just as complex. 

Sue 

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Friday, July 25, 2014 12:48 PM
To: Susan Hares; 'Jon Mitchell'
Cc: 'Jeffrey Haas'; 'idr wg'; 'Robert Raszuk'
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

Susan,

Thanks for pointing this. We will investigate the possibility to do it from
an implementation point of view. Putting timestamp at receive side is easy,
at transmit side it may be more complex and really implementation dependant
as we would need to try to have a consensus at which part of the sending
process, we would need to put the timestamp (Update formatting, Route
Queueing for the peer group ...)

Best Regards,

Stephane

-----Original Message-----
From: Susan Hares [mailto:[email-masked]]
Sent: Thursday, July 24, 2014 14:50
To: 'Jon Mitchell'; LITKOWSKI Stephane SCE/IBNF
Cc: 'Jeffrey Haas'; 'idr wg'; 'Robert Raszuk'
Subject: RE: [Idr] draft-litkowski-idr-bgp-timestamp

Jon and Stephan: 

Sue 

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Jon Mitchell
Sent: Tuesday, July 22, 2014 6:23 PM
To: [email-masked]
Cc: Jeffrey Haas; idr wg; Robert Raszuk
Subject: Re: [Idr] draft-litkowski-idr-bgp-timestamp

Jon

____________________________________________________________________________
_____________________________________________

Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
exploites ou copies sans autorisation. Si vous avez recu ce message par
erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les
pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.

This message and its attachments may contain confidential or privileged
information that may be protected by law; they should not be distributed,
used or copied without authorisation.
If you have received this email in error, please notify the sender and
delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 28 Jul 2014 15:44:16 -0400

Jon,

On Tue, Jul 22, 2014 at 05:27:35PM -0400, Jon Mitchell wrote:
> I'm supportive of your goal, however after the presentation I'm still a
> bit unclear on the need for the BGP extension to propogate the timestamp
> information.  As was brought up by Saikat today in the WG meeting, if
> update packing is different for this prefix due to the timestamp attribute
> then it loses some of it's value for providing an accurate view of your
> end to end BGP propogation time.

This is definitely true.  However, this would also be true of any other
in-bgp mechanism.  This then asks the question of why not use an outside of
BGP mechanism.

> Someone suggested BMP, it appears it
> already supports this option in paragraph 3 of Section 5 of that draft.
> The justification given that instrumenting sessions (BMP/BGP/or other
> export mechanism for the data) to all routers versus getting the
> information from egress PE only does not seem to be a large concern if
> monitoring this route propogation is of importance to the operator. 

And yet, it's a large amount of data.  The requests we've had in terms of
tuning what BMP supplies has been mostly with respect to limiting the amount
of data that is available in the stream.  Being able to watch for the probe
prefix only at the end station accomplishes the main goal.

Certainly, an alternative could be to have a specific BMP session only for
the purposes of watching for the probe routes but this still would involve
catching the routes at each collector along the path.  

Another observation that had been made during the discussion of the initial
draft version was simply the amount of transient noise at the intermediate
nodes in your reflector mesh.  The "ping pong" of convergence adds noise
that you have to ignore.  And honestly, we don't care about the multiple
hits we'd get at a given device - it's not helpful data unless you were
trying to measure reflector convergence artifacts.

> Specifying or identifying which prefixes are
> monitored (the timestamp is collected and exported for) seems also like
> something that can be handled by the local implementation based on a
> community or other attributes.

With exactly the same property you note above about breaking the route
packing. :-)

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 28 Jul 2014 15:48:55 -0400

Sue,

> included in causing variation in your flows.

We chatted further about this in terms of potential implementation impact
with this point in mind.  Here's some interesting observations:

If you have an update group (consistent export policy) where this is
applied, you would need to insert the timestamp on a per-peer basis.  In
order to avoid BGP path churn once you've generated an outbound timestamp,
you'd want it to stay the same to avoid churning path attributes. 

You'd thus timestamp once per outbound peer.  
You'd need to store that timestamp rather than generating it dynamically
each time you send that update.

And interestingly, the best many implementations can do is simply add the
timestamp at the time things are received or sent on the TCP session.  There
is no guarantee how closely that aligns to where it may be in TCP buffers.

Thus, tightly bound timestamps aren't very helpful here.  For the BMWG work,
if you recall, we suggested external measurement to bypass some of these
artifacts.

-- Jeff