Subject: [bmwg] benchmarking of routing protocols modern routers

From: Ilya Varlashkin <[email-masked]>
Date: Fri, 17 Jun 2011 22:23:08 +0200

Hi,

last several days I've spent playing in the lab turning various knobs related to routing. It occurred to me that modern routers provide quite a challenge to measuring performance of routing protocols on modern routers by observing forwarding-plane. The reason for this is FIB optimisation done on many modern platforms that results in traffic flow restored even before routing protocol converges. 

For example, I had a router where forwarding is done in software. After a link failure traffic flow has been restored within few milliseconds (less than 200ms for 10K unique destinations, and in some iterations even less than 20ms), whereas BGP hasn't finished recalculations until a minute or two later. Why did I get traffic restored before convergence completed? Because even prior failure router has pre-calculated backup path (note I'm not using MPLS FRR or IP LFA) and installed it in the FIB; when failure has been detected router simply switched to next-best route in fraction of a second and packets continued flowing through the router despite BGP was still busy with calculations. Such behaviour makes routing protocols benchmarking methodology based on observation of data-plane packets not applicable. One may argue that such optimisation should be disabled prior the tests, but that's not always feasible - some platforms (often found in the core of modern networks) cannot perform any different. Even if one can disable the optimisation, results of the benchmarking tests will be not necessarily trustworthy because chances are that vendor provided non-optimised mode only as last resort and routing software in this mode not necessarily will behave the same way as when optimisation is enabled.

Importance of this observation is that at least for BGP we have to abandon forwarding-plane-based benchmarking and start devising methodology that would look at the control-plane. This can be done by having source and sink ports of the traffic generator/protocol similator capturing routing protocol packets in parallel and measuring time of arrival of those packets rather than concluding (falsely) that "if I can pass traffic then routing protocol has converged".

What's WG in general and Al's (as chair) opinion on this?

Cheers,
iLya

---

From: Al Morton <[email-masked]>
Date: Sat, 18 Jun 2011 11:27:27 -0400

At 04:23 PM 6/17/2011, Ilya Varlashkin wrote:
>...
>Importance of this observation is that at least for BGP we have to 
>abandon forwarding-plane-based benchmarking and start devising 
>methodology that would look at the control-plane. This can be done 
>by having source and sink ports of the traffic generator/protocol 
>similator capturing routing protocol packets in parallel and 
>measuring time of arrival of those packets rather than concluding 
>(falsely) that "if I can pass traffic then routing protocol has converged".
>

First and foremost, thanks for sharing your test results and setting-up
this discussion, Ilya.  The WG is now on a course to learn and benefit
from additional observations and opinions offered on the list.

When Scott Poretsky proposed the IGP-dataplane convergence work in 2003,
it was in response to existing control-plane work (check our web pages
for the RFCs), AND the issue you've identified above didn't appear to
exist. We made lots of these measurements in out lab, and dataplane
restoration times on the order of 10 seconds were typical.

Things have changed, it seems.

Time to weigh-in, experts.

regards,
Al
bmwg chair

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Mon, 20 Jun 2011 16:37:48 +0200

Hi Ilya,

I think it emphasizes the need for correct definitions, test procedures
and specification of initial conditions while having what you observed
in mind.

RFC4098 already defines a Control Plane Convergence while using BGP
protocol messages to detect the state of convergence - so a test
executed
using this definition will provide correct results irrespective of
traffic forwarding.

What you measured would belong into the data plane space and could be
a "rerouting convergence" to restore forwarding upon a link (or next
hop)
failure in the presence of already pre-calculated forwarding table while
some 
kind of "full convergence" could be a metric measuring how long it takes
to build the initial forwarding table from scratch upon the first
routing set-up.

Jan

The climate of Edinburgh is such that the weak succumb young .... 
and the strong envy them.
                                 Dr. Johnson

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ilya Varlashkin
Sent: 17 June 2011 21:23
To: IETF BMWG
Subject: [bmwg] benchmarking of routing protocols modern routers

Hi,

last several days I've spent playing in the lab turning various knobs
related to routing. It occurred to me that modern routers provide quite
a challenge to measuring performance of routing protocols on modern
routers by observing forwarding-plane. The reason for this is FIB
optimisation done on many modern platforms that results in traffic flow
restored even before routing protocol converges. 

For example, I had a router where forwarding is done in software. After
a link failure traffic flow has been restored within few milliseconds
(less than 200ms for 10K unique destinations, and in some iterations
even less than 20ms), whereas BGP hasn't finished recalculations until a
minute or two later. Why did I get traffic restored before convergence
completed? Because even prior failure router has pre-calculated backup
path (note I'm not using MPLS FRR or IP LFA) and installed it in the
FIB; when failure has been detected router simply switched to next-best
route in fraction of a second and packets continued flowing through the
router despite BGP was still busy with calculations. Such behaviour
makes routing protocols benchmarking methodology based on observation of
data-plane packets not applicable. One may argue that such optimisation
should be disabled prior the tests, but that's not always feasible -
some platforms (often found in the core of modern networks) cannot pe
 rform any different. Even if one can disable the optimisation, results
of the benchmarking tests will be not necessarily trustworthy because
chances are that vendor provided non-optimised mode only as last resort
and routing software in this mode not necessarily will behave the same
way as when optimisation is enabled.

Importance of this observation is that at least for BGP we have to
abandon forwarding-plane-based benchmarking and start devising
methodology that would look at the control-plane. This can be done by
having source and sink ports of the traffic generator/protocol similator
capturing routing protocol packets in parallel and measuring time of
arrival of those packets rather than concluding (falsely) that "if I can
pass traffic then routing protocol has converged".

Cheers,
iLya

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "iLya" <[email-masked]>
Date: Mon, 20 Jun 2011 17:07:06 +0200

Hi Jan,

--------------------------------------------------
> to build the initial forwarding table from scratch upon the first

The problem here is not in choosing right name, but conclusions drawn from 
observing traffic restored. Particularly, in BGP draft presented during 
IETF80 several tests say that when packets have been observed at sink port 
then protocol has converged. This is what I think this is inaccurate and 
needs to be addressed.

Kind regards,
iLya

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Mon, 20 Jun 2011 17:12:23 +0200

> then protocol has converged. This is what I think this is inaccurate
and 

I agree - it is down to the metric definition and the corresponding
measurement 
definition for the metric - that draft did not relate the two things
together.

Jan

---

From: CAUCHIE Grégory <[email-masked]>
Date: Mon, 20 Jun 2011 17:26:00 +0200

Hi all,

Ilya, I have troubles understanding your point when you mention 
pre-calculated path without enabling LFAs or FRR features. Are you 
considering features like prefix-independant-convergence (that can have 
different names from the different providers) in such case?

Getting then to your point about forwarding-plane-based and 
control-plane-based benchmarking, I think the main question is "what do 
we want to acheive at the end?". As a protocol is a mean to dynamically 
react to network events in order to detour a traffic around a failure, 
having user traffic in a test seems to fit naturaly in a protocol 
benchmarking. But to get comparable results, a benchmarking procedure 
must ensures that every participants are fighting "on equal terms", and 
IINM that's where Ilya argument's come to say that it can not be always 
possible and I am in line with him. However, from your 
control-plane-based definition, what would represent a figure "just 
showing" the time that a router needs to process a "received update + 
best path selection + sending an update" if there is no link to traffic 
at the end? indeed we will have an idea of BGP processing performance, 
but can we call it convergence if we didn't observe also a FIB update? 
(that was your "falsely" comment Ilya, right?).
It seems to me that in both cases we are missing something, an I tend to 
think that an interesting solution would be to couple forwarding and 
control planes. For instance, update messages could be originated by a 
control-plane-generator or in reaction to a link-failure, then processed 
by to SUT in cascade. Benchmarking could be done one the two figures 
gained from each SUT convergence time.

Here are my 2 cents,

Greg

Le 17/06/2011 22:23, Ilya Varlashkin a écrit :
> For example, I had a router where forwarding is done in software. After a link failure traffic flow has been restored within few milliseconds (less than 200ms for 10K unique destinations, and in some iterations even less than 20ms), whereas BGP hasn't finished recalculations until a minute or two later. Why did I get traffic restored before convergence completed? Because even prior failure router has pre-calculated backup path (note I'm not using MPLS FRR or IP LFA) and installed it in the FIB; when failure has been detected router simply switched to next-best route in fraction of a second and packets continued flowing through the router despite BGP was still busy with calculations. Such behaviour makes routing protocols benchmarking methodology based on observation of data-plane packets not applicable. One may argue that such optimisation should be disabled prior the tests, but that's not always feasible - some platforms (often found in the core of modern networks) cannot pe
>   rform any different. Even if one can disable the optimisation, results of the benchmarking tests will be not necessarily trustworthy because chances are that vendor provided non-optimised mode only as last resort and routing software in this mode not necessarily will behave the same way as when optimisation is enabled.
>

---

From: "Howard C. Berkowitz" <[email-masked]>
Date: Mon, 20 Jun 2011 12:44:11 -0400

I'm sorry; are you speaking of RFC 4098 or a more recent draft?

> From: [email-masked] [mailto:[email-masked]] On Behalf Of iLya
> Sent: Monday, June 20, 2011 11:07 AM
> To: Jan Novak (janovak); IETF BMWG
> Subject: Re: [bmwg] benchmarking of routing protocols modern routers
> 
> Hi Jan,
>

---

From: Al Morton <[email-masked]>
Date: Mon, 20 Jun 2011 14:27:42 -0400

At 12:44 PM 6/20/2011, Howard C. Berkowitz wrote:

I'm sorry; are you speaking of
RFC 4098 or a more recent draft?

more recent:

http://tools.ietf.org/html/draft-papneja-bgp-basic-dp-convergence-01

---

From: Ilya Varlashkin <[email-masked]>
Date: Mon, 20 Jun 2011 23:56:21 +0200

Greg,

On Jun 20, 2011, at 17:26 , CAUCHIE Grégory wrote:

> Hi all,
> 
> Ilya, I have troubles understanding your point when you mention pre-calculated path without enabling LFAs or FRR features. Are you considering features like prefix-independant-convergence (that can have different names from the different providers) in such case?
> 

I can assure you that LFA and FRR were not involved in my tests. All I did was configuring router for installing alternative BGP route for every known prefix and tuned failure detection and IGP timers. Failure scenarios included a) change of IGP path to next-hop used in BGP routes but not change of that next-hop itself; b) disappearance of BGP next-hop from IGP. You may call it PIC feature, though that's not necessarily unambiguous description. In some cases IGP convergence played its role but so did protocol-independent failure detection (and it's not BFD). You're right that many platforms have something similar to what I've described, but they require different degree of configuration. At one extremity all possible optimisations are on by default, at the other side of the spectrum systems that require everything to be configured, and there are those falling in between.

> Getting then to your point about forwarding-plane-based and control-plane-based benchmarking, I think the main question is "what do we want to acheive at the end?". As a

That's a very good question, but you also need to bear in mine whom are you asking. From operator perspective (like you and me) we want fast traffic restoration, but vendors and researchers may want to know which sub-system is the bottleneck so they can optimise it and make traffic restoration even faster. These two groups will give different answer to that question. So in fact we need to measure both data- (for operators) and control-plane (for implementers) performance. But we need to be clear what exactly is being measured. When you're observing traffic restoration you can only be certain that outage lasted so long, but you can't declare that duration of this outage is protocol convergence speed. My mail was to bring this to attention of the WG so that we're clear what exactly is being measured and if that measurement is in line with objectives set by the document.
 
> protocol is a mean to dynamically react to network events in order to detour a traffic around a failure, having user traffic in a test seems to fit naturaly in a protocol benchmarking. But to get comparable results, a benchmarking procedure must ensures that every participants are fighting "on equal terms", and IINM that's where Ilya argument's 

Having user traffic in a test was acceptable for a protocol benchmarking when convergence depended on protocol speed. As this is no longer the case with modern systems where convergence leverages clever tricks with FIB, benchmarking based on user traffic no longer measures protocols performance. It should be very clear that there is traffic restoration and there is protocol convergence, measuring one of them is no substitute for measuring another.

> come to say that it can not be always possible and I am in line with him. However, from your control-plane-based definition, what would represent a figure "just showing" the time that a router needs to process a "received update + best path selection + sending an update" if there is no link to traffic at the end? indeed we will have an idea of BGP processing performance, but can we call it convergence if we didn't observe also a FIB update? (that was your "falsely" comment Ilya, right?).

Let's first sort out what exactly we're going to measure and then we can focus on devising a method for doing it. At the moment it seems like there is no clear consensus on what we're dealing with. 

> It seems to me that in both cases we are missing something, an I tend to think that an interesting solution would be to couple forwarding and control planes. For instance, update messages could be originated by a control-plane-generator or in reaction to a link-failure, then processed by to SUT in cascade. Benchmarking could be done one the two figures gained from each SUT convergence time.
> 

From my side I'd object to putting all into one basket because it makes us to deal with too many variables and whatever we come up with most likely will be incomplete or not truly fair. I also think that having forwarding- and control-plane convergences addressed separately will be of better value.

Kind regards,
iLya

---

From: CAUCHIE Grégory <[email-masked]>
Date: Tue, 21 Jun 2011 17:08:56 +0200

Hi Ilya,

please find my comments inline.

Best regards,

Greg

> I can assure you that LFA and FRR were not involved in my tests. All I did was configuring router for installing alternative BGP route for every known prefix and tuned failure detection and IGP tim…

[GC] in you first mail you said:

<quote>
For example, I had a router where forwarding is done in software. After a link failure traffic flow has been restored within few milliseconds (less than 200ms for 10K unique destinations, and in some iterations even less than 20ms), whereas BGP hasn't finished recalculations until a minute or two later.
</quote>

it seems that your observation is logic for case a) but shouldn't appear 
in case b), right? Is it what you wanted to say or did I miss something?

> That's a very good question, but you also need to bear in mine whom are you asking. From operator perspective (like you and me) we want fast traffic restoration, but vendors and researchers may want to know which sub-system is the bottleneck so they can optimise it and make traffic restoration even faster. These two groups will give different answer to that question. So in fact we need to measure both data- (for operators) and control-plane (for implementers) performance.

[GC] In my understanding, what we want to acheive is to gain the 
knowledge of the time needed to restore communication in the network. 
Because BMWG is chartered to produce benchmarking in lab environments, 
our testing only involves few routers. There, I agree with you  when 
saying that reducing the "time to restore communications in the network" 
into "the time for one router to change its outgoing interface" is 
clearly a too simplistic approach.
However, I tend to disagree on the fact that operators are only 
concerned by data-plane and vendors only by control-plane. As both are 
closely related, at least I care about both planes. And specifically for 
BGP, I care also about the control-plane as I know it will trigger 
convergence in the following peers of a RR-hierarchy for instance.
But at the end, I have the same opinion regarding the fact that we need 
to measure both data-plane and control-plane convergence.

>   But we need to be clear what exactly is being measured. When you're observing traffic restoration you can only be certain that outage lasted so long, but you can't declare that duration of this outage is protocol convergence speed. My mail was to bring this to attention of the WG so that we're clear what exactly is being measured and if that measurement is in line with objectives set by the document.

[GC] That's how I understood it and I am clearly in-line with you here.

> Having user traffic in a test was acceptable for a protocol benchmarking when convergence depended on protocol speed. As this is no longer the case with modern systems where convergence leverages c…

[GC] well it depends. As I said before, traffic restoration on one 
router is not always equal to traffic restoration for the whole network. 
However, I think you will agree that in absence of features like LFA or 
FRR, traffic restoration in the whole network implies protocol 
convergence. But then, the question is how to measure protocol 
convergence? For sure with BGP, when you observe an outgoing Update it 
means that all three RIBs (in, local and out) have converged. But in 
IGP, observing PDUs will not help in measuring the whole IGP convergence 
as flooding is performed before RIB updates. So again, the tricky thing 
as you've said is to be clear about what we are measuring exactly. 
Refering only to traffic or PDU may not be the right way to proceed. I 
would say that a preferred way shall be to aim for either control-plane 
or data-plane convergence, and see from the different mechanisms 
invovled how this can be acheived and to which extent it remains valid.

> Let's first sort out what exactly we're going to measure and then we can focus on devising a method for doing it. At the moment it seems like there is no clear consensus on what we're dealing with.

[GC] +1
If you are clearly refering to the 
draft-papneja-bgp-basic-dp-convergence, I think however that we shall 
wait and see for next document release after the many comments authors 
received in the last meeting.

---

From: Ilya Varlashkin <[email-masked]>
Date: Tue, 21 Jun 2011 20:57:11 +0200

Hi Greg,

On Jun 21, 2011, at 17:08 , CAUCHIE Grégory wrote:

> it seems that your observation is logic for case a) but shouldn't appear in case b), right? Is it what you wanted to say or did I miss something?
> 

No. Link failure can result in both (a) & (b) cases. It doesn't have to be failure of a direct link. And under certain conditions can also include link from ASBR to another network. But let's suspend this discussion until we sort out the objectives of the tests.

> [GC] In my understanding, what we want to acheive is to gain the knowledge of the time needed to restore communication in the network. Because BMWG is chartered to produce benchmarking in lab environments, our testing only involves few routers. There, I agree with you  when saying that reducing the "time to restore communications in the network" 

> into "the time for one router to change its outgoing interface" is clearly a too simplistic approach.

I cannot agree that lab environments are limited to few routers - with a protocol simulator I have generated virtual routers grids bigger than my actual network. In my last tests I had limited myself to 10K routes only because I didn't have protocol simulator available at the location where software routers were interconnected. However by observing the way router behaved, I'm confident that for 500K routes my convergence time would have been the same provided routers have sufficient memory.

> However, I tend to disagree on the fact that operators are only concerned by data-plane and vendors only by control-plane. As both are closely related, at least I care about both planes. And specifically for BGP, I care also about the control-plane as I know it will trigger convergence in the following peers of a RR-hierarchy for instance.
> But at the end, I have the same opinion regarding the fact that we need to measure both data-plane and control-plane convergence.
> 

Ok, I can accept that at least on operators' side both observation might be of an interest. But I still maintain position that we should address forwarding plane traffic restoration and control-plane protocols performance separately because there is different value (not in terms of actual numbers but in terms of worthiness) in the results. And also too complex system is more difficult to understand and to control.

> 
> [GC] well it depends. As I said before, traffic restoration on one router is not always equal to traffic restoration for the whole network. However, I think you will agree that in absence of features like LFA or FRR, traffic restoration in the whole network implies protocol convergence. But then, the question is how to measure protocol

No, it doesn't imply need for protocol convergence. Simplest example is an MPLS network, but even in plain hop-by-hop IP network in some scenarios protocol convergence can happen after traffic is restored.

> convergence? For sure with BGP, when you observe an outgoing Update it means that all three RIBs (in, local and out) have converged. But in IGP, observing PDUs will not help in measuring the whole IGP convergence as flooding is performed before RIB updates. So again, the tricky thing as you've said is to be clear about what we are measuring
> exactly. Refering only to traffic or PDU may not be the right way to proceed. I would say that a preferred way shall be to aim for either control-plane or data-plane convergence, and see from the different mechanisms invovled how this can be acheived and to which extent it remains valid.
> 

With IGP-only network and no LFA you may indeed get trustful results by observing data-plane. However in IGP+BGP things get more complicated. And I wouldn't rule out possibility of a routeing platform that would have LFA built into its architecture without way to disable it. So again, lets sort out what are the objectives and then see what methods are appropriate with modern routers and what aren't.

> If you are clearly refering to the draft-papneja-bgp-basic-dp-convergence, I think however that we shall wait and see for next document release after the many comments authors received in the last meeting.
> 
It was inspired by that draft.

Cheers,
iLya

---

From: CAUCHIE Grégory <[email-masked]>
Date: Wed, 22 Jun 2011 16:14:57 +0200

Hi Ilya,

Le 21/06/2011 20:57, Ilya Varlashkin a écrit :
> No. Link failure can result in both (a)&  (b) cases. It doesn't have to be failure of a direct link. And under certain conditions can also include link from ASBR to another network. But let's suspend this discussion until we sort out the objectives of the tests.

[GC] I agree with you but still don't get it all, but I have no problems 
suspending this discussion.

> I cannot agree that lab environments are limited to few routers - with a protocol simulator I have generated virtual routers grids bigger than my actual network. In my last tests I had limited myse…

[GC] I was meaning few "physical routers" or few SUT if you prefer.

> Ok, I can accept that at least on operators' side both observation might be of an interest. But I still maintain position that we should address forwarding plane traffic restoration and control-pla…

[GC] I think I never said my position was to test either one of the two. 
On the contrary, my reactions since your first mail are supporting the 
need of addressing both planes. If it wasn't clear due to my english, 
then I send my apologies.
Then, having in mind that both planes must be addressed, one point was 
either to use or not traffic and PDUs to test one plane. There I was 
arguing that for instance traffic may sometimes be needed for some 
control plane measurements, which means that using traffic for 
forwarding-plane and PDUs for control-plane is not sufficient in the 
general case. I am talking of the general case as your thread title is 
inviting me in that direction, but maybe your point was to restrict the 
discussion to BGP?

> No, it doesn't imply need for protocol convergence. Simplest example is an MPLS network, but even in plain hop-by-hop IP network in some scenarios protocol convergence can happen after traffic is r…

Sorry Ilya I don't follow you here. Could you elaborate a little bit more?
I am wondering if we have the same definition about protocol 
convergence. To me convergence happens when all best routes are 
installed in all routers FIB, implying that when one LSA/LSP is flooded 
but its computation doesn't change the best route, then convergence is 
already done.
Is your definition of convergence implying that not only all FIBs but 
also all RIBs are updated?

> With IGP-only network and no LFA you may indeed get trustful results by observing data-plane. However in IGP+BGP things get more complicated. And I wouldn't rule out possibility of a routeing platf…

[GC] I agree with you and think that was what my last sentence was 
suggesting.

> It was inspired by that draft.

[GC] By "it" do you refer to this thread? if so, as we (including Sarah 
Banks IINM) raised the same sort of comments during the meeting, 
shouldn't we wait to see how authors have made the document evolved? 
otherwise, what is your position (sorry it's not clear to me)?

Cheers,
--
Greg

---

From: "Rajiv Papneja" <[email-masked]>
Date: Tue, 28 Jun 2011 14:34:44 -0400

Hi Ilya,

We understand the concerns that you raise in terms of received data not
being an accurate measure of BGP protocol convergence and we need to account
for control plane convergence into the data plane, however based on the
handling of IGP benchmarking, we decided to tackle the data plane
convergence ahead of the control plane convergence

Based on our initial feedback from the group, and sharing the experiences of
the carriers it was felt that it is important to measure how soon
protocol/device reacts dynamically to the network events (triggers) in
detouring the traffic around a failure and restoring the customer traffic or
services on recovery. This was also a natural approach looking at how
historically protocol convergence has been tackled in the service provider
labs. We are not arguing that control plane convergence is not important,
but the current draft however deals with the data plane convergence and as
Greg correctly pointed out that for the control plane convergence
measurements one may need traffic as well.

We do not want to comment on the particular setup that triggered the
concern, and what vendor was used in your test and why BGP process was still
converging when the traffic was already routed to the backup path. But it is
important to document all the configurations or topology parameters in the
initial conditions (reporting) so that all implementations are tested on
common grounds. If one implementation behaves uniquely than others, it
should be reported effectively to highlight the differences. 

We are in agreement that measurements should accurately measure the protocol
convergence, but it is also critical to measure how the implementation
(software based or hardware based) reacts to the network failure and how
soon it switches to the backup path if available or reconverges. If a vendor
provides sub-optimal code, let the testing decide that and let the service
provider decide which implementation they would want to work with. And the
configuration of optimization should be documented as well but if upon
disabling the optimization the implementation continues to behave in
optimization mode, then it should be highlighted as part of the reporting.

Other points to ponder are why would any vendor assume that every deployment
of their device would be in an environment with backup/multiple paths and
provide optimized code for only that scenario. Another thing we have noticed
in the testing is implementations with backup paths usually tend to use
additional resources (CPU, Memory) and not all devices in all deployment
scenarios will/can be upgraded to support that. So vendors would still need
to provide optimized code for non-backup scenarios. Even in scenarios where
a backup path is present, we can still consider tests where we can have
multiple failures (for primary&  backup) and trigger the code to execute the
optimized version in the implementation.

We will try to clarify things in the next version based on AIs from the last
meeting and ML discussions, but question remains that whether we prioritize
control plane benchmarking over data plane or we look for a hybrid approach.
May be Al can comment.  

Cheers,
/Rajiv

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
CAUCHIE Grégory
Sent: Tuesday, June 21, 2011 11:09 AM
To: Ilya Varlashkin
Cc: [email-masked]
Subject: Re: [bmwg] benchmarking of routing protocols modern routers

Hi Ilya,

Best regards,

Greg

>>
> I can assure you that LFA and FRR were not involved in my tests. All I did
was configuring router for installing alternative BGP route for every known
prefix and tuned failure detection and IGP timers. Failure scenarios
included a) change of IGP path to next-hop used in BGP routes but not change
of that next-hop itself; b) disappearance of BGP next-hop from IGP. You may
call it PIC feature, though that's not necessarily unambiguous description.
In some cases IGP convergence played its role but so did
protocol-independent failure detection (and it's not BFD). You're right that
many platforms have something similar to what I've described, but they
require different degree of configuration. At one extremity all possible
optimisations are on by default, at the other side of the spectrum systems
that require everything to be configured, and there are those falling in
between.

<quote>
For example, I had a router where forwarding is done in software. After a
link failure traffic flow has been restored within few milliseconds (less
than 200ms for 10K unique destinations, and in some iterations even less
than 20ms), whereas BGP hasn't finished recalculations until a minute or two
later.
</quote>

it seems that your observation is logic for case a) but shouldn't appear in
case b), right? Is it what you wanted to say or did I miss something?

>> Getting then to your point about forwarding-plane-based and
control-plane-based benchmarking, I think the main question is "what do we
want to acheive at the end?". As a
> That's a very good question, but you also need to bear in mine whom are
you asking. From operator perspective (like you and me) we want fast traffic
restoration, but vendors and researchers may want to know which sub-system
is the bottleneck so they can optimise it and make traffic restoration even
faster. These two groups will give different answer to that question. So in
fact we need to measure both data- (for operators) and control-plane (for
implementers) performance.

>   But we need to be clear what exactly is being measured. When you're
observing traffic restoration you can only be certain that outage lasted so
long, but you can't declare that duration of this outage is protocol
convergence speed. My mail was to bring this to attention of the WG so that
we're clear what exactly is being measured and if that measurement is in
line with objectives set by the document.

>> protocol is a mean to dynamically react to network events in order to
detour a traffic around a failure, having user traffic in a test seems to
fit naturaly in a protocol benchmarking. But to get comparable results, a
benchmarking procedure must ensures that every participants are fighting "on
equal terms", and IINM that's where Ilya argument's
> Having user traffic in a test was acceptable for a protocol benchmarking
when convergence depended on protocol speed. As this is no longer the case
with modern systems where convergence leverages clever tricks with FIB,
benchmarking based on user traffic no longer measures protocols performance.
It should be very clear that there is traffic restoration and there is
protocol convergence, measuring one of them is no substitute for measuring
another.

>> come to say that it can not be always possible and I am in line with him.
However, from your control-plane-based definition, what would represent a
figure "just showing" the time that a router needs to process a "received
update + best path selection + sending an update" if there is no link to
traffic at the end? indeed we will have an idea of BGP processing
performance, but can we call it convergence if we didn't observe also a FIB
update? (that was your "falsely" comment Ilya, right?).
> Let's first sort out what exactly we're going to measure and then we can
focus on devising a method for doing it. At the moment it seems like there
is no clear consensus on what we're dealing with.

[GC] +1

>> It seems to me that in both cases we are missing something, an I tend to
think that an interesting solution would be to couple forwarding and control
planes. For instance, update messages could be originated by a
control-plane-generator or in reaction to a link-failure, then processed by
to SUT in cascade. Benchmarking could be done one the two figures gained
from each SUT convergence time.
>>
>  From my side I'd object to putting all into one basket because it makes
us to deal with too many variables and whatever we come up with most likely
will be incomplete or not truly fair. I also think that having forwarding-
and control-plane convergences addressed separately will be of better value.

---

From: Ilya Varlashkin <[email-masked]>
Date: Tue, 28 Jun 2011 22:03:35 +0200

Hi Rajiv,

> convergence ahead of the control plane convergence

you cannot use analogy with IGP because IGP requires some smart technique like LFA or MPLS FRR to reroute before SPF completes, in case of BGP things are straight forward - just install next-best path as backup and switch to it when primary is gone (and that happens MUCH before BGP converges). Keep in mind that IGP benchmarking has been conceived several years ago, and equipment has evolved since then. I know one vendor that appears to install backup paths by default (cannot vouch for that as don't have regular access to that equipment) and the other vendor has this as recommended configuration at least for some platforms. When you measure data plane convergence on such platforms, you're measuring anything but BGP, and therefore you cannot call this BGP convergence benchmarking regardless whether you attach word "control-" or "data-plane" to it.

> measurements one may need traffic as well.

I don't reject need for measuring how quick device can shift traffic in case of primary path failure, quite opposite - as an operator I'd like to make such measurements, just don't call it BGP convergence because it isn't.

> optimization mode, then it should be highlighted as part of the reporting.

disabling optimisation may result in executing code, which wasn't intended to be ran outside of special environments. Effectively you would be measuring a device that was not intended to see big world. I'd question value of such tests.

> provide optimized code for only that scenario. Another thing we have noticed

if you have good thing, why would you disable it?

> to provide optimized code for non-backup scenarios. Even in scenarios where

the systems that have that enabled by default are high-end devices that run in the heart of the networks, they have enough resources and that's on these kind of platform(s) optimisation is enabled by default. Lesser systems indeed need explicit config, but then you'd need to write in your draft that it applies only to special sub-set of routeing platforms.

> control plane benchmarking over data plane or we look for a hybrid approach.

I would suggest renaming draft to "control-plane convergence" (i.e. removing "BGP" from it) and stating in Intro what exactly are you trying to measure.

Cheers,
iLya

---

From: Ilya Varlashkin <[email-masked]>
Date: Wed, 29 Jun 2011 00:45:06 +0200

> I would suggest renaming draft to "control-plane convergence" (i.e. removing "BGP" from it) and stating in Intro what exactly are you trying to measure.

Correction - I've meant to say "data/forwarding-plane convergence".

/iLya

---

From: "Rajiv Papneja" <[email-masked]>
Date: Thu, 30 Jun 2011 10:10:44 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ilya
Varlashkin
Sent: Tuesday, June 28, 2011 6:45 PM
To: Rajiv Papneja; IETF BMWG; [email-masked]
Subject: Re: [bmwg] benchmarking of routing protocols modern routers

>> We will try to clarify things in the next version based on AIs from 
>> the last meeting and ML discussions, but question remains that 
>> whether we prioritize control plane benchmarking over data plane or we
look for a hybrid approach.
>> May be Al can comment.  
>> 
> 
> I would suggest renaming draft to "control-plane convergence" (i.e.
removing "BGP" from it) and stating in Intro what exactly are you trying to
measure.

[RP]  I believe that we should let the group, and the group leadership
decide what this work should be called. We had presented the work before we
submitted the draft in IETF 76 and 77 and in those meetings the discussions
were limited to the prioritization of control vs. data plane benchmarking.
AFAIR, the discussion on changing the motivation or not calling this as BGP
convergence did not arise. We are flexible and would want to produce work
which is useful to the carrier community, and we do not have any
reservations against calling it "data/forwarding-plane convergence," but
would like to hear opinion of other SPs, fellow BMWGers and the chair. 

I really thank you for bringing this discussion up. We respect your opinion
and would like to work with you in getting this where it should be. We have
a lot of test cases and this set  should produce a great benchmark for
establishing a fair playing field for different vendors products.  

Rgds,
/Rajiv

/iLya