Subject: RE: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing

From: "SENGUPTA,ANANDA (A-NewEngland,ex1)" <[email-masked]>
Date: Wed, 7 May 2003 04:38:56 -0600

Well, this is the reason I was readily agreeing to the work on the nomenclature draft, and then debating what tests fit in this WG in the actual test plan draft. It is clear that there is value to this work, and though all the test cases may not fit in this WG, some will.
Ananda
-----Original Message-----
From: Kevin Dubray
To: [email-masked]
Sent: 5/7/03 6:40 AM
Subject: Re: [bmwg] New WG Deliverable Proposal - Core Router Software Accelerated Life Testing

I think the work as embodied by the I-D suggests an interesting and 
useful test
scenario.

But all tests don't necessarily lend themselves to being benchmarks.
For
me, that's the real question:

Will the proposed effort yield a benchmark or an interesting test?

(I have lots of other questions, but let's start with this one. ;-) )

-Kevin

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Cataltepe, Tanju, ALABS" <[email-masked]>
Date: Mon, 12 May 2003 09:32:04 -0500

Hello,

I agree with Shankar's points. For a benchmark to be relevant
from a user or service provider's perspective,

a) it should be based upon data-plane measurements,
b) it should start from a realistic steady-state condition
   in the network control plane.

If these are not taken into account, then we can only
measure the protocol's behavior, and that doesn't
appear to be sufficient.

Tanju

-----Original Message-----
From: Shankar Rao [mailto:[email-masked]]
Sent: Friday, May 09, 2003 4:05 PM
To: 'Kevin Dubray'; [email-masked]
Cc: 'Perser, Jerry'
Subject: RE: [bmwg] New WG Deliverable Proposal - Core Router Software
Accelerated Life Testing

Kevin,

This work, in my opinion is clearly a benchmarking test, like any other
test that may measure benchmarks such as through throughput, latency,
frame loss rate, system recovery, reset etc. This effort will measure
similar benchmarks in the context of each of the configuration modules
proposed (control, data, management, security). The only differnce will
be that such benchmarking parameters will be gathered under a predefined
steady state.

For example, lets consider the routing protocol configuration set under
control plane module. Our measurement units for the ISIS routing
protocol include (Section 3.2.1.1) -

	   ISIS					Enabled/Disabled
	   ISIS-TE				Enabled/Disabled
	   Number of ISIS Adjacencies		Adjacencies
	   Number of ISIS Routes		Routes
	   Number of Nodes per Area		Nodes

Each of these measurement units can contribute to a becnhmark test. Lets
say we want to benchmark the max. number of ISIS routes. One way of
doing this is to run only ISIS on the DUT (and nothing else) and
determine the maximum number of ISIS routes supported with 0% frame loss
rate. Another way might be to do the same test, with a predefined steady
state consisting of all the other protocols that DUT may be required to
support (e.g. BGP, RSVP, etc) and gathering the same test data - how
many ISIS routes did the DUT support under these modified steady state
conditions at 0% frame loss rate? 

Regarding the concern about variability in test results, that by itself
is a benchmark in my opinion. If the DUT does exhibit widely variable
behavior, it is very concerning from an operational standpoint.
Predictability and a uniform response to stimulus is as important as
bare functionality.

In addition, even though it may be the case that increasing the number
of workload factors makes it harder to get a handle on what's going on,
the reality is that the real world works that way - most carriers are
running multiple protocols on their core devices, today. It is
concerning for me as a service provider if I hear that we, as a working
group cannot define how we can get a handle of what is happening on such
core routers even in a controlled environment.

Shankar.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kevin Dubray
Sent: Thursday, May 08, 2003 2:58 PM
To: [email-masked]
Cc: Perser, Jerry
Subject: Re: [bmwg] New WG Deliverable Proposal - Core Router Software
Accelerated Life Testing

Perser, Jerry wrote:
> Kevin,
> 
> I see Scott going down one of two paths: either this proposal is an 
> interesting software verification test, or a mixed benchmark.
> 
> I believe the title is worded such that it looks like a software 
> verification test.  If the path is an interesting test, it does fit in

> the BMWG.

I think the ADs might have something to say about the last premise.
"Interesting" isn't the sole entry condition for this WG.

> On the other hand, if this is a procedure for mixing previous 
> benchmarking RFCs, there is a desire to use it.  We have an RFC to 
> benchmark walking.  We have a benchmark for chewing gum.  But what 
> happens when you walk and chew gum?  We should focus on the metrics, 
> not if we swallow the gum.

It's my experience, that as you increase the number of workload factors
applied to a system under test, the harder it is to keep a handle on
what's going on.  Moreover, exacting replication of the workload is a
non-trivial exercise with a large number of workload factors.  Often,
responses have a wide degree fo variability - run the test once, get one
response; run it again, get another response. Sometimes, this is an 
artifact
of SUT behavior, but sometimes the test equipment behaves less than
uniformly, especially if we start mixing too many metaphors. :-)

Personally, I think this is a great test.   I also believe that the
proposed effort presents too many knobs to successfully produce a
dependable, uniform yardstick from which to gather comparative [core] 
router
reliability measurements.

> It would be valueable work if we expand mixing to more than unicast 
> and multicast frames.  The mcastm-12 I-D standardizes how to benchmark

> two classes of frames.  How about we expand this concept to other 
> benchmarks on a core router?

I'm sure there are many other factors that could be added to make the
operational profile more interesting; however, it also exacerbates the
difficulties highlighted above.

In lieu of this benchmarking forum, I believe the proposed topic is
better specified in a provider's validation test plan, where applicable
context can be specifically tailored to a provider's individual
operational requirements as well as its test equipments' capability.

-Kevin

---

From: "Cataltepe, Tanju, ALABS" <[email-masked]>
Date: Mon, 12 May 2003 09:32:04 -0500

Hello,

Tanju

-----Original Message-----
From: Shankar Rao [mailto:[email-masked]]
Sent: Friday, May 09, 2003 4:05 PM
To: 'Kevin Dubray'; [email-masked]
Cc: 'Perser, Jerry'
Subject: RE: [bmwg] New WG Deliverable Proposal - Core Router Software
Accelerated Life Testing

Kevin,

	   ISIS					Enabled/Disabled
	   ISIS-TE				Enabled/Disabled

Shankar.

-----Original Message-----

-Kevin