Subject: RE: [bmwg] New WG Deliverable Proposal - Core Router Software Acc elerated Life Testing

From: Jim McQuaid <[email-masked]>
Date: Wed, 7 May 2003 12:49:37 -0000

I haven't read the ID so this is semi-off the wall, I admit, but I think a
fundamental distinction that ought to be made is between DUT behavior and
DUT performance.  Behavior includes all sorts of operations (like
hotswap-ability and configuration mngt) that are important to operations BUT
not performance metrics in any reasonable sense of the word.  Performance is
more related to capacity planning in a sense, while behavior is related to
manageability and such.

Jim McQuaid
NetIQ Raleigh

-----Original Message-----
From: Kevin Dubray [mailto:[email-masked]]
Sent: Wednesday, May 07, 2003 8:36 AM
To: [email-masked]
Cc: SENGUPTA,ANANDA (A-NewEngland,ex1)
Subject: Re: [bmwg] New WG Deliverable Proposal - Core Router Software
Accelerated Life Testing

In section 3.4, the I-D proposes "evaluation benchmarks". (Since the
proponents produced an I-D, I'm going to assume the I-D's a more complete
statement of intent and scope. My question is, do these tests and
their collection scenario present good BMWG benchmarks?

Consider the ol' manual, cable pull test or its higher-end relative, the
module hotswap test. Experience tells us that these two tests are some
of the finest system-level tests known to routerkind.  You can even measure
some aspect of "recovery time".  But do they lend themselves to good 
"benchmarks"?

-Kevin

SENGUPTA,ANANDA (A-NewEngland,ex1) wrote:
> Well, this is the reason I was readily agreeing to the work on the
nomenclature draft, and then debating what tests fit in this WG in the
actual test plan draft. It is clear that there is value to this work, and
though all the test cases may not fit in this WG, some will.
> Ananda

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Jim McQuaid <[email-masked]>
Date: Wed, 7 May 2003 12:49:37 -0000

Jim McQuaid
NetIQ Raleigh

-----Original Message-----
From: Kevin Dubray [mailto:[email-masked]]
Sent: Wednesday, May 07, 2003 8:36 AM
To: [email-masked]
Cc: SENGUPTA,ANANDA (A-NewEngland,ex1)
Subject: Re: [bmwg] New WG Deliverable Proposal - Core Router Software
Accelerated Life Testing

-Kevin

---

From: "Perser, Jerry" <[email-masked]>
Date: Wed, 7 May 2003 09:17:51 -0700

Kevin,

I see Scott going down one of two paths: either this proposal is an
interesting software verification test, or a mixed benchmark.

I believe the title is worded such that it looks like a software
verification test.  If the path is an interesting test, it does fit in the
BMWG.

On the other hand, if this is a procedure for mixing previous benchmarking
RFCs, there is a desire to use it.  We have an RFC to benchmark walking.  We
have a benchmark for chewing gum.  But what happens when you walk and chew
gum?  We should focus on the metrics, not if we swallow the gum.

It would be valueable work if we expand mixing to more than unicast and
multicast frames.  The mcastm-12 I-D standardizes how to benchmark two
classes of frames.  How about we expand this concept to other benchmarks on
a core router?

Jerry.

> Sent: Tuesday, May 06, 2003 3:41 PM
> To: [email-masked]
> I think the work as embodied by the I-D suggests an interesting and 
> useful test
> scenario.
> 
> But all tests don't necessarily lend themselves to being 
> benchmarks.  For
> me, that's the real question:
> 
> Will the proposed effort yield a benchmark or an interesting test?
> 
> (I have lots of other questions, but let's start with this one. ;-) )
> 
> -Kevin
>