Subject: [bmwg] Which is correct?

From: Kevin Dubray <[email-masked]>
Date: Thu, 05 May 2005 15:59:57 -0400

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While
it could generate lots of discussion points in a variety of
dimensions :-), I think I'd like to use the datum to reinforce
a single point.

We, as purveyors of fine benchmarking methodology, should
aspire to derive those methodologies that promote consistent
measurement across multiple measurement implementations.  If
we don't, then the notion of "benchmark" becomes diminished.

Of course, a single data point doesn't constitute a trend, but
presuming a uniform test environment, I thought it an
interesting illustration.

-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Brough Davis" <[email-masked]>
Date: Thu, 5 May 2005 17:14:50 -0400

I'm curious how many tests were run by each vendor and if the averages were
taken (or just the min).  This is frustrating from the standpoint that
existing benchmark tools are not consistent with the statistical methods
even though they are all performing 2544 tests (which I'm pretty sure they
all are).  Notice the Netstat and Anritsu values are similar to the min
values for the other two systems.  

Because these tests are all in the micro-seconds realm are there any
reasonable number of tests and averages that should be performed...as well
as acceptable standard deviations?  My apologies if these are already
spelled out in the RFC.

Brough 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kevin Dubray
Sent: Thursday, May 05, 2005 4:00 PM
To: [email-masked]
Subject: [bmwg] Which is correct?

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While it could
generate lots of discussion points in a variety of dimensions :-), I think
I'd like to use the datum to reinforce a single point.

We, as purveyors of fine benchmarking methodology, should aspire to derive
those methodologies that promote consistent measurement across multiple
measurement implementations.  If we don't, then the notion of "benchmark"
becomes diminished.

Of course, a single data point doesn't constitute a trend, but presuming a
uniform test environment, I thought it an interesting illustration.

-Kevin

128 Byte

1024 Byte

Which are correct?