Subject: RE: [bmwg] Which is correct?

From: Jim McQuaid <[email-masked]>
Date: Thu, 5 May 2005 20:15:47 -0000

One  underlying issue, of course, is a) do the suppliers of these
measurement tools claim to be RFC2544 compliant and b) are they?

BMWG has no control over these issues, more or less, but I think that
they're the right questions.

That said, when you get down to deltas in the dozens of microsecond range
you're in a different zone.

cheers,

Jim

Jim McQuaid
Product Manager  |  NetIQ Corporation

Direct 919.767.0296  |  Fax 919.767.0201  |  www.netiq.com
4825 Creekstone Drive  |  Suite 400  |  Durham, NC, 27703

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kevin Dubray
Sent: Thursday, May 05, 2005 4:00 PM
To: [email-masked]
Subject: [bmwg] Which is correct?

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While
it could generate lots of discussion points in a variety of
dimensions :-), I think I'd like to use the datum to reinforce
a single point.

We, as purveyors of fine benchmarking methodology, should
aspire to derive those methodologies that promote consistent
measurement across multiple measurement implementations.  If
we don't, then the notion of "benchmark" becomes diminished.

Of course, a single data point doesn't constitute a trend, but
presuming a uniform test environment, I thought it an
interesting illustration.

-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Dawit Birhanu \(dawit\)" <[email-masked]>
Date: Thu, 5 May 2005 17:18:22 -0700

Measuring latency at the throughput oload is actually the main reasons
why you see so many measurement discrepancies. There is always some
deviation in data rate (100 PPM or 10 PPM, depending on the spec). If
you measure latency at line rate, which is the throughput for most
routers/switches especially at large packet size, the slight deviation
in data rate translates into significant latency variations. You can
even get different latency measurement results from the same test
equipment vendor because of this problem.  

-dawit

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of David Newman
> Sent: Thursday, May 05, 2005 3:00 PM
> To: [email-masked]
> Subject: RE: [bmwg] Which is correct?
> 
> On Thu, 5 May 2005, John Dawson wrote:
> 
> > In my estimation, RFC2544 isn't going to cut it when it 
> comes to latency.
> > You would be better off throwing the entire Methodology out 
> the window 
> > and starting from scratch.
> 
> We already have headed well down that road. For example, 2544 
> s calls for a backbone traffic pattern in which all 
> transmitters simultaneously target one receive port.
> 
> How many testers actually do that? In my experience, it is 
> far more common to measure "RFC 2544 throughput" using RFC 
> 2889 mesh patterns, which deliberately avoid presenting two 
> or more packets to the same destination at the same instant.
> 
> Of course, the latency measurements from these two patterns 
> will be very different.
> 
> John's ideas about synchronization are worth reconsidering. 
> I'd also raise two more concerns:
> 
> 1. How do we ensure that the test instrument, cabling, and 
> other factors aren't themselves contributing to the latency 
> measurement? I have seen this referred to as "intrinsic 
> latency." I don't think it should be considered instrinsic; 
> this is extra stuff that needs to be characterized and 
> compensated for. It should NOT be charged to the DUT/SUT.
> 
> 2. This has been brought up before, but is it appropriate to 
> measure latency only at the throughput oload, as suggested in 
> RFC 2544?
> 
> Reduced oload is certainly one way for vendors to claim 
> "better" results. 
> Whether it's appropriate to do so is a whole other question.
> 
> dn
>