Subject: RE: [bmwg] Which is correct?

From: "John Dawson" <[email-masked]>
Date: Thu, 5 May 2005 13:33:34 -0700

About 7 years ago, I believe it was at IETF-42, I had presented to the BMWG
a white-paper about Latency measurements and an effect, that at the time, I
called transmit synchronization.  

Pretty much the white-paper said that using 2544 as the basis for creating
traffic flows, you are going to get nothing but junk measurements when it
comes to latency.

Since then there has been no substantial body of work relative to Latency
measurements.

In my estimation, RFC2544 isn't going to cut it when it comes to latency.
You would be better off throwing the entire Methodology out the window and
starting from scratch.  The best approach, that I have been able to dream
up, is that you actually want every port on your test system using a
different gap setting -- based upon prime number values -- such that ports
can never synchronize to one another and generate artificial delays that
don't really exist.

No surprise on my part that 3 different vendors yield different latency
results.  The more vendors you add, the more different results you can
expect.

The entire thought process behind latency has to change.  But then, that was
well received 8 years ago, so I doubt it will be well received now.  I think
I still have my original research should someone care to review this topic
again.

John D.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jim
McQuaid
Sent: Thursday, May 05, 2005 1:16 PM
To: 'Kevin Dubray'; [email-masked]'
Subject: RE: [bmwg] Which is correct?

One  underlying issue, of course, is a) do the suppliers of these
measurement tools claim to be RFC2544 compliant and b) are they?

BMWG has no control over these issues, more or less, but I think that
they're the right questions.

That said, when you get down to deltas in the dozens of microsecond range
you're in a different zone.

cheers,

Jim

Jim McQuaid
Product Manager  |  NetIQ Corporation

Direct 919.767.0296  |  Fax 919.767.0201  |  www.netiq.com
4825 Creekstone Drive  |  Suite 400  |  Durham, NC, 27703

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kevin Dubray
Sent: Thursday, May 05, 2005 4:00 PM
To: [email-masked]
Subject: [bmwg] Which is correct?

Hello BMWG folk:

I pulled the following post from public a NSP support list.  While
it could generate lots of discussion points in a variety of
dimensions :-), I think I'd like to use the datum to reinforce
a single point.

We, as purveyors of fine benchmarking methodology, should
aspire to derive those methodologies that promote consistent
measurement across multiple measurement implementations.  If
we don't, then the notion of "benchmark" becomes diminished.

Of course, a single data point doesn't constitute a trend, but
presuming a uniform test environment, I thought it an
interesting illustration.

-Kevin

-------------------------------------------------------------
To: juniper-nsp at puck.nether.net
Subject: [j-nsp] Latency FastEthernet & Testing with Anritsu, Agilent,
SmartBits

Hello everybody! I tested M20 (RFC 2544).
The results:
(Juniper M20, 4 Port FastEthernet, Average Latency, in microseconds)

128 Byte
------------------------------------------------------
NetStat   26 mcs
Anritsu MD1230B  20 mcs
Agilent N2X   48 (!!!) mcs (Min=32, Max=71)
Spirent SmartBits 600B  N/A (Min=26, Max=51)
------------------------------------------------------

1024 Byte
------------------------------------------------------
NetStat   100 mcs
Anritsu MD1230B  28 mcs
Agilent N2X   177 (!!!!) mcs (Min=106, Max=293)
Spirent SmartBits 600B N/A (Min=99, Max=289)
------------------------------------------------------

Which are correct?

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg
_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: David Newman <[email-masked]>
Date: Thu, 5 May 2005 15:00:08 -0700 (PDT)

On Thu, 5 May 2005, John Dawson wrote:

> starting from scratch.

We already have headed well down that road. For example, 2544 s calls for 
a backbone traffic pattern in which all transmitters simultaneously target 
one receive port.

How many testers actually do that? In my experience, it is far more common 
to measure "RFC 2544 throughput" using RFC 2889 mesh patterns, which 
deliberately avoid presenting two or more packets to the same destination 
at the same instant.

Of course, the latency measurements from these two patterns will be very 
different.

John's ideas about synchronization are worth reconsidering. I'd also raise 
two more concerns:

1. How do we ensure that the test instrument, cabling, and other factors 
aren't themselves contributing to the latency measurement? I have seen 
this referred to as "intrinsic latency." I don't think it should be 
considered instrinsic; this is extra stuff that needs to be characterized 
and compensated for. It should NOT be charged to the DUT/SUT.

2. This has been brought up before, but is it appropriate to measure 
latency only at the throughput oload, as suggested in RFC 2544?

Reduced oload is certainly one way for vendors to claim "better" results. 
Whether it's appropriate to do so is a whole other question.

dn