Subject: test duration...

From: Paul Wiggins <[email-masked]>
Date: Fri, 24 Apr 1998 08:07:34 -0400

Hello,

I am one of many who do benchmarking within Cisco, and am interested in the
bmwg's thought process regarding the selection of a 60sec snapshot when
measuring throughput. I am benchmarking within a multi DUT environment and
was using a 10minute test duration for throughput testing. 

Obviously this longer test time may impacting the throughput rates, and
before I reduce the test time to 60 seconds, I'd like your input.

Thank you.

Paul Wiggins
Cisco Systems

---

From: [email-masked] (Kevin Dubray)
Date: Fri, 24 Apr 1998 09:25:55 -0400

Two other mundane considerations about the BMWG choice of a 30 sec. test 
duration implied by steady state.

   1. Persistence of reachability information.
   2. Convergence to a throughput result.

As you can guess, if the information mandating a frame's path to
its ultimate destination disappears during the test duration, packet
loss attributed correct protocol action vs. DUT/SUT clipping.  Of course,
this information can be pegged via static configuration or increasing
protocol timers.  (As an example, RIP works fine using a 30 sec trial, not
so fine using a 10 minute trial, presuming the tester refreshes routing
information between trials.)

To the second point, the collection of the throughput metric, as defined
by RFC 1944, is an iterative exercise.  Notwithstanding the above, converging
on a throughput metric with 10 minute trials would take longer than most
would like.  

Moreover, if extending the trial to increase the sample period is the
goal, perhaps a different metric, "packet delivery error rates" or
somesuch may be in order.  (E.g., characterizing frame loss/error as
rate of 1 X 10E-12)

You might like to ping Harvard's Network Device Test Lab ([email-masked]).
They've experimented with the throughput heuristics and probably can give
some additional insight.

-Kevin

> From [email-masked] Fri Apr 24 08:27:52 1998
> 
> The issue is whether or not throughput varies as a function of length
> (frames dropped after queues fill, etc)  Throughput should be the steady
> state response of the DUT/SUT under a given offered load.  The duration
> need only be long enough to ensure that a steady state has been reached.
> 
> Jim McQuaid
> 
> NetScout Systems
> 
> 
> [email-masked] on 04/24/98 08:07:34 AM
> 
> Hello,

---

From: [email-masked] (Kevin Dubray)
Date: Fri, 24 Apr 1998 11:37:02 -0400

> duration implied by steady state.

I believe the original choice in the RFC was 60 seconds.  This was
before my time so that's just what I remember from reading it....

> on a throughput metric with 10 minute trials would take longer than most

Indeed that practical consideration is why we actually use 30-second
trial durations.  I believe Scott found through experience that there
wasn't a difference for any of the devices he'd measured so decided
to cut the running time of the tests in half....

Basically, you want to make sure that the buffers are full in the DUT.
If the aggregate rate out of the DUT is lower than the aggregate rate
into it then the buffers will fill and eventually the DUT will drop
packets.  (I'm ignoring flow control schemes here -- other tests shed
light on those.)  You don't want the trial duration to be short enough
for the DUT to be able to make up for getting behind on transmit by
partially filling its buffers and then emptying them before the trial
is counted....

> somesuch may be in order.  (E.g., characterizing frame loss/error as

As you point out though Kevin, the methodology involves a bunch of
trials, to converge on a throughput number, or perform a packet loss
rate test.  Most statistical (as opposed to load-related) errors
should show up with a bunch of 30-second trials as well as with fewer
longer trials.  So you should find these things anyway, and can then
focus in on them if you want....

> They've experimented with the throughput heuristics and probably can give

We're always listening.  I guess I'd say that most statistical problems
turn out to be uninteresting.  They usually turn out to be manufacturing
defects that a board swap will fix.  So the sort of testing that Kevin
mentions above could be very useful for checking production quality, but
less so for checking software quality or generating a product benchmark.
The testing methodology is very good at making it clear whether problems
are load-related, time-related (some other process in the DUT interferes
with forwarding performance periodically), or statistical.  Experience
makes it even more clear....

Memory is cheap today, but networking is getting *very* fast, so a DUT
needs very fast not-so-cheap memory.  At Gigabit speeds you can't make
up for forwarding rate problems with lots of memory, because it really
is *lots* of memory.  So trial durations may not have to be as long
these days as before.  This is just guessing -- I haven't got any real
figures....

--cn

Conrad C. Nobili  N1LPM  [email-masked]  Harvard University NDTL

---

From: [email-masked] (Kevin Dubray)
Date: Fri, 24 Apr 1998 11:42:46 -0400

> before my time so that's just what I remember from reading it....

You're recollection, unlike mine, is correct, Conrad!  RFC 1944, section
24, specifies that a trial of 60 seconds SHOULD be used.  It also provides
for a shorter length trials, but suggest that a full length trial be used.

So, neither RFC 1242 nor RFC 1944 explicitly mandates a 30 second 
trial.

> So the sort of testing that Kevin

So, Conrad gets to the meat of what I was trying to say.  Longer trials
may lend themselve more to quality or reliability more than to performance
benchmarking - not that these classes are mutually exclusive!

-Kevin

---

From: Scott Bradner <[email-masked]>
Date: Fri, 24 Apr 1998 15:17:50 -0400 (EDT)

---
--

as a data point - I use 30 sec in all of my testing

Scott