Subject: Re: Test Duration

From: [email-masked] (Kevin Dubray)
Date: Fri, 24 Apr 1998 14:39:43 -0400

> I'm inferring that if a DUT's throughput rate continues to change over
> time, then it is -not- in a steady state (chicken/egg paradigm). The change
> in rate over time is then possibly caused by hardware errors or a software
> defect (one most likely affecting the DUT's resources).

To restate something Kevin pointed out earlier, a throughput number is
derived from running a series of trials according to the methodology.
This number is just that -- a number -- it doesn't change over time,
so your wording above is confusing.  If the output rate of the device
changes over time then you need to understand the phenomenon that caused
that (the most common case is that the output rate goes to zero as the
box turns into a brick as a result of a bug).  I'm just trying to be
precise with your words here -- as you mention below you really seem
to be interested in throughput derived with trials of one given duration
versus throughput derived with trials of another given duration.  I
guess I'd say that if you find a difference, and you're sure that there
are only load-related phenomena, then the throughput derived from the
longer duration trials (which presumably gives a lower throughput number)
is the better one to use.  Remember, the 60 seconds in the RFC was chosen
as a good approximation of infinity -- sort of an optimization to reduce
the time necessary to produce a benchmark.  Experimentally we've found
that 30 seconds is a good approximation of infinity too.  If you find
something different we'd love to hear more about it....

> This might be an issue of convergence between theory vs. reality. I'll try
> again. Within a multi DUT environment, executing the throughput tests as
> described in rfc1944, is it realistic to expect the same throughput rate
> found at the 60 second mark vs. the 10 minute mark. I believe I ended up
> answering this above; yes. If not, then the environment either has bad
> hardware or software defects, either one preventing the DUTs from entering
> a steady state which prevents the identification of a valid throughput number.

I'm not sure whether there's a real difference in testing one DUT or a
system of many.  The methodology still should work.  The testers and
software shouldn't need to know whether you've got one or more devices
in the middle.  We often test systems of devices....

--cn

Conrad C. Nobili  N1LPM  [email-masked]  Harvard University NDTL