Subject: Re: test duration...

From: [email-masked] (Kevin Dubray)
Date: Fri, 24 Apr 1998 09:25:55 -0400

Two other mundane considerations about the BMWG choice of a 30 sec. test 
duration implied by steady state.

   1. Persistence of reachability information.
   2. Convergence to a throughput result.

As you can guess, if the information mandating a frame's path to
its ultimate destination disappears during the test duration, packet
loss attributed correct protocol action vs. DUT/SUT clipping.  Of course,
this information can be pegged via static configuration or increasing
protocol timers.  (As an example, RIP works fine using a 30 sec trial, not
so fine using a 10 minute trial, presuming the tester refreshes routing
information between trials.)

To the second point, the collection of the throughput metric, as defined
by RFC 1944, is an iterative exercise.  Notwithstanding the above, converging
on a throughput metric with 10 minute trials would take longer than most
would like.  

Moreover, if extending the trial to increase the sample period is the
goal, perhaps a different metric, "packet delivery error rates" or
somesuch may be in order.  (E.g., characterizing frame loss/error as
rate of 1 X 10E-12)

You might like to ping Harvard's Network Device Test Lab ([email-masked]).
They've experimented with the throughput heuristics and probably can give
some additional insight.

-Kevin

> From [email-masked] Fri Apr 24 08:27:52 1998
> 
> The issue is whether or not throughput varies as a function of length
> (frames dropped after queues fill, etc)  Throughput should be the steady
> state response of the DUT/SUT under a given offered load.  The duration
> need only be long enough to ensure that a steady state has been reached.
> 
> Jim McQuaid
> 
> NetScout Systems
> 
> 
> [email-masked] on 04/24/98 08:07:34 AM
> 
> Hello,
> I am one of many who do benchmarking within Cisco, and am interested in the
> bmwg's thought process regarding the selection of a 60sec snapshot when
> measuring throughput. I am benchmarking within a multi DUT environment and
> was using a 10minute test duration for throughput testing.
> Obviously this longer test time may impacting the throughput rates, and
> before I reduce the test time to 60 seconds, I'd like your input.
> Thank you.
> Paul Wiggins
> Cisco Systems
> 
> 
> 
> 
> 
>

---

From: Paul Wiggins <[email-masked]>
Date: Fri, 24 Apr 1998 11:32:09 -0400

Hi Kevin,

Thanks for the input. I'll follow your suggestion and contact
[email-masked]. 

What I'm really after is whether it is realistic to expect a constant
throughput rate through a DUT which is in a steady state; ie., if the 60
second throughput rate is truly within 1-2 packets of packet loss, should
one expect the DUT to be able to maintain the 60 second throughput rate for
10 minutes (static routes, only UDP echo-requests, no asynchronous events
which would sap DUT resourceses).

thanks again.

---

From: Paul Wiggins <[email-masked]>
Date: Fri, 24 Apr 1998 11:32:09 -0400

Hi Kevin,

thanks again.

---

From: Paul Wiggins <[email-masked]>
Date: Fri, 24 Apr 1998 12:58:12 -0400

Conrad... thank you for staying with this for I'm very interested in
nailing this issue down.

comments embedded;

At 12:21 PM 4/24/98 -0400, Conrad C. Nobili wrote:
>I just received your e-mail (Scott must have too, though I can tell
>by his idle time that he hasn't seen it yet), though I'm not sure I
>understand what you're asking, so we'll continue discussion here in
>the hopes that together we can help you....
>
>> throughput rate through a DUT which is in a steady state;
>
>This sounds like it must be true by definition of a steady state.
>Maybe you could rephrase that somehow...?

I originally blindly accepted the term "steady state" as defined in
rfc1242's definition of "steady state" within section 3.4 "Constant Load",
and rfc1944's definition within section 24 "Trial duration". Whereby the
DUT was considered in a steady state when the DUT operating parameters are
constant and the DUT is able to continuously support a specific rate.

I'm inferring that if a DUT's throughput rate continues to change over
time, then it is -not- in a steady state (chicken/egg paradigm). The change
in rate over time is then possibly caused by hardware errors or a software
defect (one most likely affecting the DUT's resources).

Have my eyes opened? 

>
>> ie., if the 60
>> second throughput rate is truly within 1-2 packets of packet loss,
>
>I'm not sure what you're trying to express here.  A throughput
>test produces one number, let's call it a "throughput number".  A
>packet loss rate test produces a bunch of numbers, for different
>offered loads.  If you're extremely precise when you rephrase
>this we should be able to understand it....

This might be an issue of convergence between theory vs. reality. I'll try
again. Within a multi DUT environment, executing the throughput tests as
described in rfc1944, is it realistic to expect the same throughput rate
found at the 60 second mark vs. the 10 minute mark. I believe I ended up
answering this above; yes. If not, then the environment either has bad
hardware or software defects, either one preventing the DUTs from entering
a steady state which prevents the identification of a valid throughput number.

yes?

>> should
>There is an implicit expectation that most or at least most of the
>interesting phenomena we're trying to measure are purely load-related.
>I guess that's right -- that's what we're trying to measure.  Of
>course there are other phenomena that we can easily observe while
>we're running tests.  The goal for doing a performance test is to
>get a steady-state result that expresses performance as a function
>of load.  So if you observe non-load-related phenomena while testing
>you need to eliminate them (swap out the bad board giving CRC errors,
>adjust protocol timers causing route aging, whatever) and retest....

Agreed.

Thank you very much. This has been very helpful.

>
>--cn
>
>Conrad C. Nobili  N1LPM  [email-masked]  Harvard University NDTL
>
>