Subject: [bmwg] Soak tests

From: "Dr Steven A. Wright" <[email-masked]>
Date: Wed, 20 Jul 2016 11:56:23 +0200

We had some discussion around soak tests, as to what the appropriate duration should be and whether ( or what) the performance metric was under measurement.

I would suggest that these tests should be measured as the duration of stable operation under load. The measurement period starts when  full load is achieved on the system under test and terminated when (1) received load (throughput) variation exceeds some threshold (5%?) or (2) some system exception/ failure occurs. The test may be terminated after some user defined period. The user defined period for the test is usually greater than 24 hrs to catch intermittent/ probabilistic system scheduling conflicts that interfere with stable operation. 

Regards
Steven Wright

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 20 Jul 2016 14:01:36 -0400

Thanks for your comment, Steve.

There is some value in continuing to test after an error,
to see if the system recovers to the pre-error throughput
for example, but I agree that there are likely to be events
after which it would make sense to observe recovery if it happens
and stop.

Al

> -----Original Message-----
> From: bmwg [mailto:[email-masked]] On Behalf Of Dr Steven A.
> Wright
> Sent: Wednesday, July 20, 2016 5:56 AM
> To: [email-masked]
> Subject: [bmwg] Soak tests
> 
> 
> We had some discussion around soak tests, as to what the appropriate
> duration should be and whether ( or what) the performance metric was
> under measurement.
> 
> I would suggest that these tests should be measured as the duration of
> stable operation under load. The measurement period starts when  full
> load is achieved on the system under test and terminated when (1)
> received load (throughput) variation exceeds some threshold (5%?) or (2)
> some system exception/ failure occurs. The test may be terminated after
> some user defined period. The user defined period for the test is
> usually greater than 24 hrs to catch intermittent/ probabilistic system
> scheduling conflicts that interfere with stable operation.
> 
> Regards
> Steven Wright
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Dr Steven A. Wright" <[email-masked]>
Date: Thu, 21 Jul 2016 14:37:39 +0200

I agree with your comment, but was also trying to be sensitive to Pierre's concern as to where to draw the line between a performance benchmarking metric  and a diagnostic procedure. 

My intuition was too recast the soak test around a performance benchmark for the duration of "stable operation under load". Duration is a nice linear time scale for a performance metric. At some point the user has to decide how long of a duration is satisfactory for their application, but that is a separate argument that _may_ be sensitive to the application. the definition of "stable operation under load"  seems quite likely to be application specific....

> On Jul 20, 2016, at 8:01 PM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
>