Subject: Re: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

From: Al Morton <[email-masked]>
Date: Wed, 06 Sep 2006 17:23:05 -0400

I have a few slightly late comments on

>Methodology Guidelines for Accelerated Stress Benchmarking
>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.txt

Al (as a participant)

General:
Much of the material provided here is introduced as examples,
there are no instances of "MUST", "MUST NOT", "REQUIRED", or "SHALL"
in the body of the draft (other than section 2, which describes the
RFC 2119 terminology.  This seems a bit too much flexibility to me.
Certainly there are some mandatory aspects of the methodology,
such as the three phases, reporting formats, etc.

Section 3.2, Test Considerations
>    ...  A particular Configuration Set is defined and the
>    DUT is benchmarked using this configuration set and the
>    Instability Conditions.  Varying Configuration Sets and/or
>    Instability Conditions applied in an iterative fashion can
>    provide an accurate characterization of the DUT
>    to help determine future network deployments.

This seems to be a good lead-in to another sub-section
describing how the results (the "accurate characterization"
mentioned above) will be used. What does it mean to recover all
sessions when the instability conditions are removed, and
how does this influence future network deployments?
How does one determine if the Instability Conditions chosen
are severe enough? Or, too severe?

>    3.3.1 Configuration Sets
>
>    Configuration Sets may include and are not limited to the following
>    examples.

To promote benchmark comparisons, it would seem reasonable to specify
a minimum set of Parameters that MUST be reported.  The mandatory
parameters could be marked with an asterisk in these lists of examples.

>     Example Routing Protocol Configuration Set-
>            PARAMETER                            UNITS
>            BGP                                  Enabled/Disabled
>...
>            Number of MBGP Installed Routes      Routes

Add:
             Type of IGP                          Type (ISIS, OSPF)

>            IGP                                  Enabled/Disabled
>            IGP-TE                               Enabled/Disabled
>            Number of IGP Adjacencies            Adjacencies
>            Number of IGP Routes                 Routes
>            Number of Nodes per Area             Nodes

...

>    Management Configuration Set-
>         PARAMETER                               UNITS
>         SNMP GET Rate                           SNMP 
> Gets/minute
>         Logging                                 Enabled/Disabled
>         Protocol Debug                          Enabled/Disabled
>         Telnet Rate                             Sessions/Hour
>         FTP Rate                                Sessions/Hour
Change to:
          FTP Establishment Rate                  Sessions/Hour

In sections 3.3.2 and 3.3.3, "ISIS" could be replaced with "IGP"
for more generality.

Section 4. Example Test Cases Procedure
Step 10:
>             Optional - Change Configuration Set and/or Instability
>             Conditions for next iteration
This step could be another opportunity to help the reader understand the
results.  "If the results indicate that the Instability Conditions were not 
very stressful (e.g., Unstable Latency did not change, or zero sessions lost),
then increase the severity of the Instability Conditions and repeat."
Additional conditional guidance like this is needed to make the methodology
more concrete and less ambiguous.

Editorial:

Abstract:
s/Acceleratd/Accelerated/

Section 3 heading spacing does not line-up with the left margin.

OLD
    3.2 Test Considerations
    The Accelerated Stress Benchmarking test can be applied in
    service provider test environments to benchmark DUTs under
    stress in an environment that is reflective of an operational
    network.
NEW
    3.2 Test Considerations
    The Accelerated Stress Benchmarking test can be applied in
    service provider test environments to benchmark DUTs under
    stress in an environment that reflects conditions found in
    an operational network.

Section 4. Example Test Cases Procedure
In steps 7 and 9:
s/follow:/follows:/

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Arun Gandhi" <[email-masked]>
Date: Thu, 7 Sep 2006 11:24:37 -0400

Hi Al,

>            Number of Nodes per Area             Nodes

IMHO, we must also add "Number of Areas on a node" (i.e., when DUT is an
ABR).

Thanks!
Arun

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Wednesday, September 06, 2006 5:23 PM
To: [email-masked]
Subject: Re: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

Al (as a participant)

General:

Add:

...

Change to:

very stressful (e.g., Unstable Latency did not change, or zero sessions
lost),

Editorial:

Abstract:

OLD
    3.2 Test Considerations