Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

From: "Poretsky, Scott" <[email-masked]>
Date: Fri, 20 Oct 2006 12:45:39 -0400

Hi Arun,

I apologize that this response is so late.  

The BMWG makes it a point to not benchmark 'services', so I recommend we
stick with use of the word 'protocols' and so not replace it with
'services'.  The terms 'black-box' and 'white-box' have been frequently
used in the BMWG over the past 10 years.  I would be happy to use this
work item as the place to define them.  The focus of this work is
black-box benchmarking, consistent with the BMWG charter.  With this
particular work item the working group agreed there would be value in
observing White-Box benchmarks so it was preferred to include that in an
appendix.  At this time there is no plan to make further changes on
these topics unless others in the working group want to continue the
discussion.

In addition, I do not agree with your suggestion to use "number" as a
unit.  Specifically you recommended to replace unit "sessions" with unit
"number".  When answering the question "How many sessions?" could you
answer 10 numbers or 10 sessions.  No change will be made.

Also, there are 3 sentence changes you requested below for which I do
not see the value of those changes.  Maybe you could provide more
information.

Thanks for your thorough review of the documents.

Scott

-----Original Message-----
From: Arun Gandhi [mailto:[email-masked]] 
Sent: Tuesday, August 29, 2006 11:55 AM
To: 'Al Morton'; [email-masked]; Poretsky, Scott; [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Importance: High

Hello Authors,

I am little late in responding wrt the open dates but I suppose my
feedback
will get evaluated. I will also follow up with the feedback on
methodology
document shortly.

<-- Section 1 (Introduction): 

In general I believe this draft should not just address multiple
protocols
operational simultaneously but "different services" being overlaid in a
fully operational box.

> Routers in an operational network are simultaneously configured with
> multiple protocols and security policies while forwarding traffic and
> being managed.

Arun> IMHO, replace "protocols" with "services" as in ISP world (which
is
Arun> the motivation for this draft) there may be different service sets
Arun> such as l3vpn, l2vpns, VPLS, etc. along with routing and signaling
Arun> protocols configured.

> It is helpful to accelerate these network operational conditions so 
> that the router under test can be benchmarked with faster test 
> duration.

Arun> This needs revision/clarification.

> Multiple benchmarks are made for each Benchmark Plane during each 
> Phase.

Arun> Benchmarks are defined for each plane to be characterized in all 
Arun> phases.

> These benchmarks White Box benchmarks are provided in Appendix 1 for
> additional DUT behavior measurements.

Arun> Two points:
Arun> 1) This line needs revision.
Arun> 2) A new term "White" box is added without being explained. Also, 
Arun> the impression I get reading this draft is that it's a black box
Arun> test. Also, since the motivation is the benchmarking of a router
Arun> in ISP domain then for sure it is a black box testing. Regardless
Arun> we should monitor/characterize other system parameters such as 
Arun> "memory usage", "CPU utilization", "install/bring up times", and 
Arun> so on.

<-- Section 3.2.1 (Control Plane)

> The Control Plane defines the Configuration, Startup Conditions, and
> Instability Conditions of the control protocols. Control Plane 
> protocols may include routing protocols, multicast protocols, and 
> MPLS protocols.     

Arun> IMHO, this statement should be revised to be "The Control Plane 
Arun> defines the Configuration, Startup Conditions, and Instability
Arun> Conditions of the control plane protocols, which may include 
Arun> routing protocols, multicast protocols, and MPLS protocols.  

<-- Section 3.3.2.3 (Control Plane)

> Measurement units: sessions

Arun> IMHO, this should be replaced with "number" (as "sessions" in not
Arun> a measuring unit) 

Thanks,
Arun

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Monday, July 17, 2006 9:31 AM
To: [email-masked]
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

A WG Last Call period for the Internet-Drafts on

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking will be open from 17 July 2006
through
28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.tx
t

These versions incorporate comments resulting from implementation.
The authors believe that all comments from the working group have been
addressed, dating back to June 2003 when version 00 of the terminology
draft was published.

These drafts are entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the BMWG Last Call
Review Template in the message of 4/27/04 titled, "Refinements to the
BMWG Last Call Process".
Volunteers should contact [email-masked] A copy of the template is
available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review, please
*read* and express your opinion on whether or not this Internet-Draft
should be given to the Area Directors for consideration as an
Informational RFC.  Send your comments to this list or [email-masked] 

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: "Jay Karthik \(jakarthi\)" <[email-masked]>
Date: Fri, 20 Oct 2006 19:58:13 -0400

> 
> - I am wondering if we need a new term called "Peak Load" or 
> something else to indicate this, which is basically to denote 
> a sample configuration set beyond which the device under test 
> becomes unusable or unresponsive to any command line 
> instruction. If you decide to add this, another usual term 
> would be "duration of unresponsiveness" which should be ideally 0.
> 
> Scott>  Any unresponsiveness will be externally observable by packet
> loss or session loss.  It is intended that the user can 
> determine "Peak Load" by selecting values for the configuration sets.
>

Scott,

I agree with your response to this comment and others as well.

Thanks,
Jay

---

From: "Poretsky, Scott" <[email-masked]>
Date: Fri, 20 Oct 2006 19:50:10 -0400

Hi Jay,

Thanks for the response.  Sorry for the late reply.  Comments are below.

Scott

-----Original Message-----
From: Jay Karthik (jakarthi) [mailto:[email-masked]] 
Sent: Monday, August 28, 2006 9:55 PM
To: Al Morton; [email-masked]; Poretsky, Scott; [email-masked]
Cc: [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hello Authors,

I had the following comments on the terminology ID. I will follow up
with my comments on the methodology draft shortly.

Most of the comments are just nits.

- Introduction : "faster test duration" could be replaced by "shorter
test duration"

Scott> Agree

- Introduction :  "Multiple benchmarks are made for each Benchmark Plane
during each Phase."  I suggest replacing 'made' with 'proposed' or
'suggested'.

Scott> I actually meant to say 'measured' here, not 'made' or
'proposed'.

- Need to fix "These benchmarks White Box benchmarks ..... " in
Introduction.

Scott> Agree

- The white box benchmarks could be a very important for many and I
recommend adding the CPU Utilization and Available Memory benchmarks to
Table 1 for all 3 phases.

Scott> This is in the appendix, as suggested by the working group,
because the BMWG measures black-box benchmarks and it was agreed these
white-box benchmarks would be useful.

- 3.1.2 Configuration Sets. We should mention forwarding rate, besides
feature and scaling limit in the definition of the term.

Scott> Agree

- I am wondering if we need a new term called "Peak Load" or something
else to indicate this, which is basically to denote a sample
configuration set beyond which the device under test becomes unusable or
unresponsive to any command line instruction. If you decide to add this,
another usual term would be "duration of unresponsiveness" which should
be ideally 0.

Scott>  Any unresponsiveness will be externally observable by packet
loss or session loss.  It is intended that the user can determine "Peak
Load" by selecting values for the configuration sets.

Cheers -Jay

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Monday, July 17, 2006 9:31 AM
To: [email-masked]
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking

will be open from 17 July 2006 through 28 August 2006 (6 weeks).

---

From: "Poretsky, Scott" <[email-masked]>
Date: Sun, 22 Oct 2006 20:51:11 -0400

-----Original Message-----
From: Poretsky, Scott 
Sent: Saturday, October 21, 2006 7:43 AM
To: 'Tom Alexander'; [email-masked]
Cc: 'Al Morton'
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Thanks Tom!  Responses for the comments on the Terminology are below.

Scott
####################
------------------------------------------------------------------------

1. Introduction

   being managed.  To accurately benchmark a router for deployment it is

   necessary to test that router in operational conditions by 
   simultaneously configuring and scaling network protocols and security

   policies, forwarding traffic, and managing the device.  It is helpful

   to accelerate these network operational conditions so that the 
   router under test can be benchmarked with faster test duration.  
   Testing a router in accelerated network conditions is known as 
   Accelerated Stress Benchmarking.  

[Tom A: The above paragraph is essential to understanding the rationale
for the rest of the draft, but is somewhat confusing. Suggest rewriting
above paragraph as:
"Routers in an operational network are configured with multiple
protocols
and security policies while simultaneously forwarding traffic and 
being managed.  To accurately benchmark a router prior to deployment, it
is necessary to test that router under operational conditions by 
configuring and scaling network protocols and security policies, and
simultaneously forwarding traffic as well as managing the device.  

Scott> Added the 3 words changes above

[Tom A:
configuration and management represent relatively infrequent activities
during actual operation 

Scott> This statement is incorrect.

[Tom A:
(compared to traffic forwarding), it is useful
to artificially overstress (accelerate) these network operational
conditions so that the router under test can be benchmarked with lower
test durations.  

Scott> Partially agree.  Now using the sentence "It is useful to
accelerate these network operational conditions so that the router under
test can be benchmarked with a shorter test duration."

------------------------------------------------------------------------

   This document provides the Terminology for performing Stress 
   Benchmarking of networking devices.  The three phases of the Stress 
   Test: Startup, Instability and Recovery are defined along with the
   benchmark and configuration terms associated with the each phase.
   Benchmarks for stress testing are defined using the Aggregate 
   Forwarding Rate and control plane Session Count during each phase 
   of the test.  For each plane, the Configuration Set, Startup 
   Conditions, and Instability Conditions are defined.  Also defined are

   the Benchmark Planes fundamental to stress testing configuration, 
   setup and measurement.  These are the Control Plane, Data Plane, 
   Management Plane and Security Plane  Multiple benchmarks are made 
   for each Benchmark Plane during each Phase.  Benchmarks can be 
   compared across multiple planes for the same DUT or at the same 
   plane for 2 or more DUTS.  These benchmarks White Box benchmarks 
   are provided in Appendix 1 for additional DUT behavior 
   measurements.  The terminology is to be used with the companion 
   methodology document [4].  The sequence of phases, actions, and 
   benchmarks are shown in Table 1.

[Tom A: Suggest rewording above sentence beginning with "These
benchmarks
White Box benchmarks ..." as follows:
"Benchmarks of internal DUT characteristics such as memory and CPU
utilization (also known as White Box benchmarks) are described in
Appendix 1, to allow additional characterization of DUT behavior."]

Scott> Agree
------------------------------------------------------------------------

   Table 1. Phase Sequence and Benchmarks 
   III. Recovery Phase   II. Instability Phase    I. Startup Phase
   <-----------------<---<-------------------<----<--------------<
    Remove Instability   Achieve Configuration    Apply Startup
    Conditions           Set                      Conditions

    Benchmark:           Benchmark:               Benchmark:
    Recovered Aggregate  Unstable Aggregate       Stable Aggregate 
    Forwarding Rate      Forwarding Rate          Forwarding Rate

                         Degraded Aggregate 
                         Forwarding Rate          

                         Average Degraded 
                         Forwarding Rate

    Recovered Latency    Unstable Latency         Startup Latency

    Recovered Uncontrolled Recovered Uncontrolled Stable Session Count 
    Sessions Lost          Sessions Lost

    Recovery Time

[Tom A: Is there some reason why the above table reads from right to
left? It would be more logical for it to read from left to right.]

Scott> Yes.  The BMWG has discussed this on previous occasions.  Test 
Equipment displays time on a scale that scrolls right to left.  We 
want to match that.
 
------------------------------------------------------------------------

3. Term definitions
   3.1 General Terms
   3.1.1 Benchmark Planes

      Definition:  
      The features, conditions, and behavior for the Accelerated Stress 
      Benchmarking.

      Discussion:
      There are four Benchmark Planes: Control Plane, Data Plane, 
      Management Plane, and Security Plane as shown in Figure 1. The
      Benchmark Planes define the Configuration, Startup Conditions, 
      Instability Conditions, and Failure Conditions used for the test.

[Tom A: The last sentence does not make sense. Benchmark planes cannot
define the conditions; instead, conditions are defined for each plane.
Suggest rewording as:
"Configuration, Startup Conditions, Instability Conditions, and
Failure Conditions used for each test are defined for each of these
four Benchmark Planes."]

Scott> Agree

------------------------------------------------------------------------

    3.1.2 Configuration Sets

      Definition:  
        The features and scaling limits used during the Accelerated
Stress 
        Benchmarking.

      Discussion:
        There are four Configuration Sets: Control Plane Configuration
Set, 
        Data Plane Configuration Set, Management Plane Configuration
Set, 
        and Security Plane Configuration Set. 

[Tom A: The above discussion does not help me understand what a
Configuration Set is. Please consider adding text to clarify. A
couple of examples of Configuration Sets would be MOST helpful.]

Scott> Examples are in the methodology.  Al requested to have one
mandatory minimum configuration set.  I will add that here to also 
satisfy your request for an example.

------------------------------------------------------------------------

   3.1.6 Controlled Session Loss

     Definition: 
        Control Plane sessions that are intentionally brought
        down during the Stress test.

      Discussion:
        The test equipment is able to control protocol 
        session state with the DUT.

[Tom A: The above discussion specifies some capability of the
test equipment, and does not clarify what Controlled Session
Loss is. Suggest rewording as follows:
"Controlled Session Loss is performed during the test in order
to stress the DUT by forcing it to tear down Control Plane
sessions while handling traffic. It is assumed that the test
equipment is able to control protocol session state with the
DUT and is therefore able to introduce Controlled Session Loss."]

Scott> Agree
------------------------------------------------------------------------
 
   3.2.3 Management Plane

      Definition:
        The Management features and tools used for the 
        Accelerated Stress Benchmarking.

Poretsky and Rao                                               [Page 8]

INTERNET-DRAFT        Terminology for Accelerated          June 2006
                         Stress Benchmarking  

      Discussion:
        A key component of the Accelerated Stress Benchmarking is the 
        Management Plane to assess manageability of the router 
        under stress.  The Management Plane defines the Configuration, 
        Startup Conditions, and Instability Conditions of the 
        management protocols and features.  The Management Plane 
        includes SNMP, Logging/Debug, Statistics Collection, and 
        management configuration sessions such as telnet, SSH, and 
        serial console.  SNMP Gets SHOULD be performed continuously.  
        Management configuration sessions should be open 
        simultaneously and be repeatedly open and closed.  Open 
        management sessions should have valid and invalid 
        configuration and show commands entered.

[Tom A: It is difficult, in the above discussion, to determine where
terminology ends and methodology takes up. It appears that the
discussion is trying to specify methodology together with terminology.
Suggest moving the last 2 sentences (beginning with "SNMP Gets ..."
into the methodology.]

Scott> Agree

------------------------------------------------------------------------

   3.2.4 Security Plane

      Definition:
        The Security features used during the Accelerated Stress 
        Benchmarking.

      Discussion:
        The Security Plane defines the Configuration, Startup 
        Conditions, and Instability Conditions of the security 
        features and protocols.  The Security Plane includes the 
        ACLs, Firewall, Secure Protocols, and User Login.  Tunnels 
        for those such as IPsec should be established and flapped.  
        Policies for Firewalls and ACLs should be repeatedly added 
        and removed from the configuration via telnet, SSH, or 
        serial management sessions.

[Tom A: Same issue as for Management Plane: the above discussion
mixes terminology and methodology. Suggest moving the last 2
sentences (starting with "Tunnels for ...") to the methodology.

Scott> Agree.
------------------------------------------------------------------------

  3.3.5.1 Management Plane Configuration Set

      Definition:
        The router management features enabled for the 
        Accelerated Stress Benchmark.

      Discussion:
        A key component of the Accelerated Stress Benchmark is the 
        Management Configuration Set to assess manageability of the 
        router under stress.  The Management Configuration Set defines 
        the management configuration of the DUT.  Features that are 
        part of the Management Configuration Set include access, SNMP, 
        Logging/Debug, and Statistics Collection, and services such as 
        FTP, as shown in Figure 3. These features should be enabled
        throughout the Stress test. SNMP Gets should be made 
        continuously with multiple FTP and Telnet sessions operating 
        simultaneously.  FTP sessions should be opened and closed at 
        varying intervals and get and put files while open.  Telnet 
        sessions should be opened and closed at varying intervals and 
        enter valid and invalid show and configuration commands while 
        open.

[Tom A: Same issue as before - methodology is mixed in with the
terminology. Suggest moving the last 4 sentences (starting with
"These features should be enabled ...") to the methodology.]

Scott> Agree. 
------------------------------------------------------------------------

Appendix 1.  White Box Benchmarking Terminology
   Minimum Available Memory
      Definition: 
        Minimum DUT Available Memory during the duration of the 
        Accelerated Stress Benchmark.   

      Discussion:
        It is necessary to monitor DUT memory to measure this 
        benchmark.                      

[Tom A: It is nice that one has to monitor DUT memory to measure
this benchmark, but what is the significance of this benchmark?
Please explain. Even a sentence like "This benchmark enables the
assessment of reserve capacity in the DUT, as well as capacity
degradation over repeated trials" would be fine.]

Scott>  Used the sentences, "This benchmark enables the assessment of
resources in the DUT.  It is necessary to monitor DUT memory to measure
this benchmark."
Please note that when benchmarking "Minimum Available Memory" it is
necessary
to monitor DUT memory .  			
------------------------------------------------------------------------

   Maximum CPU Utilization
      Definition: 
        Maximum DUT CPU utilization during the duration of the 
        Accelerated Stress Benchmark.
        
      Discussion:
        It is necessary to monitor DUT CPU Utilization to measure
        this benchmark.         

[Tom A: Same comment as for previous White Box benchmark.]

Scott> Used the sentences, "	This benchmark enables the assessment of

resources in the DUT.  It is necessary to monitor DUT CPU Utilization 
to measure this benchmark."  Please note that when benchmarking "Maximum
CPU Utilization" it is necessary to monitor the DUT CPU.  

-----Original Message-----
From: Tom Alexander [mailto:[email-masked]] 
Sent: Saturday, October 21, 2006 3:47 AM
To: Poretsky, Scott; [email-masked]
Cc: 'Al Morton'
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Scott, Shankar,

I read the accelerated benchmarking drafts and have some comments, in
the
enclosed files. As I'm a relative newbie in BMWG I didn't feel
comfortable
sending them to the whole list (when you're about to put your foot in
your
mouth, keeping the audience small is a good idea!) so I'm sending them
directly to you instead.

Note that I've excerpted pieces of the drafts and added my comments
below,
using the general syntax "[Tom A. -  ..... ]".

Best regards,

- Tom