Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

From: "Poretsky, Scott" <[email-masked]>
Date: Mon, 13 Nov 2006 14:29:04 -0500

Thanks Jim,

Based upon this point we will retain use of the word 'protocol' and we
can focus on other matters to progress this work item.

Scott

-----Original Message-----
From: McQuaid, Jim [mailto:[email-masked]] 
Sent: Tuesday, November 07, 2006 11:49 AM
To: [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

I think that the distinction between "devices,"  "protocols," and
"services" becomes impossible and / or metaphysical at some point.  IP
routing and switching are behaviors of network elements that may the
service of a network possible.  Perhaps the real issue is the layer at
which a service is provided.

cheers,

Jim McQuaid

NetQoS

-----Original Message-----
From: Arun Gandhi [mailto:[email-masked]] 
Sent: Tuesday, November 07, 2006 11:22 AM
To: 'Poretsky, Scott'; 'Al Morton'; [email-masked]; [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hi Scott,

>> The BMWG makes it a point to not benchmark 'services', so I recommend
>> we stick with use of the word 'protocols' and so not replace it with
>> 'services'. 

> I respectfully disagree as based on this particular work item
(document's
> motivation), I'm still of the opinion that we are not characterizing
> different protocols together but in fact different services with
protocols
> per se overlaid on a SUT. Inevitably (and as nicely explained), a
router
> in an operational network is configured with multiple protocols
(ospf,bgp,
> mpls, multicast, etc.) and services (such as vpns, vpls, policies,
etc.)
> based on the role it is playing in that network. Also, there are other
> features/services such as graceful-restart (GRS), non-stop routing,
ISSU,
> and so on that may be turned on as well.

Also, here's what I just found on the BMWG charter (section 'Description
of
Working Group')...

"The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services 
                                                ^^^^^^^^^^^^^^^^^^^ 
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results."

Regards,
Arun Gandhi.

-----Original Message-----
From: Arun Gandhi [mailto:[email-masked]] 
Sent: Tuesday, November 07, 2006 10:20 AM
To: 'Poretsky, Scott'; 'Al Morton'; [email-masked]; [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hi Scott,

> The terms 'black-box' and 'white-box' have been frequently used in the
> BMWG over the past 10 years. I would be happy to use this work item as
> the place to define them. The focus of this work is black-box 
> benchmarking, consistent with the BMWG charter. With this particular 
> work item the working group agreed there would be value in observing
> White-Box benchmarks so it was preferred to include that in an
> appendix.

I totally agree.

> In addition, I do not agree with your suggestion to use "number" as a
> unit. Specifically you recommended to replace unit "sessions" with
unit
> "number".  When answering the question "How many sessions?" could you
> answer 10 numbers or 10 sessions.  No change will be made.

A little correction... i was actually thinking "number of sessions".
Sorry
for the confusion. Is this agreeable?

> The BMWG makes it a point to not benchmark 'services', so I recommend
> we stick with use of the word 'protocols' and so not replace it with
> 'services'. 

I respectfully disagree as based on this particular work item
(document's
motivation), I'm still of the opinion that we are not characterizing
different protocols together but in fact different services with
protocols
per se overlaid on a SUT. Inevitably (and as nicely explained), a router
in
an operational network is configured with multiple protocols (ospf, bgp,
mpls, multicast, etc.) and services (such as vpns, vpls, policies, etc.)
based on the role it is playing in that network. Also, there are other
features/services such as graceful-restart (GRS), non-stop routing,
ISSU,
and so on that may be turned on as well.

Also, IMHO, under Section 3.3 we should add "Number of ifls/ifds" and
"Number of policies" in anyone of the four buckets (where it seems most
fit).

Lastly, just a thought/question...Under Section 3.3.3, is it possible to
actually explicitly outline a threshold based on the system/SUT
resources in
use after which a box can be deemed as "Unstable" (e.g. a router in an
operational network using 85% of the system resources will/should be
considered unstable)? 

Thanks,
Arun Gandhi.

-----Original Message-----
From: Poretsky, Scott [mailto:[email-masked]]
Sent: Friday, October 20, 2006 12:46 PM
To: Arun Gandhi; Al Morton; [email-masked]; [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts

Hi Arun,

I apologize that this response is so late.  

The BMWG makes it a point to not benchmark 'services', so I recommend we
stick with use of the word 'protocols' and so not replace it with
'services'.  The terms 'black-box' and 'white-box' have been frequently
used in the BMWG over the past 10 years.  I would be happy to use this
work item as the place to define them.  The focus of this work is
black-box benchmarking, consistent with the BMWG charter.  With this
particular work item the working group agreed there would be value in
observing White-Box benchmarks so it was preferred to include that in an
appendix.  At this time there is no plan to make further changes on
these topics unless others in the working group want to continue the
discussion.

In addition, I do not agree with your suggestion to use "number" as a
unit.  Specifically you recommended to replace unit "sessions" with unit
"number".  When answering the question "How many sessions?" could you
answer 10 numbers or 10 sessions.  No change will be made.

Also, there are 3 sentence changes you requested below for which I do
not see the value of those changes.  Maybe you could provide more
information.

Thanks for your thorough review of the documents.

Scott

-----Original Message-----
From: Arun Gandhi [mailto:[email-masked]] 
Sent: Tuesday, August 29, 2006 11:55 AM
To: 'Al Morton'; [email-masked]; Poretsky, Scott; [email-masked]
Subject: RE: [bmwg] WG Last Call on Accelerated Stress Benchmarking
Drafts
Importance: High

Hello Authors,

I am little late in responding wrt the open dates but I suppose my
feedback will get evaluated. I will also follow up with the feedback on
methodology document shortly.

<-- Section 1 (Introduction): 

In general I believe this draft should not just address multiple
Protocols operational simultaneously but "different services" 
being overlaid in a fully operational box.

> Routers in an operational network are simultaneously configured with
> multiple protocols and security policies while forwarding traffic and
> being managed.

Arun> IMHO, replace "protocols" with "services" as in ISP world (which
Arun> is the motivation for this draft) there may be different service
Arun> sets such as l3vpn, l2vpns, VPLS, etc. along with routing and 
Arun> signaling protocols configured.

> It is helpful to accelerate these network operational conditions so 
> that the router under test can be benchmarked with faster test 
> duration.

Arun> This needs revision/clarification.

> Multiple benchmarks are made for each Benchmark Plane during each 
> Phase.

Arun> Benchmarks are defined for each plane to be characterized in all 
Arun> phases.

> These benchmarks White Box benchmarks are provided in Appendix 1 for
> additional DUT behavior measurements.

Arun> Two points:
Arun> 1) This line needs revision.
Arun> 2) A new term "White" box is added without being explained. Also, 
Arun> the impression I get reading this draft is that it's a black box
Arun> test. Also, since the motivation is the benchmarking of a router
Arun> in ISP domain then for sure it is a black box testing. Regardless
Arun> we should monitor/characterize other system parameters such as 
Arun> "memory usage", "CPU utilization", "install/bring up times", and 
Arun> so on.

<-- Section 3.2.1 (Control Plane)

> The Control Plane defines the Configuration, Startup Conditions, and
> Instability Conditions of the control protocols. Control Plane 
> protocols may include routing protocols, multicast protocols, and 
> MPLS protocols.     

Arun> IMHO, this statement should be revised to be "The Control Plane 
Arun> defines the Configuration, Startup Conditions, and Instability
Arun> Conditions of the control plane protocols, which may include 
Arun> routing protocols, multicast protocols, and MPLS protocols.  

<-- Section 3.3.2.3 (Control Plane)

> Measurement units: sessions

Arun> IMHO, this should be replaced with "number" (as "sessions" in not
Arun> a measuring unit) 

Thanks,
Arun

-----Original Message-----
From: Al Morton [mailto:[email-masked]] 
Sent: Monday, July 17, 2006 9:31 AM
To: [email-masked]
Subject: [bmwg] WG Last Call on Accelerated Stress Benchmarking Drafts

BMWG:

A WG Last Call period for the Internet-Drafts on

Terminology for Accelerated Stress Benchmarking Methodology Guidelines
for Accelerated Stress Benchmarking will be open from 17 July 2006
through 28 August 2006 (6 weeks).

URLs for these Internet-Drafts are:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-term-09.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-acc-bench-meth-05.tx
t

These versions incorporate comments resulting from implementation.
The authors believe that all comments from the working group have been
addressed, dating back to June 2003 when version 00 of the terminology
draft was published.

These drafts are entering the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
At this time, we are seeking volunteers to complete the BMWG Last Call
Review Template in the message of 4/27/04 titled, "Refinements to the
BMWG Last Call Process".
Volunteers should contact [email-masked] A copy of the template is
available here:
http://home.comcast.net/~acmacm/IDcheck/LastCallTemplate.txt

Whether or not you decide to volunteer for the directed review, please
*read* and express your opinion on whether or not this Internet-Draft
should be given to the Area Directors for consideration as an
Informational RFC.  Send your comments to this list or [email-masked] 

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg