Subject: [bmwg] WG status

From: Al Morton <[email-masked]>
Date: Fri, 01 Oct 2010 10:05:30 -0400

BMWG,

When we parted after our session in July, we had the
following action items pending, as captured in the meeting
minutes:
http://www.ietf.org/proceedings/78/minutes/bmwg.html

I've added my view of current status below each item.

ACTION Items:
------------------

Start another WGLC on the RESET draft when revised to address the 
post-submission-deadline discussions on the list and at the meeting.
	draft-ietf-bmwg-reset-01  	 2010-07-09    	Active
	Need revised draft.

Revised versions of IPsec drafts needed (addressing IESG review).
	Need revised drafts.

Revised versions of IGP-Dataplane drafts needed (addressing IESG review).
	Need revised drafts.

WG comments on key issues (see the detailed minutes).
	List has been rather quiet.

Complete Re-chartering Discussions in the WG.
	Proposed Charter will be discussed at next IESG Telechat (Oct 7).

I've appended the proposed charter text below,
Al
bmwg chair

===========================================================================
From: IESG Secretary <[email-masked]>
To: [email-masked], [email-masked], [email-masked]
Subject: Internal WG Review: Recharter of Benchmarking Methodology (bmwg)

A new charter for the Benchmarking Methodology (bmwg) working group in the
Operations and Management Area of the IETF is being considered.  The draft
charter is provided below for your review and comment.

Review time is one week.

The IETF Secretariat

Benchmarking Methodology (bmwg)
---------------------------------------------------
Current Status: Active Working Group
r3 Last Updated 2010-09-29

Chair:
   Al Morton <[email-masked]>

Operations and Management Area Directors:
   Ronald Bonica <[email-masked]>
   Dan Romascanu <[email-masked]>

Operations and Management Area Advisor:
   Ronald Bonica <[email-masked]>

Mailing Lists:
   Address:      [email-masked]
   To Subscribe: [email-masked]
   Archive:      http://www.ietf.org/mail-archive/web/bmwg/

Description of Working Group:

The Benchmarking Methodology Working Group (BMWG) will continue to
produce a series of recommendations concerning the key performance
characteristics of internetworking technologies, or benchmarks for
network devices, systems, and services. Taking a view of networking
divided into planes, the scope of work includes benchmarks for the
management, control, and forwarding planes.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

The set of relevant benchmarks will be developed with input from the
community of users (e.g, network operators and testing organizations)
and from those affected by the benchmarks when they are published
(networking and test equipment manufacturers). When possible, the
benchmarks and other terminology will be developed jointly with
organizations that are willing to share their expertise. Joint review
requirements for a specific work area will be included in the detailed
description of the task, as listed below.

To better distinguish the BMWG from other measurement initiatives in the
IETF, the scope of the BMWG is limited to the characterization of
implementations of various internetworking technologies
using controlled stimuli in a laboratory environment. Said differently,
the BMWG does not attempt to produce benchmarks for live, operational
networks. Moreover, the benchmarks produced by this WG shall strive to
be vendor independent or otherwise have universal applicability to a
given technology class.

Because the demands of a particular technology may vary from deployment
to deployment, a specific non-goal of the Working Group is to define
acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the operation
inter-networking technologies.

The BMWG will communicate with the operations community through
organizations such as NANOG, RIPE, and APRICOT.

In addition to its current work plan, the BMWG is explicitly tasked to
develop benchmarks and methodologies for the following technologies:

(continuing, but new wording)
* BGP Control-plane Convergence Methodology (Terminology is complete):
With relevant performance characteristics identified, BMWG will prepare
a Benchmarking Methodology Document with review from the Routing Area
(e.g., the IDR working group and/or the RTG-DIR). The Benchmarking
Methodology will be Last-Called in all the groups that previously
provided input, including another round of network operator input during
the last call.

(continuing)
* SIP Networking Devices: Develop new terminology and methods to
characterize the key performance aspects of network devices using SIP,
including the signaling plane scale and service rates while
considering load conditions on both the signaling and media planes. This
work will be harmonized with related SIP performance metric definitions
prepared by the PMOL working group.

(new)
* Flow Export and Collection: Develop terminology and methods to
characterize network devices flow monitoring, export, and collection.
The goal is a methodology to assess the maximum IP flow rate that a
network device can sustain without losing any IP flow information or
compromising the accuracy of information exported on the IP flows,
and to asses the forwarding plane performance (if the forwarding
function is present) in the presence of  Flow Monitoring.

(new)
* Data Center Bridging Devices:
Some key concepts from BMWG's past work are not meaningful when testing
switches that implement new IEEE specifications in the area of data
center bridging. For example, throughput as defined in RFC 1242 cannot
be measured when testing devices that implement three new IEEE
specifications: priority-based flow control (802.1Qbb); priority groups
(802.1Qaz); and congestion notification (802.1Qau).
Since devices that implement these new congestion-management
specifications should never drop frames, and since the metric of
throughput distinguishes between non-zero and zero drop rates, no
throughput measurement is possible using the existing methodology.
This work will update RFC 2455 and exchange periodic Liaisons with
IEEE 802.1 DCB Task Group, especially at WG Last Call.

(new)
* Content Aware Devices:
New classes of network devices that operate above the IP layer of the
network stack require a new methodology to perform adequate
benchmarking.  Existing BMWG RFCs (RFC2647 and RFC3511) provides useful
measurement and performance statistics, though they may not reflect the
actual performance of the device when deployed in production networks.
Operating within the limitations of the charter, namely blackbox
characterization in laboratory environments, the BMWG will develop a
methodology that more closely relates the performance of these devices
to performance in an operational setting. In order to confirm or
identify key performance characteristics,  BMWG will solicit input from
operations groups such as NANOG, RIP and APRICOT.

(new)
* LDP Dataplane Convergence:
In order to identify key LDP convergence performance characteristics,
BMWG will solicit input from operations groups such as NANOG, RIP and
APRICOT. When relevant performance characteristics have been identified,
BMWG will jointly prepare a Benchmarking Terminology Document with the
Routing Area (e.g., the MPLS working group and or the RTG-DIR), which
would define metrics relevant to LDP convergence. The Benchmark
definition document would be Last-Called in all the working groups that
produced it, and solicit operator input during the last call. The work
will then continue in BMWG to define the test methodology, with input
and review from the aforementioned parties.

Goals and Milestones:

Updated Milestones

Done      Terminology For Protection Benchmarking to AD Review
Sep 2010  Networking Device Reset Benchmark (Updates RFC 2544) to IESG
           Review
Dec 2010  Methodology For Protection Benchmarking to IESG Review
Jun 2011  Terminology for SIP Device Benchmarking to IESG Review
Jun 2011  Methodology for SIP Device Benchmarking to IESG Review
Jul 2010  Basic BGP Convergence Benchmarking Methodology to IESG Review.

New Milestones

Feb 2011  Methodology for Flow Export and Collection Benchmarking to
           IESG Review
Jun 2011  Methodology for Data Center Bridging Benchmarking to IESG
           Review
Dec 2011  Terminology for Content Aware Device Benchmarking to IESG
           Review
Dec 2011  Methodology for Content Aware Device Benchmarking to IESG
           Review
Dec 2011  Terminology for LDP Convergence Benchmarking to IESG Review
Dec 2011  Methodology for LDP Convergence Benchmarking to IESG Review

REMOVED Milestones

Dec 2008  Net Traffic Control Benchmarking Methodology to AD Review.
Dec 2008  Router Accelerated Test Terminology to IESG Review
Dec 2008  Router Accelerated Test Methodology to IESG Review

---

From: Kevin Dubray <[email-masked]>
Date: Fri, 1 Oct 2010 10:58:30 -0400

Al,

I like the re-tooled charter.  It preserves the historical intent of developing "universal measuring tapes" for internetworking devices while providing specific clarity regarding areas of responsibility as well collaboration with other teams/communities.

Perhaps, a minor area to discuss is:

"An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the operation
inter-networking technologies."

I think it might be beneficial to a) reconcile the above to the comment in the minutes regarding technology vs technology implementations, and b) augment the notion of "operations" with capabilities.  Many operators are interested in describing an area of "assured behavior in a nominal environment"; many those operator's engineering teams are looking for describing how implementations provide maximal benefit in some particular dimension or set of dimensions while assessing [future?] behavior when capability is exceeded.   The notion is that there is benefit in addition to "real-life" testing that is well suited to the clinical evaluation in which the BMWG operates.  If you buy-in to the above, a slightly modified wording might take on the appearance of:

"An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the capability and operations
of inter-networking technology implementations."

Not too painful. :-)

-Kevin