Subject: [bmwg] Revised Charter

From: Al Morton <[email-masked]>
Date: Mon, 28 Apr 2008 14:01:26 -0400

BMWG,

As discussed at our last meeting, we plan to revise our charter
(minimally) to add some new work items and update all milestones.

Please reply with any comments by May 12, 2008.
Note that some author teams need to suggest dates for
expired milestones (see below).

thanks and regards,
Al
bmwg chair

-------------------------------------------------------------
Benchmarking Methodology Working Group (BMWG)
...
Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services
that are built from these technologies.

Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
for the common, unambiguous reporting of benchmarking results.

To better distinguish the BMWG from other measurement initiatives in
the IETF, the scope of the BMWG is limited to technology
characterization using simulated stimuli in a laboratory environment.
Said differently, the BMWG does not attempt to produce benchmarks for
live, operational networks. Moreover, the benchmarks produced by this WG
shall strive to be vendor independent or otherwise have universal
applicability to a given technology class.

Because the demands of a particular technology may vary from
deployment to deployment, a specific non-goal of the Working Group is
to define acceptance criteria or performance requirements.

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the
operation inter-networking technologies.

In addition to its current work plan, the BMWG is explicitly tasked
to develop benchmarks and methodologies for the following technologies:

  * MPLS Forwarding:  Develop specific methods to characterize the latency
    and forwarding performance of MPLS devices, extending the fundamental
    recommendations of RFC 1242 and RFC 2544 to this networking technology.

  * SIP Networking Devices: Develop new terminology and methods to
    characterize the key performance aspects of network devices using SIP,
    including the signaling plane scale and service rates while considering
    load conditions on both the signaling and media planes. This work
    will be harmonized with related SIP performance metric definitions
    prepared by the PMOL working group.

Goals and Milestones:
...
< lots of completed milestones, then >
...
Done	  	IPv6 Benchmarking Methodology to AD Review

REVISED
June 2008              IPsec Device Benchmarking Terminology to IESG Review
June 2008              IPsec Device Benchmarking Methodology to IESG Review

NEW
April 2009              Methodology for MPLS Forwarding to AD Review.
June 2009               Terminology for SIP Device Benchmarking to IESG Review.
June 2009               Methodology for SIP Device Benchmarking to IESG Review.

TO BE REVISED YET
Dec 2007                Net Traffic Control Benchmarking Methodology 
to AD Review.
Dec 2007                Router Accelerated Test Terminology to AD Review
Dec 2007                Router Accelerated Test Methodology to AD Review

Dec 2007                Router Accelerated Test Method for EBGP to AD Review
Dec 2007                Router Accelerated Test Method for 
Operational Security to AD Review
Feb 2008                Terminology For Protection Benchmarking to AD Review
Feb 2008                Methodology For Protection Benchmarking to AD Review
Jul 2009                Basic BGP Convergence Benchmarking 
Methodology to AD Review.

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Mon, 28 Apr 2008 16:21:35 -0400

Hi Al,

Thanks for sharing this good news with us.  With specific to the
following extension -

>     recommendations of RFC 1242 and RFC 2544 to this 

It seems fair and desirable to also include MPLS applications such as
TE, VPN (both layer3 and layer2) etc. in the above text, though it may
be implicit to a lot of us.

> April 2009              Methodology for MPLS Forwarding to AD Review.

The timeline seems appropriate, though this specific work item could be
ready well before that. We will submit the draft-ietf- version soon for
the WG review. 

Thanks.

Cheers,
Rajiv 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Al Morton
> Sent: Monday, April 28, 2008 2:01 PM
> To: [email-masked]
> Subject: [bmwg] Revised Charter
> 
> BMWG,
> 
> EBGP to AD Review

---

From: Al Morton <[email-masked]>
Date: Mon, 28 Apr 2008 16:28:43 -0400

At 04:21 PM 4/28/2008, Rajiv Asati (rajiva) wrote:
>We will submit the draft-ietf- version soon for the WG review.

The revised charter has to be approved first...

>be implicit to a lot of us.

Can you suggest the wording of the work item?
It's not clear that this can (or should) be a
work item for "all things MPLS".

Al
bmwg chair

---

From: "Tom Alexander" <[email-masked]>
Date: Tue, 29 Apr 2008 11:34:43 -0700

Al and BMWG,

I would also like to see something included in the charter that permits IP
devices incorporating wireless links to be covered by the scope of BMWG.
There seems to be some controversy around the concept, and this would be a
good opportunity to resolve it.

Something as simple as one sentence added at the end of the first paragraph
would suffice, for example:

"BMWG may make recommendations applicable to both wireless and wired
internetworking technologies."

The wireless world is rapidly converging to an all-IP environment; for
example, all the upcoming 4G cellular protocols (LTE, UMB and WiMAX) are
based on a pure IP infrastructure, with no TDM or circuit-switching present
at all. (802.11 Wi-Fi got there long ago.)

Broadband wireless is quickly becoming THE ubiquitous access technology for
LANs and MANs. IETF-defined protocols are at the heart of broadband
wireless. Thus it is appropriate for BMWG to define recommendations for
measuring their performance.

Thanks,

- Tom A.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Al
Morton
Sent: Monday, April 28, 2008 11:01 AM
To: [email-masked]
Subject: [bmwg] Revised Charter

BMWG,

thanks and regards,
Al
bmwg chair

Goals and Milestones:
...

REVISED

NEW
April 2009              Methodology for MPLS Forwarding to AD Review.
June 2009               Terminology for SIP Device Benchmarking to IESG
Review.
June 2009               Methodology for SIP Device Benchmarking to IESG
Review.

TO BE REVISED YET

---

From: Al Morton <[email-masked]>
Date: Tue, 29 Apr 2008 15:16:25 -0400

>good opportunity to resolve it.

Tom,

Although there might be some controversy in the future,
I don't think the wording of our charter precludes the
work you've described (or the drafts you've prepared).

In case it's not clear, let me describe the process we've
begun to follow.  When we take-up new work in BMWG, we'll have
a minor re-chartering exercise (like this) to explicitly
add the new work items, and seek IESG approval.

So, where the process of adding work items that achieved sufficient
BMWG support *also* included the consensus of our ADs in the past,
now it will include the consensus of the IESG.
This will raise awareness of new work we take on,
and help us get the cross-area review and other support we need.

When we satisfy all the sufficient criteria to take-up
the WLAN work you've proposed, then we get to repeat this
minor re-chartering exercise and the explicit text you're
seeking will be a part of that proposal.

hope this helps,
Al

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 29 Apr 2008 14:29:29 -0500

> expired milestones (see below).
[...]
> June 2009  Methodology for SIP Device Benchmarking to IESG Review.

Al: I have not been able to get a hold of my co-authors -- they
are busy with the IEEE CQR conference, I think.  So, at the risk
of speaking on their behalf, the deadline for June 2009 for this
work sounds eminently reasonable.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
2701 Lucent Lane, Rm. 9F-546, Lisle, Illinois 60532 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
WWW:   http://www.alcatel-lucent.com/bell-labs
_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 29 Apr 2008 22:30:16 +0200

Tom,

I do not believe that speaking about  'wireless and wired
internetworking technologies' in such a way is useful. It is not clear
what this means. Nothing in the charter prevents BMWG to deal with layer
2 technologies of various kinds - wired and wireless. However not all
metrics for all protocols need to be defined differently on different
LAN or access layers, some do, other are agnostics. I prefer to deal
with the cases of wired and wireless technologies from case to case, and
to be explicit when we need to be. Do you believe that something needs
to be said on this respect for the MPLS Forwarding or SIP Network
Devices items? 

Dan

> Behalf Of Tom Alexander
> Sent: Tuesday, April 29, 2008 9:35 PM
> To: [email-masked]; 'Al Morton'
> Subject: Re: [bmwg] Revised Charter
> 
> Al and BMWG,
> 
> I would also like to see something included in the charter 
> that permits IP devices incorporating wireless links to be 
> covered by the scope of BMWG.
> There seems to be some controversy around the concept, and 
> this would be a good opportunity to resolve it.
> 
> Something as simple as one sentence added at the end of the 
> first paragraph would suffice, for example:
> 
> "BMWG may make recommendations applicable to both wireless 
> and wired internetworking technologies."
> 
> The wireless world is rapidly converging to an all-IP 
> environment; for example, all the upcoming 4G cellular 
> protocols (LTE, UMB and WiMAX) are based on a pure IP 
> infrastructure, with no TDM or circuit-switching present at 
> all. (802.11 Wi-Fi got there long ago.)
> 
> Broadband wireless is quickly becoming THE ubiquitous access 
> technology for LANs and MANs. IETF-defined protocols are at 
> the heart of broadband wireless. Thus it is appropriate for 
> BMWG to define recommendations for measuring their performance.
> 
> Thanks,
> 
> - Tom A.
> 
> 
> Note that some author teams need to suggest dates for expired 
> milestones (see below).
> 
> thanks and regards,
> Al
> bmwg chair
> 
> Benchmarking Methodology Working Group (BMWG) ...
> The major goal of the Benchmarking Methodology Working Group 
> is to make a series of recommendations concerning the 
> measurement of the performance characteristics of various 
> internetworking technologies; further, these recommendations 
> may focus on the systems or services that are built from 
> these technologies.
> 
> Each recommendation will describe the class of equipment, 
> system, or service being addressed; discuss the performance 
> characteristics that are pertinent to that class; clearly 
> identify a set of metrics that aid in the description of 
> those characteristics; specify the methodologies required to 
> collect said metrics; and lastly, present the requirements 
> To better distinguish the BMWG from other measurement 
> initiatives in the IETF, the scope of the BMWG is limited to 
> technology characterization using simulated stimuli in a 
> laboratory environment.
> Said differently, the BMWG does not attempt to produce 
> benchmarks for live, operational networks. Moreover, the 
> benchmarks produced by this WG shall strive to be vendor 
> independent or otherwise have universal applicability to a 
> given technology class.
> 
> deployment to deployment, a specific non-goal of the Working 
> Group is to define acceptance criteria or performance requirements.
> 
> An ongoing task is to provide a forum for discussion 
> regarding the advancement of measurements designed to provide 
> insight on the operation inter-networking technologies.
> 
> In addition to its current work plan, the BMWG is explicitly 
> tasked to develop benchmarks and methodologies for the 
> following technologies:
>

---

From: "Tom Alexander" <[email-masked]>
Date: Tue, 29 Apr 2008 19:23:29 -0700

Dan,

Thanks for the clarification. I have no issue with taking wired and wireless
benchmarking on a case-by-case basis. The reason I suggested this was
because of a discussion at a previous IETF as to whether the BMWG charter
had to be modified to permit benchmarking of IP equipment used in wireless
networks. Since the charter is being revised, I figured this would be a good
time to start a discussion on the subject.

The other thing that threw me off was that the charter revision explictly
included MPLS, so I figured that perhaps we were moving to a "must be
explicitly chartered" phase. Al has set me straight on this in his previous
reflector post, so that's all cleared up now.

If there is no need to be explicit about wired vs. wireless, then clearly
the BMWG charter implicitly includes wireless internetworking technologies,
and there should be no need to modify the charter in that respect.

By the way, I'm curious: what criteria need to be satisfied in order to take
up benchmarking of wireless LAN switches as a work item?

Thanks,

- Tom

-----Original Message-----
From: Romascanu, Dan (Dan) [mailto:[email-masked]] 
Sent: Tuesday, April 29, 2008 1:30 PM
To: Tom Alexander; [email-masked]; Al Morton
Subject: RE: [bmwg] Revised Charter

Tom,

Dan

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 30 Apr 2008 10:04:27 +0200

Hi Tom,

See in-line. 

Dan

> From: Tom Alexander [mailto:[email-masked]] 
> Sent: Wednesday, April 30, 2008 5:23 AM
> To: Romascanu, Dan (Dan); [email-masked]; 'Al Morton'
> Thanks for the clarification. I have no issue with taking 
> wired and wireless benchmarking on a case-by-case basis. The 
> reason I suggested this was because of a discussion at a 
> previous IETF as to whether the BMWG charter had to be 
> modified to permit benchmarking of IP equipment used in 
> wireless networks. Since the charter is being revised, I 
> figured this would be a good time to start a discussion on 
> the subject.
> 
> The other thing that threw me off was that the charter 
> revision explictly included MPLS, so I figured that perhaps 
> we were moving to a "must be explicitly chartered" phase. Al 
> has set me straight on this in his previous reflector post, 
> so that's all cleared up now.

I am a fan of 'explicit chartering'. This is exactly why I prefer to be
explicit and mention 'wired and wireless' whenever there is a need to
define something specific for one or another rather than making a
generic statement. 

> 
> If there is no need to be explicit about wired vs. wireless, 
> then clearly the BMWG charter implicitly includes wireless 
> internetworking technologies, and there should be no need to 
> modify the charter in that respect.

Yes, I believe that this is the case. 

> 
> By the way, I'm curious: what criteria need to be satisfied 
> in order to take up benchmarking of wireless LAN switches as 
> a work item?

Same as for any other work item - interest and consensus that it's
needed, committed editors and reviewers, no prior standards or parallel
work in the IETF or other SDOs on the same subject (though we may reuse
or refer to pieces of other work who fit in the scope)

---

From: Al Morton <[email-masked]>
Date: Wed, 30 Apr 2008 08:54:34 -0400

At 04:04 AM 4/30/2008, Romascanu, Dan (Dan) wrote:
>Tom wrote:

And, space available on the charter created by completing existing work.
This has always been an issue in my experience, with consequences
both good and bad.

Al

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 30 Apr 2008 14:57:42 +0200

Of course. Thanks for reminding this important aspect, Al. 

Dan

> Sent: Wednesday, April 30, 2008 3:55 PM
> To: Romascanu, Dan (Dan); Tom Alexander; [email-masked]
> > > By the way, I'm curious: what criteria need to be 
> satisfied in order 
> > > to take up benchmarking of wireless LAN switches as a work item?
> >
> This has always been an issue in my experience, with 
> consequences both good and bad.
> 
> Al
> 
>