Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?ch

From: Jim McQuaid <[email-masked]>
Date: Tue, 5 Oct 2004 18:16:57 -0000

This thread is tending toward the "polemic."  I suggest we all obtain and
read the current charters for the WGs in question.  A charter can always be
amended, but it's a good starting point.

I think the issue isn't so much "real world networks" versus "lab networks."
The original IPPM / BMWG split might be better thought of in terms of
audience.  IPPM was for network operators & planners.  BMWG was one element
of that, of course, but clearly *was* started with an eye toward providing a
standard to replace manufacturers marketing claims back at the start of the
router wars, so it was element-oriented and somewhat more enterprise useful
compared to network operator useful.

Jim McQuaid

NetIQ

-----Original Message-----
From: Russ White [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 1:04 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [bmwg] Is the BMWG a proper home for this I-D?ch

> Now we have evolved to the request to make a BCP for measuring network 
> convergence.  I have to ask a question first asked last April regarding 
> this work - Why is the IPPM not the best place to discuss this work?

IPPM deals with testing real world networks, correct? While BMWG deals with 
lab networks, correct? If that's so, there's necessarily going to be some 
overlap in testing methodologies between the two.

> In addition, for the benefit of the BMWG could you please clearly explain 
> what is being benchmarked, how it is being benchmarked, and why you think 
> this is appropriate for the BMWG.

Why does testing have only to do with specific benchmarks performable only 
on specific devices by a given manufacturer? Is this working group really 
that narrow minded? Just as each draft we pass has an "applicability" 
section, at least, and an applicaability draft at best, we should also 
endeavor to show how to use testing in a larger sense. For instance, the 
types of things this draft outlines--what are the things a tester really 
needs to think about when developing a test, beyond specific techniques? 
What are the things a user of testing information must look for when 
examining results, beyond: "I put a packet in here, and out it came over 
there?"

There's a lot more to testing than specific tests, as just the existence of 
applicability tests shows. This type of document would be an extension of 
these applicability documents, to provide a set of best practices for all 
testers and users of tests of network systems.

> Please note that at this time I am neither "for" or "against" this 
> becoming a work item.  However, I would like to read answers to these 
> basic questions so I can make an informed personal decision.  We eagerly 
> await your answers to these basic questions.

I hope this WG decides to actually pursue work beyond point tests, and 
start to deal with all the issues in benchmarking network systems. I 
realize the WG is mostly made up of vendors of testing equipment, but there 
are some of us here who are concerned, as well

Russ

__________________________________
[email-masked] CCIE <>< Grace Alone

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Russ White <[email-masked]>
Date: Tue, 5 Oct 2004 15:10:56 -0400 (Eastern Daylight Time)

> read the current charters for the WGs in question.  A charter can always 
> be amended, but it's a good starting point.

So, let's start with this: IPPM is in the transport working group, while 
BMWG is in operations. That seems to imply something different than 
anything said on this thread, thus far. IPPM seems to deal with traffic 
flow measurement, the measurement of transport, while BMWG seems to deal 
with network measurement, in terms of benchmarking. So, from this angle, a 
best practices on routing protocols convergence wouldn't fit within IPPM, 
since it's not a transport issue at all.

> I think the issue isn't so much "real world networks" versus "lab 
> networks." The original IPPM / BMWG split might be better thought of in 
> terms of audience.  IPPM was for network operators & planners.  BMWG was 
> one element of that, of course, but clearly *was* started with an eye 
> toward providing a standard to replace manufacturers marketing claims 
> back at the start of the router wars, so it was element-oriented and 
> somewhat more enterprise useful compared to network operator useful.

Even in terms of this, all the users of the tests you're talking about, the 
people using the products of the BMWG to evaluate vendor's testing, would 
be network operators, correct? So, that split doesn't make much sense to 
me, either, based on the terms listed here. If such a best practices 
document were to serve as guidance to test developers, I don't see where 
the conflict with what you're saying above comes in.

Now, finally, to the charters themselves. IPPM's charter says:

--

The IPPM WG will produce documents that define specific metrics and
procedures for accurately measuring and documenting these metrics.
The metrics are:

   - connectivity
   - one-way delay and loss
   - round-trip delay and loss
   - delay variation
   - loss patterns
   - packet reordering
   - bulk transport capacity
   - link bandwidth capacity

--

I don't see anything in there about routing protocols convergence, which is 
what the draft in question is/would be, if we continue to work on it. 
BTW--to make certain we come to agreement on at least this one point--the 
draft in question is about routing protocols convergence. Is there anyone 
who reads something else into it?

There's a section on measuring routing protocols convergence through 
traffic flow, but that's no different than 
draft-ietf-bmwg-igp-dataplane-conv-meth-03.txt. In fact, I don't see this 
document as being any different than draft-ietf-bmwg-hash-stuffing-00.txt, 
which is already a working group document. If we reject this document on 
those grounds, then the WG should reconsider the acceptance of those two 
documents, as well.

Based on IPPM's charter (quoted above) and area (transport), there's no way 
a routing protocols document would fit there.

Now, let's turn to BMWG's charter:

--

The major goal of the Benchmarking Methodology Working Group is to make
a series of recommendations concerning the measurement of the
performance characteristics of various internetworking technologies;
further, these recommendations may focus on the systems or services
that are built from these technologies.

....

An ongoing task is to provide a forum for discussion regarding the
advancement of measurements designed to provide insight on the
operation internetworking technologies.

--

I assume there's a missing "of" in the last sentence quoted above. Now, 
since this draft does/would apply to in the lab benchmarks, their 
construction and understanding their results, about specifically about an 
internetworking technology (routing protocols), and their performance.... I 
just don't see how this doesn't fit into the BMWG charter as it sits today. 
The last paragraph certainly implies this as well.

I don't read anything in the charter that limits the BMWG to only specific 
point tests, nor do I see anything placing routing protocols within IPPM's 
charter.

Does anyone see anything different in this?

:-)

Russ

---

From: "Robert Holley \(rholley\)" <[email-masked]>
Date: Tue, 5 Oct 2004 15:55:01 -0400

My 2 cents.

--

The major goal of the Benchmarking Methodology Working Group is to make a
series of recommendations concerning the measurement of the performance
characteristics of various internetworking technologies; further, these
recommendations may focus on the systems or services that are built from
these technologies.

....

I think the last clause of the sentence above validates the point for
inclusion pretty directly. Due to the observation that network convergence
is the performance measurement of a system/service which is built on routing
protocols.

Regards
Bob