Subject: [bmwg] an ID for BMWG to consider

From: [email-masked] (scott bradner)
Date: Tue, 26 Oct 2004 13:42:02 -0400 (EDT)

Re: Benchmarking Methodology for Wireless LAN Devices 
(draft-alexander-wlan-meth-00.txt)

Tom Alexander and I put this ID about testing WLAN devices - 
is this something that BMWG might be interested in working on?

Scott

_______________________________________________
bmwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/bmwg

---

From: Al Morton <[email-masked]>
Date: Thu, 28 Oct 2004 12:16:31 -0400

>is this something that BMWG might be interested in working on?

BMWG,

As I started reading through Tom and Scott's draft,
I remembered that one of the earliest BMWG meetings I attended
began with controversy.  It was IETF-48 in Pittsburgh, Kevin couldn't
make it and Scott stood in as chair.  The agenda included an
item to discuss a proposal on wireless network benchmarking.

The meeting report says:

>The bmwg met Monday morning 7/31/00. Scott Bradner acted as chair due to 
>the normal chair's being unable to attend.
>
>The first item on the agenda was a discussion on a proposal to have the 
>bmwg develop documents about testing of performance in a wireless 
>environment. The decision of the AD and the ad-hoc chair was that at this 
>time it did not look like the proposed work fits within the bmwg area so 
>the offer of a presentation was declined.
>...

Since this is part of BMWG's history, we should deal with it head-on.
There are several different circumstances today that encourage
us to revisit this past decision in new light, including:

*  BMWG has a new Charter, dating from early 2003.

*  There is a detailed draft describing the work currently proposed
(I understand that only slides were available IETF-48.  I never saw
the slides, and I'm not suggesting a comparison of the proposals.)

So, Kevin and I would like to encourage BMWG folks to read the draft
http://www.ietf.org/internet-drafts/draft-alexander-wlan-meth-00.txt
and comment on the list.

thanks for your attention,
Al

---

From: David Newman <[email-masked]>
Date: Thu, 28 Oct 2004 09:19:40 -0700 (PDT)

> is this something that BMWG might be interested in working on?

I support the development of this work within the bmwg.

Rationales:

a) 802.11 is very different than 802.3, with many implications for IP 
device benchmarking

b) some existing bmwg metrics such as throughput are not meaningful in the 
context of benchmarking wireless devices (though this ID uses the term 
"throughput")

c) this ID defines new tests for measuring WLAN-specifc events such as 
roaming, ESS delay and delay variation, and rate adaptation

d) without (c), "it's just wireless" is a common and unacceptable refrain 
when it comes to 802.11 testing

dn

---

From: Kevin Dubray <[email-masked]>
Date: Fri, 29 Oct 2004 09:24:51 -0400

> a) 802.11 is very different than 802.3, with many implications for IP 

The implication on IP devices is an important component to consider.
Anyone care to list those implications for the less than
802.11 savvy amongst us?  (It appears the proposed document hopes
that RFC1242 et al. makes that binding.)

---

From: David Newman <[email-masked]>
Date: Fri, 29 Oct 2004 09:59:59 -0700 (PDT)

> that RFC1242 et al. makes that binding.)

Here are just some quick examples off the top of my head. I'm sure there 
are others:

1. Throughput -- The air is a very lossy medium, and thus zero-loss 
measurements such as throughput aren't meaningful. We should not abuse the 
1242 definition by attempting to label as "thoughput" measurements that 
involve loss.

2. Delay, jitter, and other time-related metrics -- The 802.11 protocols 
acknowledge every single frame at L2 and also send significant amounts of 
control traffic.

While it's possible (though uncommon) to measure delay of individual 
packets or instantaneous jitter from two packets, those probably won't be 
meaningful indicators of how long a DUT/SUT holds up application traffic. 
There are probably more meaningful, WLAN-specific goalposts to be defined.

3. 802.11-specific stuff -- There are some WLAN events, such as roaming 
time, that existing bmwg methodologies do not describe.

dn