Subject: RE: Input solicited: IETF problem pinpointing at the plenary?

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 30 Oct 2002 13:00:51 -0800

At 11:50 AM 2002-10-30, Eastlake III Donald-LDE008 wrote:
>I don't have a problem will collecting more statistics but you have to be very careful about what they mean and what you try to optimize. I would think length of time for a draft would be more important than number of revisions since revisions are so highly dependent on editor and WG working style.  If a small number of revisions gets to be viewed as an indication of quality, you can be sure that lots of WGs will do work outside the ID process, to the detriment of the community, and suddenly an amazing number of the drafts submitted to the IESG will be -01 or -02 or whatever else the "perfect" value is perceived to be.

I agree that the number of revisions made BEFORE submission to
the AD are not a particularly interesting statistic.  Note that
I, however, suggested the number of revisions made AFTER submission
to the AD would be interesting.

A high average number of revisions made AFTER submission would be
indicative of a document quality problem.  We should be submitting
documents to our ADs which don't require significant revision.

Kurt

>-----Original Message-----
>From: Kurt D. Zeilenga [mailto:[email-masked]]
>Sent: Wednesday, October 30, 2002 11:53 AM
>To: Bernard Aboba
>Cc: Frank Kastenholz; Harald Tveit Alvestrand; [email-masked]
>Subject: Re: Input solicited: IETF problem pinpointing at the plenary?
>
>At 06:57 AM 2002-10-30, Bernard Aboba wrote:
>>Ultimately, you've got to have some metrics that describe the performance
>>of the process -- and some understanding of what is limiting that
>>performance. Otherwise, we're just tinkering without much of a scientific
>>method behind it.
>
>I would be interested in knowing the average number of revisions
>an I-D undergoes from the time it is recommended to the AD to the
>time it enters the RFC-editor queue.   If the IETF is performing
>well, this number should be quite small.  A large number is
>indicative of document quality problems.
>
>It's my general opinion that many of the "process problems"
>are symptoms of document quality problems.  Basically, poor
>quality documents tend to act as DoS attacks... sucking away
>process/review bandwidth from quality documents.
>
>I think we not only need to look at ways to improve document
>quality (thus reducing the process/review bandwidth required)
>but also process mechanisms to ensure process/review time is
>fairly allocated to each document in the queue.
>
>Kurt

---

From: "David R. Oran" <[email-masked]>
Date: Wed, 30 Oct 2002 17:24:25 -0500

> At 11:50 AM 2002-10-30, Eastlake III Donald-LDE008 wrote:
[snip]
> documents to our ADs which don't require significant revision.
Not necessarily. It could be authors or WG chairs doing a trial-and error 
probe of the search space of acceptable language that might get through the 
IESG. Given the alleged problem of having a one-bit feedback channel from 
the IESG I believe I have actually seen this behavior in action.

Dave.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 30 Oct 2002 15:03:45 -0800

At 02:24 PM 2002-10-30, David R. Oran wrote:
>--On Wednesday, October 30, 2002 1:00 PM -0800 "Kurt D. Zeilenga" <[email-masked]> wrote:
>Not necessarily. It could be authors or WG chairs doing a trial-and error probe of the search space of acceptable language that might get through the IESG. Given the alleged problem of having a one-bit feedback channel from the IESG I believe I have actually seen this behavior in action.

I agree that statistic could be indicative of other problems.  As with
all statistics, we do need to be careful as the conclusions drawn from
them.  As they say, one can prove anything with statistics...

Anyways, I do think it would be interesting to statistic to explore.  In
particular, it would be interesting to interview key parties involved
in the progression of the most (and least) revised after submission to
AD I-Ds to see if there are any common types of problems (or lack of
problems).

Kurt