Subject: RE: Input solicited: IETF problem pinpointing at the plenary?

From: [email-masked]
Date: Wed, 30 Oct 2002 10:04:18 +0200

Hi Harald,

I think this is a good idea.  I am willing to participate.  I have some
concrete ideas on how to improve the work, without beating up the IESG 
(too much).  My main suggestions fall under the following topics.

	1) enabling better mangement practices, better tool support.
	2) enabling better use of IETF resources, perhaps looking 
		at a more formal role of directorates in helping
		process the work.
	3) enabling better transparencies - reviewers should be
		identified, so follow-up questions can be asked.
	4) better guidelines of all sorts for document writers, so
		that something can be pointed at to say why the IESG
		has taken a particular stand (for example on extensibility).

I'd be happy to write up the above issues in more detail.

John

> -----Original Message-----
> From: ext Harald Tveit Alvestrand [mailto:[email-masked]]
> Sent: 30 October, 2002 04:24
> To: [email-masked]
> Subject: Input solicited: IETF problem pinpointing at the plenary?
> 
> 
> Hello, folks!
> 
> The present IESG plenary agenda shows an one-line item:
> 
>  "The IETF organization - status, challenges and future directions" - 
> Harald Alvestrand
> 
> What I will be trying to do with this presentation is to 
> review how the 
> IETF is working - what is working well, what is working less 
> well, and what 
> are problems that we really have to do something about.
> 
> Don't take me wrong - I think the work we are doing is important, we 
> actually get quite a lot of it done, and what we produce is generally 
> useful (sometimes VERY useful) to the Internet community.
> 
> But many people have told me that they percieve problems in 
> the current 
> process, problems with how we choose what to work on and how 
> we choose to 
> work on those things; problems with how we resolve conflicts, 
> and problems 
> with how we do the management and quality control of the 
> specifications we 
> produce.
> 
> One such input is draft-huston-ietf-pact-00; there have been 
> other, less 
> formal ones.
> 
> When chatting with Marshall Rose about this draft, we came up 
> with the idea 
> of trying to embed some of the input from the community within the 
> presentation - turning it somewhat from a "presentation by 
> me" into a kind 
> of community panel.
> 
> If we're going to do that, we need a few (3-5?) volunteers 
> who can take a 
> stab at drafting 2-3 slides each identifying what they see as 
> problems and 
> possible ideas for solutions, huddle together for a while in 
> Atlanta to get 
> the slides to match up, and speak the words at the plenary.
> 
> What do you think of the idea? Who wants to work with me on this?
> 
>                     Harald
> 
>

---

From: [email-masked]
Date: Wed, 30 Oct 2002 09:42:30 -0600

> Quick pithy answer: too many documents and not enough 
> security-guru bandwidth.

The security-guru bandwidth is definitely a concern that we
in the Mobile IP WG can very well relate to. It is extremely
frustrating to be unable to get attention from security savvy
folks until very late in the process.

-Basavaraj

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Wed, 30 Oct 2002 12:21:38 -0700

The lack of transparency is a huge problem for the organization.  The
draft-huston-ietf-pact-00 document sets out some good directions, not
the least of which is more careful tracking of documents and decisions
(or lack thereof).  It is important for an organization to gather
enough information to be able to diagnose, and with luck, heal itself.

  > What's happening in the working groups reflects what's
  > happening in the IESG, too.  IETF process is *not*
  > transparent - increasingly less so, in fact.  Is that
  > causing problems in the day-to-day work of the organization?
  > Maybe - that's something that needs examination, as well.

Hilarie

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 30 Oct 2002 14:35:06 -0500

In message <[email-masked]>, "The Purple Strea
>enough information to be able to diagnose, and with luck, heal itself.

Yes.  We hope that the new draft-tracker tool will be a big help to the 
entire community.  It will let people see the status of things; it will 
also show all the interactions and state changes for each document.

		--Steve Bellovin, http://www.research.att.com/~smb (me)
		http://www.wilyhacker.com ("Firewalls" book)

---

From: Eastlake III Donald-LDE008 <[email-masked]>
Date: Wed, 30 Oct 2002 14:50:31 -0500

I don't have a problem will collecting more statistics but you have to be very careful about what they mean and what you try to optimize. I would think length of time for a draft would be more important than number of revisions since revisions are so highly dependent on editor and WG working style. If a small number of revisions gets to be viewed as an indication of quality, you can be sure that lots of WGs will do work outside the ID process, to the detriment of the community, and suddenly an amazing number of the drafts submitted to the IESG will be -01 or -02 or whatever else the "perfect" value is perceived to be.

On higher attendance, I don't see lurkers as much of a problem. What is more of a problem is that, as the number of participants in an acitivy grows, the size of any particular small fraction increases. If some position is held by 1%, for a small IETF, that's probably 1 person. For a big IETF/WG, it's at least a handful. And this turns out to make a substantial difference. Psychological experiements show that minorities of 1 are fairly easy to dissuade while 2 or more who reinforce each other are much much more steadfast inntheir opinions. As the IETF and its most active working groups get larger, you have more self reinforcing small minorities with increasingly extreme positions.

Donald

-----Original Message-----
From: Kurt D. Zeilenga [mailto:[email-masked]]
Sent: Wednesday, October 30, 2002 11:53 AM
To: Bernard Aboba
Cc: Frank Kastenholz; Harald Tveit Alvestrand; [email-masked]
Subject: Re: Input solicited: IETF problem pinpointing at the plenary?

At 06:57 AM 2002-10-30, Bernard Aboba wrote:
>Ultimately, you've got to have some metrics that describe the performance
>of the process -- and some understanding of what is limiting that
>performance. Otherwise, we're just tinkering without much of a scientific
>method behind it.

I would be interested in knowing the average number of revisions
an I-D undergoes from the time it is recommended to the AD to the
time it enters the RFC-editor queue.   If the IETF is performing
well, this number should be quite small.  A large number is
indicative of document quality problems.

It's my general opinion that many of the "process problems"
are symptoms of document quality problems.  Basically, poor
quality documents tend to act as DoS attacks... sucking away
process/review bandwidth from quality documents.

I think we not only need to look at ways to improve document
quality (thus reducing the process/review bandwidth required)
but also process mechanisms to ensure process/review time is
fairly allocated to each document in the queue.

Kurt

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 30 Oct 2002 17:47:11 -0500

In message <[email-masked]>, Randy Bush writes:
>
>> One process resolution that might address this sort of situation is for
>> DISCUSSes to be made public, so that the pool of people out there with
>> relevant information and the cycles to deal with it can assist in
>> responding to them. I understand something like this is "in the works",
>> but I'm not well-informed as to its state vector.
>
>before atlanta

Right.  We wanted it available earlier, but the code just isn't quite
ready.  I don't think I have to explain to people on this list that
software development can take time, and it does no one any good to to 
turn everyone loose on buggy code and dirty data.

---

From: Eastlake III Donald-LDE008 <[email-masked]>
Date: Thu, 31 Oct 2002 10:46:16 -0500

Hi,

While there are other problems, one problem I see is excessive queuing delay inside the network. The natural rhythm of the IETF is more or less set by the frequency of IETF meetings at about 1/3 of a year. Usually, a delay of a month or two is percieved as OK. When delays exceed 3 or 4 months, things are perceived as slow. When delays exceed a year, other than rare exceptions for which there is a good reason for extraordinary delay which has been communicated back to the submitter, they are perceived as totally bogged down. There are presently a number of Informational WG submissions to the IESG that have been queued for over a year. There are also a number of Informational drafts submitted by the RFC Editor to the IESG that have been queued for over a year. It seems to me that one of the reasons the Internet works is that it doesn't queue much inside the network. I understand that IESG members are volunteers and have only so many hours, being in somewhat the same situation, a!
!
nd that the document that are the most important and urgent get handled reasonably quickly, but excessive queue delay necessarily causes problems. When a document has been queued for over a year with no feedback, the submitter/WG, besides feeling neglected, is/are afraid to revise or withdraw it, as long as it has any value, feeling they might almost be at the front of the queue and will fall back and have to wait another year+ if they touch it. Thus the queue is clogged with drafts some of which are increasingly out of date and/or decreasingly relevant. And when it is finally reviewed, the reviewer knows that any complaints or suggestions sent back to the submitter will be received by someone who hasn't heard anything for a long time, and is thus likely to think, if not say, "Why the hell wasn't I told of this a year ago?" and "If I do fix this, does that mean another year+ of delay until the revised version is looked at?" Furthermore, I would hazard a guess that the situatio!
!
n must be depressing for some ADs who see this pile of old work looming over them.

I'm not sure what the solution is but some time ago I made a somewhat similar complaint to the RFC Editor, IAB, and IESG concerning the length of time approved documents, particularly standards track documents, were spending in the RFC Editor queue. While I do not know if my messge was the cause, a little while later the aged table of how long approved drafts have been in the RFC Editor queue appeared at IETF Plenaries and the situation was pretty quickly cleared up. There is a saying: "Whatever is measured improves." Possibly just tracking I-Ds better and published a table of how many I-Ds have been queued in the IESG for how long would work the same magic. And maybe that was about to happen anyway.

Thanks,
Donald
========================================================
Donald E. Eastlake 3rd, Motorola,
MS: M2-450, 20 Cabot Boulevard, Mansfield, MA 02048 USA
[email-masked], 1-508-851-8280

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 31 Oct 2002 15:01:21 -0500

In message <[email-masked]>, Avri Doria writes:
>
>  I find it 
>baffling, e.g., that the Security folks would allow something to go 
>through IETF Last call without a thorough analysis.

Unfortunately, thorough security analyses are *hard*, and we don't have 
enough people who are interested and qualified.  Right now, some of us 
are involved in a long discussion about whether or not there are flaws 
in some proposed new security protocols, and if so, what should be done 
about them.  (My perception of the current status is that the flaw only 
appears under certain operational circumstances.  We next need to agree
on how likely the circumstances are.  After that, we need to agree on a 
fix -- and there may be a flaw in the fixes that have been proposed...)
In other words, even in our own area it isn't obvious that we've gotten 
it right.

>  I suggest that the 
>IETF Last Call is the place where deficiencies, especially those that 
>are not in the specific talent base of the WG members, should be 
>discovered.  I believe that the ADs could perhaps make use of their 
>directorates to ensure this happens. Whenever a draft in is in IETF Last 
>Call, each interested AD should make sure that one or more  of her or 
>his directorate is assigned to doing a detailed analysis of the document 
>and that this review go to the WG chair and editors.  Speaking of IETF 
>Last Call comments, what happens to them anyway?   Perhaps the documents 
>fro my group that went through IETF Last Call never got any comments, 
>but I have never seen any.

The ADs use them in their evaluations.  In particular, the sponsoring 
AD will often work with the authors to make any necessary fixes, even 
before the document goes to the IESG as a whole.
>
>Which brings up another question, and perhaps a metric that could be 
>collected (yes, a mechanism for collecting it would be needed - perhaps 
>an on line form that folks fill in after doing an IETF LC review).  How 
>many people, other then ADs, actually review documents in IETF LC.  I 
>admit that to date i have only read those i am personally interested in, 
>and have rarely done anything that could be called a proper review.  Am 
>i alone in this behavior?  What use is the IETF Last Call if no one 
>really does a solid review?  I believe there should a through review 
>during IETF LC before the IESG considers a document, and that a WG 
>should address all LC comments before the IESG deals with a document. 
>I.e. bring RFCs half way to being refereed.

I agree that a thorough review is desirable.  It's hard, and as you 
note, most people only read documents that they're interested in.
>
> And i think better use 
>might be made of the directorates. Instead of just being a bunch of 
>sages who advise the ADs on weighty matters, perhaps they can actually 
>be volunteered for work, such as reviews.

That is, in fact, the way the security directorate works.  More to the 
point, we're trying to get security people involved in the early parts 
of a design -- it's often the fundamental early decisions that cause 
the difficult security problems in the final version.  My judgment is 
that early involvment, rather than late review, is a better allocation
of scarce resources.  But sure, I'd love to have both.

---

From: [email-masked]
Date: Fri, 1 Nov 2002 00:37:40 +0200

Steve,

> Unfortunately, thorough security analyses are *hard*, and we 
> don't have enough people who are interested and qualified.  

In NSIS, Hannes Tschofenig [[email-masked]] seems
like a candidate for you Security Guru Apprentiship program.  What
is the course fee?

More seriously, part of the problem (IMO) is that IETF (IESG/IAB)
should use the comminity better (that includes directorates, especially
in a more formal, documented manner).  If there is good, talented
people out there, we should lead away from the dark side and to the 
light.

We should support community involvement in the processes, otherwise
what is the point of the exercise?

John