Subject: Re-revised idea for early cross-area review

From: Brian E Carpenter <[email-masked]>
Date: Wed, 15 Feb 2006 16:16:32 +0100

Hi,

Coming back to an old discussion with a new spin (a dispatcher):

As you know,the IESG is concerned to ensure that documents are
reviewed for possible cross-area issues as early as possible, before
they are in final IESG review. This is to avoid "late surprises"
where serious cross-area problems are found after a WG has
completed its work on a draft. On the other hand we don't want
to add bureaucracy or steps in the process, so this kind of review
needs to be done in parallel.

The overhead of asking for every draft to be reviewed at an early
stage by every Area would be enormous and often pointless. Thus,
the best option appears to be generalist reviews at an early stage.
We have some experience now with generalist reviews at a late stage,
through the Gen-ART team that Harald set up, and we know that
it can work reasonably well with a team of reviewers and someone
to act as "dispatcher."

The logical moment to solicit early cross-area review
is when a WG formally adopts a draft as a WG item (i.e. the draft
is added to the WG charter page). At that point, the draft is
one that is by definition on its way through the process.

We'd like your comments on a way to achieve this.  The thought
is not to add any new process step or hurdle for the WGs.  It may add a
a bit of additional, but integral, work responding to any reviews
that are sent.  In outline (with details to be decided),

- a new list, perhaps [email-masked], will be created. Anybody
   can join it, but joining it is understood to indicate willingness
   to carry out early reviews.

- the list membership is made visible

- whenever a WG formally adopts a draft as a WG item (i.e. the draft
   is added to the WG charter page), the secretariat will generate a
   message to the early-reviews list, saying something like

     The FOOBAR WG has adopted draft-ietf-foobar-XXX-00.txt as a WG draft
     and solicits cross-area review. Comments should follow the guidelines
     at <URL> and should be sent to [email-masked] within four weeks

- NEW: we appoint a volunteer dispatcher, who will assign drafts to reviewers
   on a round-robin basis, to ensure that each draft gets at least one review

- reviewers will send their generalist reviews to the early-reviews list
   for archiving, and to the WG concerned

- WGs are expected to treat these generalist reviews seriously, especially
   when they indicate serious cross-area problems or gaps. But their formal
   place in the IETF process is the same as any other comments

If you think the general idea is good, we'll solicit your inputs for
the early review guidelines, on the foundations of the work done by the
former ICAR WG.

(It was also commented that we need a tool to support reviews and
comment tracking - agreed, but it's a separate issue.)

      Brian

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Feb 2006 08:28:20 -0800

Brian E Carpenter wrote:

I reviewed MIBs at Cisco for 10 years, and my experience is
that early review is very expensive.  It takes a lot of extra time and 
effort
for both the authors and the reviewers, because the document
under review is not stable yet.

I suggest that instead of reviewing a document when a WG first
accepts an individual submission as a WG item, wait until the
first WG Last Call.  That's when the WG Chair(s) think the
document is stable.

Andy

---

From: Jari Arkko <[email-masked]>
Date: Wed, 15 Feb 2006 19:17:42 +0200

I've done a few early reviews, and do not think they are that
expensive. Maybe even less expensive than full final
reviews.

It is true that the stuff is far from stable at -00. However,
as a reviewer you need to have a different take on
the review than what you normally do in the final
stages. Namely, you should look at the architectural
issues, missing pieces (this security scheme
won't fly, you are ignoring congestion control),
etc.

There's no need to look at the nits. Look at whether
the idea is workable, and what other things the authors
need to worry about when completing their work.

I support Brian's review suggestion.

--Jari

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 15 Feb 2006 12:51:35 -0500

> under review is not stable yet.

There are multiple levels of reviews that can be done on a document.

A final review during (first) last call makes sense.  It catches technical
details and walks through the specification with a (hopefully) fresh
set of eyes.

A level of "review" that I think we could all benefit is simply someone
watching the evolution of a document of interest earlier in the process.
The goal would be to prevent issues that may be problematic early
on from becoming (by analogy) wet concrete that is difficult to reshape.

My favorite example of this is something that is deployed and is
highly resistant to change in spite of what comments might follow
from last call review.

-- 
Jeff Haas 
NextHop Technologies

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Feb 2006 11:26:53 -0800

> won't fly, you are ignoring congestion control),

I would rather see the effort go into "Guidelines for Foo" RFCs
that we have been seeing lately.  Give the authors the info
they need to pass the review, without a time-consuming additional
review process.

I've never heard of forced review of a document that the WG
knows is full of problems, rough terminology, little or spotty WG consensus,
and will change significantly before it is ready for publication.
That's just about every draft-00 I've ever seen.  The cure for
the "waiting until IETF Last Call" syndrome shouldn't cost more
to solve than the original problem itself.

If a WG Chair or AD, who is doing a good job of overseeing a WG,
decides a particular draft needs an early review from another area,
then that's different -- that's just people doing their job.

Andy

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 15 Feb 2006 12:31:42 -0800

Hi -

> From: "Brian E Carpenter" <[email-masked]>
> To: <[email-masked]>
> Sent: Wednesday, February 15, 2006 7:16 AM
> Subject: Re-revised idea for early cross-area review
...
> the early review guidelines, on the foundations of the work done by the
...

I like the general idea.  In reading some of the other comments in this
thread, I think we may be tripping over different understandings and
expectations for "reviews" at different points in a document's journey
through the process.  I also think that "big picture" or "generalist"
comments are especially useful early in the cycle, so that considerations
like manageability, internationalization, security, scalability, and so on
are given some thought from the very beginning, not as WG last call
comments.  The very fact that a -00- frequently will not have fully
stable terminology or even structure makes it easier to address
problems like terminology conflicts from the outset, rather than having
to work around "established usage" when WG last call happens.

Randy

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 15 Feb 2006 20:59:22 -0500

>won't fly, you are ignoring congestion control),

Precisely -- that's what early review is all about.  You want to catch 
the big things, the things that if left alone would require massive 
architectural changes at IETF Last Call, and that just won't happen.

As for Brian's original question: this is the most important part:

> NEW: we appoint a volunteer dispatcher, who will assign drafts to reviewers

Without something like this, it won't work.  With this, it has a chance 
*if* the ADs make it clear that architectural comments need to be taken 
seriously.  (This may require an AD review of the document and comments 
at this point.)  

Whenever I pushed back on a document while AD, there was always the 
issue of "late comments".  At times, I had to accept things I really 
didn't like, because I knew they wouldn't be changed.  This, though, is 
early, when there's much less excuse for ignoring serious issues, even 
those that imply the need for a major redesign.

Go for it.

		--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Feb 2006 18:16:46 -0800

>> on a round-robin basis, to ensure that each draft gets at least one review

Q: what happens when the number of review requests exceed the number of 
reviewers,
or a review is taking a really long time?  (How long is a really long 
time in this case?)

Do WGs proceed anyway with draft-01 and beyond?
Or wait until their draft-00 review is finished before proceeding?

Andy

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 15 Feb 2006 21:23:50 -0500

>Or wait until their draft-00 review is finished before proceeding?

The dispatcher finds another reviewer and downgrades the delinquent 
assignee.  Russ and I kept track of which security directorate members 
didn't pull their weight (and I assume that he and Sam still do); we 
found it essential to getting things done.

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Feb 2006 18:31:08 -0800

> found it essential to getting things done.

So the answer is that a WG will wait to proceed until the draft-00 
review is completed,
but the possibility of low reviewer resources is remote or zero.

Andy

---

From: Thierry Ernst <[email-masked]>
Date: Thu, 16 Feb 2006 11:37:59 +0900

I support a compulsory review at the 1st WG Last Call, and earlier
reviews when and if appropriate, i.e.  at the request of the a WG chair
(anyone, not necessarily the one chairing the WG where the draft is
published) or an AD (anyone).

IMHO, making the review compulsory at the 1st opus of the draft as a WG doc
would make the process time consumimg for the drafts not in good shape
(but useful for those which already are in good shape). 

If the review is to be compulsory for -00,  drafts not in sufficient good
shape should not be published. Then this could work too.

My 2 cents.
Thierry

On Wed, 15 Feb 2006 12:31:42 -0800

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Wed, 15 Feb 2006 21:39:54 -0500

>but the possibility of low reviewer resources is remote or zero.
That's how I'd run it if it were my job.

---

From: [email-masked]
Date: Thu, 16 Feb 2006 06:40:34 +0200

Brian,

This would be good a good thing.  I think keeping track of reviewers and
their reviews would be a good thing, just in case a WG gets a really
bad early review, they could check the reviewer's previous reviews.

I think that the early reviews should be non-blocking, and care has
to be taken in order to ensure quality.  -00 drafts tend to be 
incomplete and of lesser quality than drafts in IETF LC, so these
need to be considered during the review.

Additionally, the early review should be included in the (eventual)
write-up when the draft is finally submitted to the IESG for
final IETF review.

Question - would this be for all documents, or (as I would suggest)
non-Informational documents.

John

>-----Original Message-----
>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of ext Brian E Carpenter
>Sent: 15 February, 2006 17:17
>To: [email-masked]
>reviewed for possible cross-area issues as early as possible, 
>before they are in final IESG review. This is to avoid "late surprises"
>to add bureaucracy or steps in the process, so this kind of 
>review needs to be done in parallel.
>
>The overhead of asking for every draft to be reviewed at an 
>early stage by every Area would be enormous and often 
>pointless. Thus, the best option appears to be generalist 
>reviews at an early stage.
>We have some experience now with generalist reviews at a late 
>stage, through the Gen-ART team that Harald set up, and we 
>know that it can work reasonably well with a team of reviewers 
>and someone to act as "dispatcher."
>
>The logical moment to solicit early cross-area review is when 
>a WG formally adopts a draft as a WG item (i.e. the draft is 
>added to the WG charter page). At that point, the draft is one 
>that is by definition on its way through the process.
>
>is not to add any new process step or hurdle for the WGs.  It 
>may add a a bit of additional, but integral, work responding 
>to any reviews that are sent.  In outline (with details to be decided),
>
>If you think the general idea is good, we'll solicit your 
>inputs for the early review guidelines, on the foundations of 
>the work done by the former ICAR WG.
>
>(It was also commented that we need a tool to support reviews 
>and comment tracking - agreed, but it's a separate issue.)
>
>      Brian
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 16 Feb 2006 09:37:25 -0500

>non-Informational documents.

To me, it's most important for architecture documents or whatever other 
documents set up basic data flow.  I've seen WGs where the documents I 
disliked most were the detailed protocol specs, but that's rare.  To 
give a current example, if a document came up that used MD5 instead of 
SHA-256, it's an easy fix.  But it's much more of a problem if there's 
no data flow that allows negotiation of the hash function, since there 
will almost certainly be a new one in 4-5 years.  Adding a new data 
flow at or after Last Call is painful and will meet with a lot of 
resistance.