Subject: RE: Re-revised idea for early cross-area review

From: "Romascanu, Dan \(Dan\)" <[email-masked]>
Date: Wed, 15 Feb 2006 18:59:50 +0200

IMO the sooner the expert review and feedback, the better. However I
agree with Andy that this would multiply the efforts of the reviewers
(although I am not sure about the authors) beyond the scalability
threshold that a group of voluntary reviewers can carry. Taking this
into account review by the time, or even part of the first WGLC seems to
me an acceptable compromise, worth being experimented with. 

Regards,

Dan

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Andy Bierman
> Sent: Wednesday, February 15, 2006 6:28 PM
> To: Brian E Carpenter
> Cc: [email-masked]
> Subject: Re: Re-revised idea for early cross-area review
> 
> Brian E Carpenter wrote:
> 
> I reviewed MIBs at Cisco for 10 years, and my experience is 
> that early review is very expensive.  It takes a lot of extra 
> time and effort for both the authors and the reviewers, 
> because the document under review is not stable yet.
> 
> I suggest that instead of reviewing a document when a WG 
> first accepts an individual submission as a WG item, wait 
> until the first WG Last Call.  That's when the WG Chair(s) 
> think the document is stable.
> 
> 
> Andy
> 
> > Hi,
> >
> > Coming back to an old discussion with a new spin (a dispatcher):
> >
> > As you know,the IESG is concerned to ensure that documents are 
> > reviewed for possible cross-area issues as early as 
> possible, before 
> > they are in final IESG review. This is to avoid "late surprises"
> > where serious cross-area problems are found after a WG has 
> completed 
> > its work on a draft. On the other hand we don't want to add 
> > bureaucracy or steps in the process, so this kind of review 
> needs to 
> > be done in parallel.
> >
> > The overhead of asking for every draft to be reviewed at an early 
> > stage by every Area would be enormous and often pointless. 
> Thus, the 
> > best option appears to be generalist reviews at an early stage.
> > We have some experience now with generalist reviews at a 
> late stage, 
> > through the Gen-ART team that Harald set up, and we know 
> that it can 
> > work reasonably well with a team of reviewers and someone to act as 
> > "dispatcher."
> >
> > The logical moment to solicit early cross-area review is when a WG 
> > formally adopts a draft as a WG item (i.e. the draft is 
> added to the 
> > WG charter page). At that point, the draft is one that is by 
> > definition on its way through the process.
> >
> > We'd like your comments on a way to achieve this.  The 
> thought is not 
> > to add any new process step or hurdle for the WGs.  It may 
> add a a bit 
> > of additional, but integral, work responding to any reviews 
> that are 
> > sent.  In outline (with details to be decided),
> >
> > - a new list, perhaps [email-masked], will be 
> created. Anybody
> >   can join it, but joining it is understood to indicate willingness
> >   to carry out early reviews.
> >
> > - the list membership is made visible
> >
> > - whenever a WG formally adopts a draft as a WG item (i.e. the draft
> >   is added to the WG charter page), the secretariat will generate a
> >   message to the early-reviews list, saying something like
> >
> >     The FOOBAR WG has adopted draft-ietf-foobar-XXX-00.txt 
> as a WG draft
> >     and solicits cross-area review. Comments should follow 
> the guidelines
> >     at <URL> and should be sent to [email-masked] within four weeks
> >
> > - NEW: we appoint a volunteer dispatcher, who will assign drafts to 
> > reviewers
> >   on a round-robin basis, to ensure that each draft gets at 
> least one 
> > review
> >
> > - reviewers will send their generalist reviews to the 
> early-reviews list
> >   for archiving, and to the WG concerned
> >
> > - WGs are expected to treat these generalist reviews seriously, 
> > especially
> >   when they indicate serious cross-area problems or gaps. But their 
> > formal
> >   place in the IETF process is the same as any other comments
> >
> > If you think the general idea is good, we'll solicit your 
> inputs for 
> > the early review guidelines, on the foundations of the work done by 
> > the former ICAR WG.
> >
> > (It was also commented that we need a tool to support reviews and 
> > comment tracking - agreed, but it's a separate issue.)
> >
> >      Brian
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> 
> 
>

---

From: Allison Mankin <[email-masked]>
Date: Wed, 15 Feb 2006 12:57:52 -0800

This is a different sort of review from a WGLC review.

Rather than considering the details of the stabilized
document after the working group gets through with it,
an early review tries to look at a bigger picture:

- Is the approach technically strong in a big picture way?
- Do the reviewers see issues the WG may have missed
  about the choice of the protocol to extend, or design of the
  protocol?
- How does the protocol look in terms of its cross-cutting
  design points including:
    - security and privacy
    - congestion awareness/avoidance
    - internationalization
    - use of addresses, dns

There are some wonderful questions for an early review in an
IAB document, RFC 3426, Section 3, which is just called Questions.

Early review would get an extra measure of cross-cutting quality
into the work.   Great WG Chairs and editors do this, but outside 
reviewers would add more of it.

Allison

> that early review is very expensive.  It takes a lot of extra time and 
> effort
> for both the authors and the reviewers, because the document
> under review is not stable yet.
> 
> I suggest that instead of reviewing a document when a WG first
> accepts an individual submission as a WG item, wait until the
> first WG Last Call.  That's when the WG Chair(s) think the
> document is stable.
> 
> 
> Andy