Subject: Revised idea for early cross-area review

From: Brian E Carpenter <[email-masked]>
Date: Thu, 03 Nov 2005 15:32:09 +0100

Hi,

I'm afraid this has been pending for several months. Here's
a revised proposal following the list discussion in May.

The IESG is concerned to ensure that documents are reviewed for
possible cross-area issues as early as possible, before they
are in final IESG review. This is to avoid "late surprises"
where serious cross-area problems are found after a WG has
completed its work on a draft. On the other hand we don't want
to add bureaucracy or steps in the process, so this kind of review
needs to be done in parallel.

Ideally, such reviews should be solicited from specific reviewers.
But that creates extra work to find such reviewers when they're
needed. The logical moment to solicit early cross-area review
is when a WG formally adopts a draft as a WG item (i.e. the draft
is added to the WG charter page). At that point, the draft is
one that is by definition on its way through the process.

We'd like your comments on a way to achieve this.  The thought
is not to add any new process step or hurdle for the WGs.  It may add a
a bit of additional, but integral, work responding to any reviews
that are sent.  In outline (with details to be decided),

- a new list, perhaps [email-masked], will be created. Anybody
   can join it, but joining it is understood to indicate willingness
   to carry out early reviews.

- the list membership is made visible, at least to all WG Chairs

- whenever a WG formally adopts a draft as a WG item (i.e. the draft
   is added to the WG charter page), the secretariat will generate a
   message to the early-reviews list, saying something like

     The FOOBAR WG has adopted draft-ietf-foobar-XXX-00.txt as a WG draft
     and solicits cross-area review. Comments should follow the guidelines
     at <URL> and should be sent to [email-masked] within four weeks

- the WG chair(s) will also specifically solicit reviews from one or two
   members of the early-reviews list, at their discretion. They will specify
   particular areas of concern if appropriate. (It's worth noting the obvious:
   the absence of early reviews doesn't imply that there are no cross-area
   issues.)

- if you think the general idea is good, we'll solicit your inputs for
   the early review guidelines, on the foundations of the work done by the
   former ICAR WG.

(It was also commented that we need a tool to support reviews and
comment tracking - agreed, but it's a separate issue.)

      Brian

---

From: Russ White <[email-masked]>
Date: Thu, 03 Nov 2005 09:37:56 -0500

>   to carry out early reviews.

Another way to do this would be to depend on the directorates to do some 
of the work. I know routing has one, and probably other areas, as well.

:-)

Russ

- -- 
[email-masked] CCIE <>< Grace Alone

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 03 Nov 2005 11:27:27 -0500

>   the absence of early reviews doesn't imply that there are no cross-area

This provision concerns me a great deal -- there's no load-leveling or 
load-limiting.  When I'm asked to review documents as part of my role 
in the security directorate, I know that the ADs are spreading the load.
Here, any individual WG chair is unaware of what his or her colleagues 
are doing.

		--Steven M. Bellovin, http://www.cs.columbia.edu/~smb

---

From: Melinda Shore <[email-masked]>
Date: Thu, 03 Nov 2005 11:48:37 -0500

> Here, any individual WG chair is unaware of what his or her colleagues

Reviewers can always say "no," or better still, "no, but talk to <x>,"
and do their own load management.  It seems to me that the reason
early review experiments have failed so far is that there's been a lack
of volunteers for the reviewer pool (and I've been remiss about this,
myself), so the risk is that there won't be sufficient resources to
review some number of documents.  I hope that in the worst case those
documents won't turn out to be any worse than they are now, so we'd
still be ahead, I think.

Melinda

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 03 Nov 2005 11:56:16 -0500

>documents won't turn out to be any worse than they are now, so we'd

Maybe -- but I think going through the area directorates is a better 
approach.

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 3 Nov 2005 12:10:58 -0500

Brian,

On Thu, Nov 03, 2005 at 03:32:09PM +0100, Brian E Carpenter wrote:
> completed its work on a draft.

IMO, we could do a little better at helping interested observers
see that new documents may affect their working groups.  For example,
work in l2vpn and l3vpn occasionally affect or use BGP in interesting
ways.  Being able to see that the work in question affects BGP from
day one would let interested observers in IDR that don't follow either
of those WGs to track the document.

A keyword directory of some sort might be one tool by which this could
be done.  Examples would be the name of the WG and individual protocols.
Such keywords could show up in the ASCII announcements and could be
queryable in the online databases.

-- 
Jeff Haas 
NextHop Technologies

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 03 Nov 2005 20:18:14 +0100

> Here, any individual WG chair is unaware of what his or her colleagues 

Right. For Gen-ART, we have a human dispatcher who performs
loose load balancing. Do you think we will need a human (or robotic)
dispatcher for this?

    Brian

---

From: "David B Harrington" <[email-masked]>
Date: Thu, 3 Nov 2005 14:47:05 -0500

Hi,

Going through the area directorates has the advantage that the AD can
see who is not carrying their part of the load and remove them from
the directorate if desired.

Going through the area directorates has the advantage that the AD can
prioritize documents based on factors the reviewers may not be aware
of, such as document dependencies, SDO liaisons, etc.

dbh

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Steven M. Bellovin
> Sent: Thursday, November 03, 2005 11:56 AM
> To: Melinda Shore
> Cc: [email-masked]
> Subject: Re: Revised idea for early cross-area review 
>

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 03 Nov 2005 14:51:57 -0500

>loose load balancing. Do you think we will need a human (or robotic)
I suspect it should be the AD who "owns" that directorate.  (Russ and I 
kept track of to whom we assigned which documents -- and how long it 
took to get an answer....)

---

From: Jari Arkko <[email-masked]>
Date: Fri, 04 Nov 2005 00:08:37 +0200

I support the idea of more early reviews and more cross-area
review. Who wouldn't? But I think among the many things that
need improvement, this probably can give us a lot of bang for
the buck...

As people noted, some directorates and other groups are
doing this already.

I tend to agree that a dispatch function is useful, particularly
if there's a human in the loop.

Anyway, regarding tool support and mailing lists, I at least
already have most of what I need to detect new work;
draft-ietf-*-00.txt from IETF announce does the trick. I
don't mean to say that a new list wouldn't be useful, but
just that I don't think lack of information is main the
main cause lack of reviews. For that we'd have to
look at the following factors, imho:

o  Pool size. Need to go beyond the usual suspects
    who are likely already members of some board or
    the other, and probably also busy.

o  Effect of review team existence to the actual
    increase in reviews. Experts in <X> will in any
    case be likely to review <X> work. Question is
    getting people to review new things.

o  Motivation. This is the key. Not sure I have a
    good proposal here, but from personal experience
    I know that there's a MAJOR difference between an
    automated announcement from the secretariat
    and a personal request from the chair or AD.
    Seeing something good come out of the review,
    such as a bug being fixed is great, of course.
    Getting acked in the acknowledgment section
    is nice. Some people may like to be nominated
    as the Senior Inspectors in the Byte Order
    Review Board. We need to use these things
    better than we do today, but still at the end
    of the day, the main motivation is probably
    business related. Your company needs to care
    about the reviewed stuff, otherwise you aren't
    spending a whole lot of time on it.

o  Cross-area expertise. This is tricky, because
    it turns out that review is not easy if you don't
    know both your own field of expertise and
    the field that you are review. For instance,
    security review is very hard because you need
    to understand both the application domain so
    that you know what problems need to be
    prevented and security domain so that you
    know what tricks can be used.

--Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 04 Nov 2005 02:15:21 +0100

Everybody (not just Jari),

Please let's not rerun the discussions we had in ICAR.

IMHO, either we do something simple and cheap, or we will
end up doing nothing at all.

And I'd also point out that directorate reviews are
area-specific. True cross-area review looks at a whole
range of possible interactions. Both are useful, but they
are different.

     Brian

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Thu, 03 Nov 2005 21:13:29 -0500

>range of possible interactions. Both are useful, but they

I'm suggesting that WG chairs ask a different AD, "could you ask your 
directorate to review this?"

---

From: "David B Harrington" <[email-masked]>
Date: Fri, 4 Nov 2005 00:12:26 -0500

As a MIB Doctor, I frequently review MIBs for different areas.

David Harrington
[email-masked] 

> [mailto:[email-masked]] On Behalf Of Brian E Carpenter
> Sent: Thursday, November 03, 2005 8:15 PM
> To: Jari Arkko
> Cc: [email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 4 Nov 2005 09:35:05 +0100

> of, such as document dependencies, SDO liaisons, etc.

I like to add: Going through an area directorate has the advantage
that reviewers get a recognition for their work - they become member
of an area directorate.

/js

-- 
Juergen Schoenwaelder		    International University Bremen
<http://www.eecs.iu-bremen.de/>	    P.O. Box 750 561, 28725 Bremen, Germany

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 05 Nov 2005 18:37:38 +0100

> As a MIB Doctor, I frequently review MIBs for different areas.

Well understood and highly useful. But MIBs are a bit of a special
case.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 05 Nov 2005 18:40:15 +0100

> directorate to review this?"

Well, we're about to have seven technical areas. Are we really going to
ask for 6 separate cross-area reviews? I am really thinking of reviews
by generalists who will look for red flags that might relate to *any*
IETF area.

    Brian

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Sat, 05 Nov 2005 17:48:31 -0500

>by generalists who will look for red flags that might relate to *any*

Sounds like something for the General Area....

Seriously -- your original proposal said nothing of that.  It simply 
said that WG chairs should ask one or two specific reviwers for 
comments.  I, frankly, interpreted that as looking for specific skill 
sets among the reviewers.  If you want a scan for arbitrary cross-area 
problems, I agree that area directorates aren't the best choice.  I 
remain of the opinion that an "executive director" of the review team 
is necessary.

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 06 Nov 2005 01:49:56 +0100

> remain of the opinion that an "executive director" of the review team 

Understood. I need to clarify the text and I doubt if that will happen
this week, somehow.

    Brian