Subject: New idea for early cross-area review

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 May 2005 09:43:34 +0200

Hi,

The IESG is concerned to ensure that documents are reviewed for
possible cross-area issues as early as possible, before they
are in final IESG review. This is to avoid "late surprises"
where serious cross-area problems are found after a WG has
completed its work on a draft.

We'd like your comments on a new idea on how to achieve this.  The thought
is not to add any new process step or hurdle for the WGs.  It may add a
a bit of additional, but integral, work responding to any reviews
that are sent.  In outline (with details to decided),

- a new list, perhaps [email-masked], will be created. Anybody
   can join it.

- whenever a WG formally adopts a draft as a WG item (i.e. the draft
   is added to the WG charter page), the secretariat will generate a
   message to the adopt-announce list, saying something like

     The FOOBAR WG has adopted draft-ietf-foobar-XXX-00.txt as a WG draft
     and solicits cross-area comments from the community. Comments should
     follow the guidelines at <URL> and should be sent to [email-masked].

- if you think the general idea is good, we'll solicit your inputs for
   the early review guidelines, on the foundations of the work done by the
   former ICAR WG.

      Brian

---

From: Jari Arkko <[email-masked]>
Date: Tue, 10 May 2005 11:08:56 +0300

I think it is a great idea. Early review, around the time that drafts
are adopted as WG drafts would be very useful.

(I wonder if it would be an idea to use the regular ietf-announce
list for this, just to make sure more people will be seeing the
announcements. I know its more traffic on traffic on the list,
but if you are on it now, you must be pretty good with the Delete
button already.)

--Jari

---

From: Kurt Erik Lindqvist <[email-masked]>
Date: Tue, 10 May 2005 11:04:06 +0200

>   the early review guidelines, on the foundations of the work done  

So in principle [email-masked] is a filtered version of IETF- 
Announce? I think that cross-area review is important but I am a bit  
worried that just mailing out the I-D announcement is not going to  
give enough information on which areas are/could be touched upon. But  
I think it's a good start that might be worth trying.

- kurtis -

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 10 May 2005 07:34:07 -0400

It is worth a try.
I am doubtful, since most experience indicates that getting work out of 
people is much more effective if they are asked individually.
But the cost seems low, and it might work.

Yours,
Joel

---

From: Loa Andersson <[email-masked]>
Date: Tue, 10 May 2005 13:57:57 +0200

All,

this is in general a good idea, one concern though.

There is a risk that we are slowly raising the hurdle for getting
documents accepted as wg doc were accepted as a wg doc, this was
to identify that this was something the working took revision
control over and there were a rough consensus that this was an
in charter work item for the working group. If a nec working group
document automaticall go to a cross-area-review, this might add
requirements to the finish of the doc. And thus delay the acceptance
of the doc as a wg doc.

/Loa

-- 
Loa Andersson

Principal Networking Architect
Acreo AB                           phone:  +46 8 632 77 14
Isafjordsgatan 22                  mobile: +46 739 81 21 64
Kista, Sweden                      email:  [email-masked]
                                            [email-masked]

---

From: Jari Arkko <[email-masked]>
Date: Tue, 10 May 2005 15:11:47 +0300

Hi Loa,

I think the proposal was that the WG adopts the draft and
THEN solicits feedback from others. That is, this review would
not be a checkbox item that you have to complete before being
able to adopt a draft.

(But I agree with others who commented that you get more
reviews when you ask people individually.)

--Jari

---

From: James Carlson <[email-masked]>
Date: Tue, 10 May 2005 08:32:33 -0400

> not be a checkbox item that you have to complete before being

Although this sounds like a decent idea for the heady days when a WG
is first starting -- and attracting well-intentioned and useful
proposals that might be lacking in one or two slightly off-topic areas
-- I'm not so sure you'll _necessarily_ want to see the sort of grot
that is flung towards older groups.

In particular, pppext is a museum of bad ideas.  The group is serving
its purpose in reviewing things that would otherwise just be published
as-is via the Informational-track vanity press, but that doesn't mean
they're necessarily worth the time of anyone else.

I wouldn't be opposed to seeing them on some other list for collective
amusement, but I can certainly think of better ways to spend time.

-- 
James Carlson, KISS Network                    <[email-masked]>
Sun Microsystems / 1 Network Drive         71.234W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.497N   Fax +1 781 442 1677

---

From: Loa Andersson <[email-masked]>
Date: Tue, 10 May 2005 14:38:35 +0200

Jari,

I understood the work flow, point was that knowing how
the human mind work it will (un)consciously become a
tick box - especially if it is an automated process.

/Loa

-- 
Loa Andersson

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Tue, 10 May 2005 10:01:03 -0400

In message <6.2.1.2.0.20050510073309.02f771b8@localhost>, "Joel M. Halpern" wri
>people is much more effective if they are asked individually.

That was my reaction, too.

		--Prof. Steven M. Bellovin, http://www.cs.columbia.edu/~smb

---

From: Tim Bray <[email-masked]>
Date: Tue, 10 May 2005 07:17:27 -0700

On May 10, 2005, at 12:43 AM, Brian E Carpenter wrote:

> We'd like your comments on a new idea on how to achieve this.

Here's the relevant W3C experience: in essence, we found that nobody 
would take the time to review something from another WG until it was in 
last call.  The consequence is that for substantial or controversial 
drafts, they typically have two or more "last calls".

As one consequence, it is fairly common for a WG to solicit reviews of 
particular drafts from one or more specific other WGs; it's often the 
case that you can predict which direction surprises might come from.  
You get an action item to the chair to solicit a commitment from the 
chair of the other WG and report back; frequently, nagging is required. 
  -Tim

---

From: Mark Allman <[email-masked]>
Date: Tue, 10 May 2005 10:36:31 -0400

> would take the time to review something from another WG until it was
> in last call.  The consequence is that for substantial or
> controversial drafts, they typically have two or more "last calls".
> 

I think this is generally the right approach and at the right level.  I
share Joel's concerns about another announce list.  I just doubt that
will work.  How about this ...

  + The IESG says unambiguously that early cross-area review is
    important.  (This sounds easy, but in fact we found the lack of
    solid IESG support in ICAR to be problematic.)

  + When a WG adopts a new item the chairs are responsible for drumming
    up reviewers from various other areas of the IETF (i.e., non-WG
    members).  It wouldn't have to be one WG approaching another WG as
    Tim suggests (although, it certainly could be in some cases where
    that is appropriate).  But, it could also be as easy as finding a
    security person to do a quick early pass.  Put the burden on the
    chairs to drum up the reviewers (maybe with a few guidelines).
    However, the ADs might have to help out a bit if the chairs do not
    know folk in some area within the IETF.  But, the ADs involvement
    should be minimal --- i.e., the (nagging) load gets distributed.
    (And, the ADs might just have a stock list of people they can fire
    off ... "ask A, B, C and D".)

  + Maybe the ADs could quickly vet the list of reviewers and look for
    holes.  (Not in the reviews, but in the expertise of the folk doing
    the reviewing.)

  + Early reviewers for a document are acknowledged as such within the
    document.

I think it needs to be lightweight on the IESG.  However, the thing is,
early review isn't free.  It cannot be free.  There is a cost.  We can
distribute the cost, but we cannot eliminate the cost.  However, I also
think the IESG really needs to support it -- even if that means using a
little of their time -- or it won't fly.  And, I think it needs to
involve personal requests not announcements or people just won't do the
reviewing.  That's my two bits.

allman

-- 
Mark Allman -- ICIR -- http://www.icir.org/mallman/

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 May 2005 16:44:17 +0200

Loa,

We can't fix the human mind. But are you saying it's a bad thing
if people prepare their drafts for cross-area review before
WG adoption, rather then a year later when they get hit with a
DISCUSS?

     Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 May 2005 16:48:03 +0200

> chair of the other WG and report back; frequently, nagging is required. 

We will never get away from nagging. As I just said, we need to get
in the mindset that causes nagging to occur.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 May 2005 16:46:49 +0200

> That was my reaction, too.

There's no doubt of it. And what this would do, I think, is move
us collectively closer to the mindset that early cross-area review
is a normal thing. I'd be delighted if a side effect was to prompt
people to ask individuals for cross-area review, with the
announcement acting as a gentle reminder.

    Brian

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Tue, 10 May 2005 10:56:02 -0400

>From where I sat, it wasn't a question of lack of IESG support, it was 
the lack of WGs and documents at the right stage to benefit.

---

From: "Randy Presuhn" <[email-masked]>
Date: Sat, 14 May 2005 10:31:40 -0700

Hi -

> From: "Tim Bray" <[email-masked]>
> To: "Brian E Carpenter" <[email-masked]>
> Cc: "Joel M. Halpern" <[email-masked]>; <[email-masked]>
> Sent: Tuesday, May 10, 2005 7:17 AM
> Subject: Re: New idea for early cross-area review
...
> drafts, they typically have two or more "last calls".
...

I think early review is valuable and important, but...

Others have written about how painful and time-consuming it is to review
documents that are in "rough" shape.  Looking at a -00- document, or
even some "last call" documents, it's easy to get bogged down in all
the i-d nits that the editors have failed to take care of.  This leads me to
believe that people will avoid looking at the earlier versions of documents
because they feel it will be a waste of their time.  It's hard to pay attention
to the "big picture" when a document is an editorial mess, or an embedded
MIB module or ABNF is riddled with errors.

Randy

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 10 May 2005 12:15:03 -0400

>     holes.  (Not in the reviews, but in the expertise of the folk doing

There's also the problem of finding someone with appropriate
cross-functional knowledge.

One of the problems I've noted when I've been requested to do a review
of some other WG's document is the knowledge mass involved.  While I
might, e.g., know BGP very well, if I'm asked to review something
for GMPLS I have a large body of documents I must review to provide
appropriate context. 

Effectively, this means the reviewer must already be working cross-
functionally between the working groups.  If not, they must have the time
to digest the knowledge base involved.  That's a lot of work.

I'd suggest that the single biggest way to draw the unsolicited
attention of someone who has interest in work that is cross-functional
is to make sure that the dependencies are readily visible once a
proposal gets to a certain level of maturity.  Imagine if you would
a system where the draft is "appropriately" tagged and this tagging
allows one to notice that something of interest has cropped up.  To give
an example, I would like to subscribe to something like an RSS feed of
documents that refer to using or changing BGP or BFD.

Something like this could even be put in the internet-drafts announcements.

All of this said, nothing will replace solicited reviews at last call.

-- 
Jeff Haas 
NextHop Technologies

---

From: Sam Hartman <[email-masked]>
Date: Tue, 10 May 2005 17:14:29 -0400

>>>>> "Steven" == Steven M Bellovin <[email-masked]> writes:

    Steven> In message <6.2.1.2.0.20050510073309.02f771b8@localhost>,
    Steven> "Joel M. Halpern" wri tes:
    >> It is worth a try.  I am doubtful, since most experience
    >> indicates that getting work out of people is much more
    >> effective if they are asked individually.

    Steven> That was my reaction, too.

I think there is a case where this might end up working.  If I believe
there is a significant chance of something negative happening if I
don't act then I'm likely to act.

I think it may be possible to organize groups around particular
technologies that happen to take these announcements and review uses
of these technologies.  Some such groups may be organized by ADs; some
may be self organizing.

The goal of this proposal is to give people who want to do cross-area
review experiments the tools they need in order to know what might
need reviewing.

---

From: Mark Allman <[email-masked]>
Date: Tue, 10 May 2005 17:46:17 -0400

> might, e.g., know BGP very well, if I'm asked to review something for
> GMPLS I have a large body of documents I must review to provide
> appropriate context.

I think that you'd pick a variety of folk for their particular
expertise.  I do gen-art reviews.  I look for what I know about and
general things that look funny or don't make sense.  A whole lot of the
things I review I am not intimate with.  So, if you know BGP and
something in a document throws up a BGP red-flag then you mention it.
And, you leave security to someone else and congestion control to a
different person.  It's great when one person can cross a ton of areas,
but I think that getting a variety of different eye-balls on something
is probably the most tractable way to improve things.  Cross area review
(IMO) is a shared task, not a detailed review that one needs to "study
up" for.

allman

---

From: Jari Arkko <[email-masked]>
Date: Wed, 11 May 2005 10:32:48 +0300

>MIB module or ABNF is riddled with errors.
I agree its hard, but its really important to focus on the
right things in an early review. I.e. does this thing fit in
the right place in the architecture? Can it be deployed? Would
there be other better approaches? Does it have major
holes in functionality, security, characteristics? Not I-D
nits, not running ABNF verification tools etc. But a major
problem can be that sometimes early drafts just contain
the core pieces of a protocol but very little explanations
around it, which makes it harder to understand the proposal's
role in the overall architecture.

As an aside, different WGs adopt -00 drafts in different
stages. I confess that we have sometimes used WG draft
adoption as a extortion tool to make sure that its well baked
and that people do enough work, before committing to
a new charter item. ("That other approach looks good too.
Why don't you make a new revision and try to make
yours perfect so that we'll pick it and not the other one."
Or "There seems to be open issues. If you succeed in closing
them we'll consider it.")

--Jari

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 11 May 2005 09:06:20 +0100

>   the early review guidelines, on the foundations of the work done by the

Since the WG chairs have to explicitly approve a draft as WG,
why not have them explicity list the WG that ought to review the
work. The secretariate can then CC those WGs either by posting the
advisory to the WG list, or CCing the chairs.

Stewart

---

From: Rohan Mahy <[email-masked]>
Date: Wed, 11 May 2005 17:23:09 +0200

Hi,

I've been monitoring this thread which does not appear to be 
converging.  I believe that cross-area review is important, but I am 
extremely skeptical that early cross-area review as described in this 
thread will produce much value.  I'm also concerned that any new 
procedure that we institute that doesn't produce results will increase 
the general mistrust for new procedures in the future.

In my experience, a WG will occasionally start driving toward a known 
pothole (or landmine) that the IETF already knows how to avoid in 
another context with the application of an appropriate cross-area clue. 
  I don't believe that most WGs end up needing this kind of clue on 
every (or even most) WG drafts.  Yet having folks unfamiliar with a 
specific WG try to determine out of context when the WG is going into 
weeds is a bit like asking an Hawaiian islander help an Inuit navigate 
through the Sahara desert.

So, I am proposing a concrete suggestion.  Rather than try to have an 
outside group determine when a WG needs a clue by examining every WG 
draft, each WG should have at least one experienced cross-area person 
that pays attention to that specific WG.  The purpose of this 
cross-area person per WG is only to raise the red flag that the WG may 
be wandering into the weeds on a new WG draft and that a particular 
type of cross-area review is warranted.  Once the red-flag is raised, 
we need volunteers with various types of clues (security, DNS, URIs, 
congestion, MIB, IPv6, etc.) who can go help out the WG and especially 
the relevant authors early in the process.  Note, that some ADs and 
chairs *already* request cross area review in certain areas, and that 
certain pools of clue-full folks already have a process to assign a 
reviewer (ex: security reviewers, MIB doctors).

I think the critical part of the proposal is that the act of raising 
the red-flag is separate from the act of reviewing flagged drafts, and 
that each WG has a cross-area clue-full person (separate from the AD) 
who only needs to be looking out for potential problems in the big 
picture.

Thoughts?

thanks,
-rohan

---

From: "David B Harrington" <[email-masked]>
Date: Wed, 11 May 2005 15:33:05 -0400

If you have enough reviewers to cover all WGs, and if the IESG
supports assigning these reviewers/technical advisors as part of
chartering a WG, then I would support such an approach.

For a concrete step in the meantime, I recommend that the chair's
training presentation should advise chairs to seek cross-area review
early (if it doesn't already).

David Harrington
[email-masked] 

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Rohan Mahy
> Sent: Wednesday, May 11, 2005 11:23 AM
> To: Brian E Carpenter
> Cc: Joel M. Halpern; [email-masked]

> > We'd like your comments on a new idea on how to achieve this.  The

> > thought
> >     The FOOBAR WG has adopted draft-ietf-foobar-XXX-00.txt as a WG

> > draft
> >   the early review guidelines, on the foundations of the 
> work done by 
> > the
> >   former ICAR WG.
> >
> >      Brian
> >
> >
> >
> >
> >
> >
> 
> 
>