Subject: [Icar] reviewer pools

From: Mark Allman <[email-masked]>
Date: Mon, 24 May 2004 08:20:06 -0400

I want to see if we can get some discussion going on how to design pool
of reviewers for ICAR-type experiments.  Some specific questions that
jump to mind:

  + What should the granularity of the reviewer pool be?  Should we have
    one big pool for the IETF (ala SIRS)?  Or, should their be an area
    review team?  Or, a WG review team?  Or, a review team specific to
    each i-d?  (That seems to be the space.  Or, what else could their
    be?)  Is there any value in having more than one granularity of team
    and having handoffs or overlap in some fashion?

  + Who should be eligible for these review teams?  E.g., the kind of
    concrete track record of IETF work outlined in SIRS?  E.g., teams
    populated by AD "nominations"?  E.g., teams populated by WG chair
    "nominations"?  E.g., purely voluntary teams with no hard
    requirements on who can volunteer?

If folks could give their opinions on the above questions (and, anything
else in this space, naturally), that'd be wonderful!

Thanks,
allman

---

From: David Partain <[email-masked]>
Date: Mon, 24 May 2004 14:54:54 +0200

Hi,

>     and having handoffs or overlap in some fashion?

My take is that we should (at least initially) use the current
area granularity.  It's what we have above the WG level and
I think WG granularity will be unmanageable.  Areas give us
a reasonable administrative level to organize around.

>     requirements on who can volunteer?

I think this is a hard issue.

Given the "volunteer" nature of the IETF, I think we should let
volunteers prove their worth.  That's why my draft suggests
a volunteer pool.  I believe that adding requirements on who
may review and who may not is going to reduce the chances of
getting sufficient numbers of reviewers.  I realize that this
opens the door to people who perhaps have no business giving
"expert" review, but it also may allow people to gain a track
record as reviewers despite lack of previous experience.

Cheers,

David

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Melinda Shore <[email-masked]>
Date: Mon, 24 May 2004 10:33:00 -0400

>     and having handoffs or overlap in some fashion?

It seems to me that this question is at least a little related to the
question of the role of the "technical adviser" assigned to some
working groups.  I'm not sure that function is working very well, but
assuming it is, having a per-wg review team would appear to overlap
with the technical advisor.  That's not necessarily a bad thing.

Another related question is how the reviewers are selected or assigned.
My preference is to have a general pool of reviewers and have the 
reviewers
get per-document review responsibility, but that implies that there's
someone shepherding the document review, as well, and making sure that
it happens in a timely way and doesn't just drop through the cracks.
That person would request the review.  My primary concern with how
the reviewer pool functions is the possibility of gaps in coverage,
either due to expertise or availability, and I think I would tend
to optimize how reviewers are selected/provided around that issue -
how do you ensure that every document that needs review gets it?

Melinda

---

From: Mark Allman <[email-masked]>
Date: Mon, 24 May 2004 11:01:05 -0400

I am not arguing for one way or the other, just trying to explore ...

> reviewers get per-document review responsibility, but that implies
> that there's someone shepherding the document review, as well, and
> making sure that it happens in a timely way and doesn't just drop
> through the cracks.  That person would request the review.  My primary
> concern with how the reviewer pool functions is the possibility of
> gaps in coverage, either due to expertise or availability, and I think
> I would tend to optimize how reviewers are selected/provided around
> that issue - how do you ensure that every document that needs review
> gets it?

I'm interested in hearing your thoughts on area review teams(*) then.
That makes things much more explicit ... i.e., "we need a reviewer from
the security team" rather than fishing around a large general pool (as
you note is your preference) for someone who can cover security
documents.  Does pre-sorting make for a system that is less prone to
"gaps"?  (Of course, with some alternate downsides, maybe.)

(*) A thought that just popped into my head is that "area teams"
    wouldn't necessarily have to fall along traditional boundaries -
    but, might.  Call them "topic teams" or something.  We could have
    topic teams on security (traditional IETF area) and architecture
    (not an area), for instance.  Just throwing it out there ... I am
    not sold on it.

Thanks!

allman

---

From: Melinda Shore <[email-masked]>
Date: Mon, 24 May 2004 11:21:00 -0400

> "gaps"?  (Of course, with some alternate downsides, maybe.)

I'm not sure.  We're stretched as thin as can be on security
reviewers today, for example.  Will having a more formal review
process make more security reviewers available?  Probably not.

I asked SIRS to review one document for midcom.  We got three
reviews out of it.  They were all useful, and one of the really
interesting things that happened is that at least one of the
reviewers hadn't been familiar with our work or, I think,
with the area (border policy enforcement) in general, and that
brought a valuable fresh pair of eyes to the document.

Part of the question is related to when the review happens, too -
earlier in the process you need someone that brings a macro
perspective to the entire problem, and later in the process,
presumably the macro questions have been answered and you need
someone with more specific, focused expertise to address particular
problems within the documents.

The issue I'm interested in seeing cross-area review deal with
is the broad problem of working groups starting out well and
somehow going off the rails early on.  That, I think, benefits
from wisdom and experience rather than specific area expertise.

Melinda

---

From: Mark Allman <[email-masked]>
Date: Mon, 24 May 2004 11:35:39 -0400

> process make more security reviewers available?  Probably not.

I was trying to ask about your thoughts on using smaller teams (i.e.,
not one large pool) as a way to manage the process with finer
granularity and hence avoiding the "gaps" you noted.

Clearly, there are resource issues no matter which way you go.  You
thoughts on those are useful (as always), thanks!

allman