Subject: [Icar] an early review experiment

From: Mark Allman <[email-masked]>
Date: Tue, 18 May 2004 14:02:34 -0400

Folks-

Joel and I chatted with some folks from the change process coordination
team this afternoon.  We would really like to get some discussion going
on the mailing list with regards to moving ahead on *an* early review
experiment.  Our sense is that even though there are several documents
in the early review space that there is an intersection among them that
people would like to see, but that has not yet emerged in a concrete
way.  In an effort to push along the conversation we are going to ask
some broad sorts of question below to try to synthesize thoughts into an
experimental design.  If this doesn't work (for whatever reason) our
fallback is to try to take each concrete proposal and beat it into shape
and try experiments with all that we can beat into enough shape to do
so.  

In that context, the broad questions that we'd like to discuss are:

  + At what point should early review happen?  Pre-WG document?  WG -00
    document?  Pre-WGLC?  All of the above?  Other?

  + Who is responsible for finding reviewers for a document?

  + From what pool are the reviewers drawn?

  + Precisely, what is produced?

  + What does a reviewer do with a review?

  + What other things need nailed down before we can think about an
    experiment?

(And, note, some of these are thrown out for completeness and we believe
there is decent consensus on the answer - e.g., the "what to do with a
review?" question.)

Discussion would be appreciated!

allman

--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

---

From: Pekka Savola <[email-masked]>
Date: Tue, 18 May 2004 23:10:02 +0300 (EEST)

>     document?  Pre-WGLC?  All of the above?  Other?

Because it seems *very* difficult to kill off WG work items after they 
have been accepted, some review should happen just before accepting a 
document as a WG work item.

This should be done to kill any obvious bogosities and alert of
cross-area problems.

A second set of early reviews should happen at some point after -00 WG 
document and between/around WGLC.  Whenever the document is deemed 
suitably stable for consumption..

>   + From what pool are the reviewers drawn?

These two are IMHO the most difficult questions from the practical 
point of view.

For example, as WG chair, I've been instructed to get review from 
e.g., apps area for a document.  Despite begging from the apps ADs, 
mailing numerous times on appsarea mailing list, getting a person 
signed up for review (who ultimately did nothing), I have to say it 
must be *very* clear who has the "responsibility" for producing 
reviewers/reviews for the document.

This cannot just be the WG chair or the shepherding AD in question.  
Maybe if there was an area-specific early review board, the chair(s)  
of that board would be responsible for getting the document reviewed.

As to the pool where the reviewers should come from.. that's maybe 
slightly easier.  This stuff would just need to be organized 
sufficiently well, with sufficient buy-in from the IESG, and 
appreciation by the peers.  The criteria like used for SIRS or to get 
ARTS might work here, as well as volunteers.  But the most important 
thing is that those people do actually review documents (at least 1 
per month), and there is a system to make sure the review will end up 
getting done.
 
>   + Precisely, what is produced?

A SIRS-like decree like "ship it, needs polish, needs a lot of work,
kill this madness" would probably be useful, but this should be 
coupled with technical comments as well.

>   + What does a reviewer do with a review?

Did you mean 'what does the reviewee (whose document one reviewed) do 
with the review?' as I don't understand why the above question is 
relevant.

Assuming so, this is again a good question.  The review should be sent
tot he document editor(s), WG chairs, and possibly the associated AD
-- and one of which should file it in an issue tracker if used.  The
reviews should also be publicly stored in a web location.

The comments should be treated with equal weight as IESG Discuss 
comments.
 
>   + What other things need nailed down before we can think about an

It's already implicit from the above, but it would probably make sense 
to consider at more length when this kind of review should be 
initiated and by whom. I.e., whose responsibility is it to start 
the review process, and ensure that sufficient review gets done 
before going to the next step in a draft's lifetime (e.g., 
adopting it as WG item, etc.).

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 19 May 2004 10:06:56 +0200

These questions are discussed in detail in
draft-carpenter-solution-sirs-01.txt
but I have done some cut and paste below.

>     document?  Pre-WGLC?  All of the above?  Other?

      It is not expected
      that every version of an Internet Draft should be
      submitted for SIR review. However, it is advisable to
      request reviews at the very beginning (to check for
      fundamental issues), as major technical issues are
      resolved, and again just before the document is
      submitted for IESG approval (or to the RFC Editor).
      Thus three SIR review cycles per document may be
      considered the minimum. The document author should
      ensure that the SIRs reviewing a document understand
      what stage in its life cycle it has reached, so that
      they can review it appropriately.

      Both Working Groups and individual submitters should
      realise that carding should start early (to detect and
      hopefully fix fundamental problems) and be repeated as
      often as needed (to avoid submitting inadequate
      documents to the IESG).
>   + Who is responsible for finding reviewers for a document?

      2.2.1.    Working Group documents

      For documents being processed by a Working Group (WG),
      the WG solicits the assistance of SIRs.

      [which to my mind means the WG Chairs or Secretary]
      ...
      2.2.2.    Individual submissions

      For individual submissions, the document author(s)
      should solicit SIR reviews, according to the same
      principles applied to Working Group documents,

>   + From what pool are the reviewers drawn?

      An initial target of 100 people is suggested. It is
      recognized that this may be insufficient, so the number
      should be revised in the light of experience.

      Q.   Is this anything like the right number?
      A.   Assume we need 9 reviews per year (3 review cycles,
           3 reviewers each time) for each of 200 RFCs. That gives each
           SIR one review every 20 days, approximately. This seems
           high, and suggests that the final target should be 200
           people.

      The goal is to identify a team of people with adequate
      experience contributing to IETF technical work, who are
      likely to be trusted as a group to be both technically
      expert and unbiased.

>   + Precisely, what is produced?

There is a whole section (3.3) about this in the SIRs draft,
from which this is a very small extract:

      SIRs should review for all kinds of problems, from
      basic architectural or security issues, Internet-wide
      impact, technical nits, problems of form and format
      (such as IANA Considerations or incorrect references),
      and editorial issues. As a draft progresses from its
      initial, "-00" version towards one that is ready for
      submission, successive SIR reviews should progress from
      the general architectural level to the editorial level.

>   + What does a reviewer do with a review?

3.1. Reviewing in Public

      ... All reviews will be published on the web
      site, with adequate tooling (linked to
      the ID Tracker) so that SIRs can publish reviews
      without adminstrative overhead, every review can be
      readily found, and all reviews will be automatically
      archived. In fact, a WG or document author in need of
      reviews should be able to request them through the web
      site
>   + What other things need nailed down before we can think about an

 From the SIRs experience, do not even think about starting until
you have a good web site up & running where (a) all reviewers can be
registered (b) authors and WG Chairs can request reviews and
(c) reviewers can post, and the public can read, their reviews.

       Brian

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 19 May 2004 06:48:02 -0500

I'm registered as a SIR, so I'm obviously pretty comfortable with the
view of SIRs in this draft, but it seems to me like one tipping point
needs to be pointed out.

From: "Brian E Carpenter" <[email-masked]>
Sent: Wednesday, May 19, 2004 3:06 AM

>       the general architectural level to the editorial level.

It seems to me that I'm seeing two types of reviewer pools proposed -
a reviewer pool that ties to a particular Area of expertise (so the
transport ADs provide transport reviewers to the other Area Review
Teams, and these reviewers look for transport sanity), or a reviewer
pool that reviews as generalists, as do SIRs (and as do ADs, during AD
review).

There are reasons for both, even for early cross-area review. It Would
Be Nice if we didn't try to choose between them.

> (c) reviewers can post, and the public can read, their reviews.

Amen.

---

From: Mark Allman <[email-masked]>
Date: Wed, 19 May 2004 09:07:10 -0400

> with the review?' as I don't understand why the above question is 

No.  I was asking where reviews go, how the information generated by the
reviewer is fed back to the WG / authors.

> The comments should be treated with equal weight as IESG Discuss 

This brings up a good point that should have been included on my list of
questions.  

  + How binding are early, cross area review comments on authors /
    editors / WGs?

I'd like to hear additional opinions on this point.  (And, Pekka's
assertion above leads to many questions.  I'd rather not rat-hole on
those until we figure out what sort of high-level feelings folks have on
this question.)

Thanks!

allman

---

From: Melinda Shore <[email-masked]>
Date: Wed, 19 May 2004 10:37:07 -0400

> reviewer is fed back to the WG / authors.

I prefer that it go to the wg mailing list - the wg needs to know
what's going on, and the process needs to remain as transparent
as possible.  I think a bigger question is how the completed reviews
are managed - whether they're maintained in a database (the draft
tracker?), whether or not they should be bundled with the draft
when its submitted to the IESG, etc.

>   + How binding are early, cross area review comments on authors /

I'm not sure they're *binding*, although they're surely more weighty.
Worst case scenario:  wg wants to do something inane, early review
catches it, wg says "nope, we really intend to do this," inanity gets
through wg last call and is submitted to the IESG, the IESG says "this
is inane, we're not going to publish it."  The wg has had plenty of
warning that they're off track, presumably the responsible AD has had
the same notice, and it's completely on the shoulders of the wg and the
wg chair for having produced inane work.  That strikes me as a healthy
process.  Having reviewers tell the working group what they can and
cannot do seems potentially stifling to me.

Melinda

---

From: Mark Allman <[email-masked]>
Date: Wed, 19 May 2004 10:49:50 -0400

> I'm not sure they're *binding*, although they're surely more weighty.

"More weighty" than ... what?

If I were to sketch the "weighyness" space of early, cross area reviews,
it'd be something like this:

  + one end of the spectrum would be that ICAR-type comments must be
    addressed before the document moves forward

  + the other end of the space seems to me to be that the ICAR-type
    reviews are just like any other review that comes in from a
    community member (i.e., no special status)

  + there may be middle ground

It'd be nice to hear thoughts on where folks are in this space.  

(If I were to interpret what has been said thus far, I'd say Pekka's
note falls under the first bullet and Melinda's note under the second.
Roughly.  If that is way off, of course you should yell because I don't
want to put words in anyone's mouth.)

Thanks!

allman

---

From: Melinda Shore <[email-masked]>
Date: Wed, 19 May 2004 11:04:07 -0400

>   + there may be middle ground

I'm in the middle ground.  It seems to me that what ICAR-type reviews
bring to the wg is external context and persective that isn't available
within the working group.  I wouldn't like to see working group
responsibility for the quality of their own product be shifted to non-
participants.  What I think we need is more checks as documents
progress, and while a working group needs to take review comments
very seriously, sometimes a reviewer is full of crap, too.  I don't
think it's the role of this working group (icar) to make fundamental
changes to the standardization process and responsibilities.  More
validity checks on working group documents earlier in the process: 
great.
Changing the process so that reviewers become responsible for working
group documents: not so great.  And giving reviewers what's essentially
veto power over documents gives them responsibility for those documents.

Melinda

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 May 2004 13:33:05 -0600 (MDT)

On Wed, 19 May 2004, Mark Allman wrote:

> If I were to sketch the "weighyness" space of early, cross area
> reviews, it'd be something like this:
>

IMO, WG MUST publicly address all reviewer concerns, but can do so any
way it wants. WG MAY reject to act on any comment, for any reason, but
MUST publicly document that the comment is rejected and SHOULD
document rejection rationale.  IESG SHOULD NOT review documents with
unaddressed concerns.

For these rules to be meaningful/effective, there has to be a
mechanism/tool to register specific concerns and WG responses to them
(e.g., ID Tracker DISCUSS items on steroids). Just exchanging freeform
e-mails is not good enough.

It seems to me that same rules should apply to any review, regardless
of its source. There may be periods when WG does not accept reviews
for specific documents though (because they are not ready for any kind
of review or because the WG has finished working on them and wants
IESG to take over).

One simple way to manage review timings is to have both First Call and
Last Call for each draft (at least). These calls will make it clear
when reviews are welcome (and when reviewer comments are virtually
guaranteed to be addressed).

I do not know whether the above fits any of the three bullets you
proposed. It seems lile the second bullet because all reviews have the
same status (but I would call that status "special" compared to the
current "no special status" situation), but it also matches the intent
of the first bullet as far as final IESG action is concerned.

HTH,

Alex.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 19 May 2004 12:39:27 -0700

--On 19. mai 2004 09:07 -0400 Mark Allman <[email-masked]> wrote:

> I'd like to hear additional opinions on this point.

having been through some 70 reviews in the Gen-ART group, I know very well 
that some review comments are based on misunderstandings, and some only 
point out that the point needs to be better explained... I think that it's 
vital that review comments be *addressed*, but frankly, sometimes the WG or 
the author will be *right*!

(I sometimes miss more intensive arguments from a WG against an AD's 
opinions... I should be careful what I wish for :-)

---

From: Mark Allman <[email-masked]>
Date: Wed, 19 May 2004 16:17:17 -0400

I want to take a tangent for a moment...

> document rejection rationale.  

I'd like to hear whether folk's thoughts WGs completely ignoring
comments.  I understand that it *could* be a problem and that it could
be aggivated by a new early review scheme.  But, I am wondering whether
folks think WGs currently ignore comments they do not like without
comment.  This is different than having a discussion and then not
following the given comments.  I am talking about comments that are sent
and then completely ignored -- no discussion, no response, nothing ....
And, this is also different from any particular machinery (e.g., filing
the comments in a proper issue tracker, etc.).

Thanks!

allman

---

From: Pekka Savola <[email-masked]>
Date: Wed, 19 May 2004 23:21:58 +0300 (EEST)

> the author will be *right*!

What Harald said.  IMO it's a requreirement that the comments have to
be addressed.  If you think the comment is off base, send a mail to
_discuss_ why you think so.  If the person agrees, the comment is
addressed, no need to change anything.  Or add something else wherever
to clarify.

If you can't reach consensus (e.g., you think the suggestion is bad
one, but the review will not relent), bigger guns must come to the
play to judge the situation, e.g., the appropriate ADs, or whatever.

The whole point of early cross-area review would be useless if the 
comments could be ignored on whim.  And on the other hand, if the 
commenters are not sufficient good judges of what's a critical issue 
and what's not, we'd have an equally big problem.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 May 2004 16:22:02 -0400

Most of these comments, while fair concerns, seem to be primarily addressed 
to the general question of how working groups manage comments.  General 
working group process is probably outside of the scope of this working group.
Yours,
Joel M. Halpern

PS: To be specific, I do not think that the ICAR working group can direct 
that working groups should (much less must) have problem trackers, and 
should (or must) address all comments received, regardless of source.  Even 
though those may be good ideas.

---

From: Allison Mankin <[email-masked]>
Date: Wed, 19 May 2004 13:27:19 -0700

Hi, Mark,

>   + What does a reviewer do with a review?

I think reviews which are part of the ICAR process need
to be public.  I agree with Melinda's reasoning.  In addition, the
working group can get better at cross-area thinking, which can happen
best if the review's in the open.

>    community member (i.e., no special status)

I'm in the middle, for now.  You started out asking for this to be a
discussion of an experiment.  After we experiment, my position could
change.

I'd like to see all reviews result in some kind of response about how
comments are addressed.  ICAR-type comments should require* that this
response be prepared and sent to the mailing list.  But if the
document does not change at all in response to the review, that is OK:
the document should not block (that is, the Chair's, WG's and AD's
discretion with respect to the document should not change).

The ICAR review still has clout: from the enforced response, the Chair
should lead a discussion, and the result should be a serious working
group consideration of the review.

In my experience of quite a few AD reviewss (and those of AD's
directorates), there's a steep learning curve for cross-area review,
and also a difficult line between when something is actually a working
group consensus matter rather than a cross-area matter.  My experience
is that the IESG learns how best to cross-area review from reviewing
each others' reviews and from reviewing hundreds of documents in all the
areas.  There are certainlly other ways to get there, but we have no
handle on this now.

Bottom line: I think it's important to see how well the IETF can
produce consistent cross-area reviewers, which includes developing the
shared and concrete view of what the cross-area field is.  For this
experiment, I'm against blocking documents

Allison.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 May 2004 14:35:28 -0600 (MDT)

On Wed, 19 May 2004, Mark Allman wrote:

> I am wondering whether folks think WGs currently ignore comments
> they do not like without comment.  This is different than having a
> discussion and then not following the given comments.

In my limited experience, I have seen enough cases where my comments
were not addressed. Thus, I suspect that there are lots of cases like
that, especially in "controversial" areas.

Please note that I personally do not think there is an important
difference between "discussing and then ignoring" a comment and
"ignoring without discussion" (as far as the scope of your questions
goes). The result is the same -- the comment is not addressed. Without
a concise statement of action, informal and fuzzy mailing list
discussions mean little if anything.

> And, this is also different from any particular machinery (e.g.,
> filing the comments in a proper issue tracker, etc.).

While there is certainly a theoretical difference, in practice, it
makes little sense to ask whether citizens are usually law-obeying if
the country does not keep a record of law violations or, worse, does
not define what a violation is. You will only hear about high-profile
murders and neighbor disputes which are unlikely to give enough
quality information about the general state of affairs.

Alex.

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 May 2004 14:42:46 -0600 (MDT)

On Wed, 19 May 2004, Joel M. Halpern wrote:

> Most of these comments, while fair concerns, seem to be primarily
> addressed to the general question of how working groups manage
> comments.  General working group process is probably outside of the
> scope of this working group.

The charter seems to place any mechanism that encourages review of
documents in scope. There is probably nothing more discouraging than
seeing your comments ignored!

> PS: To be specific, I do not think that the ICAR working group can
> direct that working groups should (much less must) have problem
> trackers, and should (or must) address all comments received,
> regardless of source.

I do not see why not. The charter does not seem to place such
essential mechanisms out of scope. Am I missing something?

Thanks,

Alex.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 19 May 2004 17:43:31 -0400

It is possible that you did not intend as large a scope as I read your 
comments as covering.  If so, I apologize.

Mandating that issues raised in ICAR describe reviews must be responded to 
in some fashion by the working group is something within charter for ICAR 
to say.

Mandating that a WG must respond formally and on record to all comments it 
receives at any point in its work may or may not be a good idea, but is a 
working group process question, not a cross-area review question.  I had 
read your (elided for brevity) comments as calling for the later.  I 
apologize if the error was in my reading.

Yours,
Joel M. Halpern

---

From: Melinda Shore <[email-masked]>
Date: Wed, 19 May 2004 17:43:53 -0400

> I'd like to hear whether folk's thoughts WGs completely ignoring

I've seen working groups fail to adequately address comments and
then have it come back to bite them in later review or even when
the IESG is reviewing a document.  I've also seen them completely
ignore comments, but haven't seen one ignore comments from
someone with a formal role.

The thing is that the reasons for them doing so vary, from the
comments being muddled and irrelevant to the working group being
insular and overly committed to the wrong path.  I would tend to
look for a chairing failure when it happens.

Melinda

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 May 2004 16:16:28 -0600 (MDT)

On Wed, 19 May 2004, Joel M. Halpern wrote:

> Mandating that a WG must respond formally and on record to all
> comments it receives at any point in its work may or may not be a
> good idea, but is a working group process question, not a cross-area
> review question.  I had read your (elided for brevity) comments as
> calling for the later.

I was calling for both, I guess. Even if ICAR scope prevents us from
developing a general mechanism for all reviews, it does not prevent us
from developing an in-scope mechanism that can be used equally well
for out-of-ICAR-scope reviews.

For example, we can say that WG MUST address all First Call and Last
Call cross-area review comments. We can they try to apply the same
principle to all First Call and Last Call comments, not just
cross-area ones.

Alex.

---

From: Mark Allman <[email-masked]>
Date: Wed, 19 May 2004 22:56:14 -0400

Joel-

WG chair sparring for fun & profit. :-)

> regardless of source.  Even though those may be good ideas.

While I agree that general changes are out-of-scope for this WG, I think
that something like requiring an issue tracker for some given ICAR
scheme / experiment is well within bounds.  And, if said issue tracker
is more generally useful than just for ICAR things then so much the
better.  It would seem as if we would just hamstring an ICAR mechanism
if we were to declare such things as out-of-scope.

allman

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 19 May 2004 22:34:04 -0500

I may be going through a phase, but I would prefer to see ICAR reviews
with no special status.

My dream (and perhaps the dream of others) is that ICAR reviews are
accessible from the document tracker. If this happens, it seems
trivial enough for the IESG, or whomever we finally decide actually
approves publication, to e-mail the ICAR reviewers and ask if there
are still outstanding issues in the reviewer's opinion.

If "no", the working group resolved the issues.

If "yes", the IESG knows which end of the haystack to start looking
at. If the review comment seems important enough to "discuss" (in the
IESG sense of the word), it will be resolved, later and more painfully
than was necessary.

My experience has been that reviewees were either thrilled to have
someone carefully reading their document, or were so detached that (I
am not making this up) a document that hit the IESG telechat agenda
within the last two months referred to keywords ala RFC 2119 AS A
DRAFT:

   [KEYWORDS] Bradner, Scott, "Key Words for use in RFCs to Indicate
        Requirement Levels", draft-bradner-key-words-03.txt, January,
        1997.

- so no one had looked for nits in the document in the last seven
years.

Either way, I don't think making a document editor agree with me is
necessary or desirable.

YMMV, of course.

Spencer

---

From: Alex Rousskov <[email-masked]>
Date: Wed, 19 May 2004 22:20:14 -0600 (MDT)

> are still outstanding issues in the reviewer's opinion.

IMO, the above would give us 80% of the benefit with 20% of the
effort, which is a wonderful ratio for any pilot experiment. There are
two tricky points here, I think:

	1) The comments must be individually identifiable (like
	   DISCUSS items). A long blob of text without specific
	   items for WG to address is not good enough.

	2) ID Tracker would have to authenticate reviewers
	   one way or another to prevent unauthorized access
	   and reviewer identity theft. If the experiment is
	   limited to few super-qualified SIRs, then this step is
	   easy. If the experiment is democratic, then it is not.

> Either way, I don't think making a document editor agree with me is

Agreed.

Alex.

---

From: "Spencer Dawkins" <[email-masked]>
Date: Thu, 20 May 2004 07:06:26 -0500

From: "Alex Rousskov" <[email-masked]>
To: "Spencer Dawkins" <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, May 19, 2004 11:20 PM
Subject: Re: [Icar] an early review experiment

>    easy. If the experiment is democratic, then it is not.

I would think that we would have this as an ICAR requirement because
people are sending unauthenticated private e-mail to ADs containing
forged comments today at Last Call time ("Hi, this is Geoff Huston,
and you need to DISCUSS this draft, because it *totally* overlooks
this really cool technique, please overlook the patent issues").

I have no idea whether this is happening (and if the fiendish
pseudo-reviewers are really gifted identity thieves, neither do the
ADs), but I am comfortable not worrying about it in ICAR, especially
if the ADs contact the apparent reviewers about any remaining issues
("no, I didn't write that review, thanks for asking").

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 20 May 2004 06:27:35 -0700

--On 19. mai 2004 16:22 -0400 "Joel M. Halpern" <[email-masked]> 
wrote:

> addressed to the general question of how working groups manage comments.
> General working group process is probably outside of the scope of this
> working group.
> Yours,
> Joel M. Halpern
>
> should (or must) address all comments received, regardless of source.
> Even though those may be good ideas.

hmmmm......

I think we have a little more power here than that....

the charter says:

> It is an explicit goal of the WG to come up with mechanisms
> encouraging early review of the documents while ideas are still
> in the formation stage.
.....
> The WG will cooperate with others in starting and evaluating
> experiments with early community and structured reviews. The
> evaluation of such experiments may be published as Informational
> RFCs if the group so desires.

When this text was originally drafted, I think we thought we'd have an 
MPOWR WG looking at WG processes in general; we did not find traction for 
that.

But - this group is certainly chartered to make experiments. It is a valid 
part of an experiment to ask a WG to volunteer to commit to address all 
comments raised in review (and to have another WG commit to NOT committing 
to that, if a control is needed), and it's within the charter of this WG to 
evaluate this experiment and conclude (if we still think so) that mandating 
that comments be addressed is a Good Thing (or not).

Then we can figure out how the IESG should act on that recommendation.

But - get the experiment running!

                       Harald

---

From: Mark Allman <[email-masked]>
Date: Thu, 20 May 2004 23:12:54 -0400

> ("no, I didn't write that review, thanks for asking").

I agree that this doesn't seem to be something to tackle in ICAR.  But,
I will note that we could encouraged signed reviews.  And, also, I think
the more public the review the less chance of it being forged without
someone noticing and making a fuss.  I.e., if Spencer is asked for a
review and a fradulent review comes and then the real review comes then
there is a very visible conflict.  And, Spencer might notice the
fraudulent version himself (i.e., "hey, I'd have never written that
slop").

Now, if someone were to impersonate me, I might just let it slide since
it'd probably make me look smarter. :-)

allman