Subject: [Icar] Progress?

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 13 Apr 2004 21:19:54 -0400

Folks,
     We are now at mid-April.  We are not making much progress on proposals 
for early or late review.  We have two documents available at the moment.
     draft-carpenter-icar-sirs-00.txt
     draft-zinin-icar-arts-00.txt

The first is a starting point on early reviews, the second a starting point 
on later / formal reviews.
There has been no discussion of either document on the mailing list.
To try to help, this note picks on one side, and asks one specific 
question.  The context of the rest of this is early review.

Taking it as given that we would like to see further experimentation with 
early reviews, it follows that we need some proposals for ways that might 
be tried.  To help folks who might like to make proposals (I hope there are 
some), it seems useful to know what properties a good early review proposal 
should meet.

Question for the list:
     What are the goals a good early review proposal should meet, or 
properties it should have?

Some ideas for things we might discuss:
Should the proposal explicitly address cross-area, or just outside the WG, or ?
Should the proposal address the reported problems from other review panels 
with calling for volunteers to review specific documents?
Should the proposal have detailed requirements for the review, or just 
guidelines as to what an initial summary needs to say?
Should the proposal include requirements / restrictions on qualifications 
of reviewers?
Should a proposal provide for formal inclusion of early review in the 
standards process, or should it just be a mechanism for a WG getting early 
feedback?
Should the proposal provide incentives for WGs to make use of early 
reviews, or "rewards" for doing so?

Yours,
Joel M. Halpern

PS: While it is still very drafty, we do have a working group web page with 
links to both those documents at 
http://www.machshav.com/~icar.  Suggestions for additional content we 
should have on that page would be helpful.

_______________________________________________
Icar mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/icar

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 14 Apr 2004 07:41:10 -0700

--On 13. april 2004 21:19 -0400 "Joel M. Halpern" <[email-masked]> 
wrote:

> PS: While it is still very drafty, we do have a working group web page
> with links to both those documents at http://www.machshav.com/~icar.
> Suggestions for additional content we should have on that page would be
> helpful.

I suggest that we add links to running experiments on the page.
The SIRS is one example.
I offer http://www.alvestrand.no/ietf/gen/review-guidelines.html as another.

                        Harald

---

From: Dave Crocker <[email-masked]>
Date: Wed, 14 Apr 2004 09:41:34 -0700

Joel,

Thanks for pursuing this.  By way of trying to help prime the pump:

JMH> Should the proposal explicitly address cross-area, or just outside the WG, or ?

Cross-area, versus outside the wg, can be viewed as two different
concerns, both very important.  Hence, I think the proposal should
deal explicitly with both, but leave the actual choices to the working
group.

This deferral to the wg is based on the view that the proposal should
provide a mechanism that is available to the working group, but that
the wg chooses whether to use it, rather than having the proposal
impose anything on the wg.

JMH> Should the proposal address the reported problems from other review panels
JMH> with calling for volunteers to review specific documents?

That would be nice, but not essential.  All feedback is relevant.
Requiring that a particular proposal deal with the entire range of
possible feedback could be more distracting than beneficial.

JMH> Should the proposal have detailed requirements for the review, or just
JMH> guidelines as to what an initial summary needs to say?

If we view proposals as offering services to working groups, rather
than imposing requirements to them, then the the question of review
process and content detail become a matter of quality and style,
rather than formal requirement. More detail is usually better, but not
always.

JMH> Should the proposal include requirements / restrictions on qualifications
JMH> of reviewers?

Again, this is a matter of addressing quality control of the review
process.  The more random and unpredictable the backgrounds and skills
of the reviewers, the more random and unpredictable will be their
reviews.

JMH> Should a proposal provide for formal inclusion of early review in the
JMH> standards process, or should it just be a mechanism for a WG getting early
JMH> feedback?

I think the formal view should be the latter, and that the former will
evolve naturally from that, as the reviewing is shown to improve
quality and timeliness.

JMH> Should the proposal provide incentives for WGs to make use of early
JMH> reviews, or "rewards" for doing so?

BY implication, yes, but not by formal rule.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 15 Apr 2004 18:47:54 +0200

To a large extent I agree with Dave, but not everywhere...

> impose anything on the wg.

But at the end of the process, the IESG will impose cross-area review
on the WG; that's what the IESG does. So if we want to ease the perception
that the IESG is a bottleneck, we need to *encourage* WGs to solicit (and
react to) cross-area review. 

> rather than formal requirement. More detail is usually better, but not

For some strange reason I think the level of guidelines for the content
of a review in the SIRs draft is about right. Note, they are guidelines,
not requirements.

In some of the reviews I've done recently for Harald, I've been amazed -
I've seen both void Security sections and missing tables of contents
in documents already on the IESG's table. So it seems we do need to
apply elementary guidelines, even before getting to review the meat of
a document.
> of the reviewers, the more random and unpredictable will be their

We need to make the system self-correcting (i.e. people who turn out
to be weak reviewers simply don't get asked back.) 

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Brian E Carpenter 
Distinguished Engineer, Internet Standards & Technology, IBM

---

From: Dave Crocker <[email-masked]>
Date: Fri, 16 Apr 2004 09:19:07 -0700

Brian,

>> impose anything on the wg.
BEC> But at the end of the process, the IESG will impose cross-area review
BEC> on the WG; that's what the IESG does. So if we want to ease the perception
BEC> that the IESG is a bottleneck, we need to *encourage* WGs to solicit (and
BEC> react to) cross-area review.

Definitely correct.

The significant point, however, is that having this become a de facto
requirement is not part of defining or using the review process
itself.  Making it effectively mandatory is outside of the critical
path for initiating the review capability.

Rather, it develops as a requirement later, as the utility of early,
wg-initiated reviews becomes evident. Hence it is the usefulness of
the reviews that prompts working groups to get them done, rather than
the imposition of an externally imposed and externally managed rule.

I should comment that we need to be careful about thinking that the
only important reviews are cross-area and that cross-area reviews are
always required.  An algorithm change to BGP does not need a review by
Apps and well might not need a review by anyone outside of the Routing
area.  Definition of an RFC2822 header might not need review outside
of Apps.  It all depends on the nature of the thing being specified.

(But, yes, we currently do not do nearly enough cross-area review.)

>> rather than formal requirement. More detail is usually better, but not
BEC> For some strange reason I think the level of guidelines for the content
BEC> of a review in the SIRs draft is about right. Note, they are guidelines,
BEC> not requirements.

yeah, pretty remarkable.  equally astonishing is that i agree with
you...

d/

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 Apr 2004 10:36:09 -0400

Harald-

> I suggest that we add links to running experiments on the page.  The
> SIRS is one example.  I offer
> http://www.alvestrand.no/ietf/gen/review-guidelines.html as another.

Good thoughts.  I added pointers to both of these (as well as your late
xarea review i-d) to the web page.  The page is at:

    http://www.machshav.com/~icar

Folks- please do not hesitate to point out additional items that belong
on that page.

Thanks!

allman

--
Mark Allman -- ICIR -- http://www.icir.org/mallman/

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 Apr 2004 12:38:38 -0400

Dave-

Thanks for the input!

I want to push back a little to understand the thinking ....

> the imposition of an externally imposed and externally managed rule.

First, I think that we can seperate the "review function" from the
"process function" that might mandate/suggest/whatever the review
function, as you suggest.

But, I wonder if getting this all going is a chicken-and-egg thing.
I.e., if we don't invoke the review function as an explicit step in the
process will anyone do it?  It seems clear that if we could get folks to
start doing it then things may well snowball nicely (i.e., the community
would see that it was worthwhile).  But, in the absence of experience
that concretely shows that some sort of early review function is, in
fact, useful and worthwhile what is the incentive for a WG to invoke the
review function?

The other thing I wonder about is new-ish folk in the IETF.  People
bring ideas in that are not really baked, but they don't realize it (new
UDP-based app with no congestion control, no thoughts given to security,
etc.).  What is the incentive for these people to turn around and ask a
bunch of people they don't know to pick apart their work when they are
Obviously Correct?

SIRS didn't get much activity in the grand scheme of things.  So, in the
absence of expicit direction how do we get some sort of review function
going?

Thanks!

allman

---

From: Dave Crocker <[email-masked]>
Date: Mon, 19 Apr 2004 10:06:42 -0700

Mark,

MA> I.e., if we don't invoke the review function as an explicit step in the
MA> process will anyone do it?

We already have some working groups doing it. How do we get more? I
think that this is a matter of culture and educating folks about
benefits and, to some extent, expectations. Let's assume that
participants want good things. This is a reasonable assumption almost
all the time. This means that a benefits-oriented approach should
work, without the need of coercion.

Simple examples of how to 'sell' the idea of early, (external)
(cross-area) reviews:  Working Group chair training touts
it. ADs suggest it -- repeatedly.  Experienced participations suggest
it -- repeatedly.

I think there is also benefit in having a centralized index for
accessing reviews, so that the community can see what it happening in
that front.  It also serves to mark the possible snowball effect that
you cite.

MA>   But, in the absence of experience
MA> that concretely shows that some sort of early review function is, in
MA> fact, useful and worthwhile what is the incentive for a WG to invoke the
MA> review function?

If reviewing proposals were an odd or new idea then we might have to
worry about selling the basic concept.  But this is all pretty
standard stuff in the academic and engineering community.  Getting
fresh sets of expert eyes to look things over is not a controversial
concept and the benefits are well-established.

MA> The other thing I wonder about is new-ish folk in the IETF.  People
MA> bring ideas in that are not really baked, but they don't realize it (new
MA> UDP-based app with no congestion control, no thoughts given to security,
MA> etc.).  What is the incentive for these people to turn around and ask a
MA> bunch of people they don't know to pick apart their work when they are
MA> Obviously Correct?

The question is what is the incentive for a *group* to call for
reviews.  Individuals do not matter much, but the group does.  So the
review process serves to raise important questions about a spec -- or
provide confirmation of its constructs -- before there has been
long-term investment in the effort.  That's pretty strong incentive.

MA> SIRS didn't get much activity in the grand scheme of things.

I've commented on the tendency to dismiss the SIRS experience before.
Quick summary:

SIRS was started during summer and by mid-Fall there were rumors it
had failed. The negative effect this is likely to have had on
continued use of SIRS should not be underestimated.

Particularly given typical IETF timeframes, that anyone would declare
a strategic IETF experiment to have failed after such a short time
suggests deeper problems with either patience levels or agendas.

SIRS did virtually no "marketing" either proactively such as prodding
working groups to use its services, or passively, such as collecting
review data (statistics and source copies of reviews) in a single
place.

On the other hand, it got good signup by folks offering to do reviews
and it did get some review requests.

d/

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 Apr 2004 13:26:03 -0400

> benefits and, to some extent, expectations.

I'll buy this.  But, there is a fine line between developing culture and
developing rules, I think.  I.e., if we're relying on folk (especially
ADs) to cajole WGs into invoking the review function then we may be
basically creating de-facto rules.

> concept and the benefits are well-established.

I agree.

The downside is time.  People already think the IETF process is too
slow.  Adding another step may simply look like lengthening the
process.  Of course, the hope is that we're also lopping off a
significant chunk at the end.  But, that is the part that we do not yet
have concrete examples of.  If you could show data that said "early
reviewed docs took X months from -00 to RFC and non-early review docs
took Y months from -00 to RFC, where X << Y" then the case would be
clear and compelling.

> I've commented on the tendency to dismiss the SIRS experience before.

I was not insinuating that it was a failure.  We'd do things differently
if we had it to do over.  My point was that its existence did not
immediately yield people clamoring for reviews.  If getting a breadth of
early reviews was self-evidently good then I'd have expected more.

I do think I understand your answer to my original question, though.  I
think it boils down to having a big organizational push to try some
things (one thing, two things, whatever).

allman

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 Apr 2004 13:28:07 -0400

I agree.

If folks are interested I am happy to host some sort of an archive on
the ICAR web pages.  I am not offering anything fancy.  But, I am
offering *something* for whatever reviewing experiments we want to
undertake.

(Harald's General Area Review Team has an archive.)

allman

---

From: Dave Crocker <[email-masked]>
Date: Mon, 19 Apr 2004 11:04:05 -0700

Mark,

MA> I'll buy this.  But, there is a fine line between developing culture and
MA> developing rules, I think.  I.e., if we're relying on folk (especially
MA> ADs) to cajole WGs into invoking the review function then we may be
MA> basically creating de-facto rules.

In the long-run, yes of course.  However the difference in the
near-term is massive.  Near-term is a seductive appeal to benefits,
with voluntary adoption, rather than a coercive imposition from an
external authority.

MA> The downside is time.  People already think the IETF process is too
MA> slow.  Adding another step may simply look like lengthening the
MA> process.

Given current IETF timeframes, it is more likely that review processes
will reduce the amount of time needed, rather than increasingly it, by
virtue of early, basic improvements in the specification.

MA>  Of course, the hope is that we're also lopping off a
MA> significant chunk at the end.

right.  but more than that.  better focus of the working group.

>> I've commented on the tendency to dismiss the SIRS experience before.

MA> I was not insinuating that it was a failure.  We'd do things differently
MA> if we had it to do over.

I'm not so sure.  That would depend on the characteristics of the
process that one believes should be different.  I do not recall seeing
arguments against points of the SIRS structure or in favor of
alternative designs, that argue for superiority of results.

MA>   My point was that its existence did not
MA> immediately yield people clamoring for reviews.

Of course not.  This is a community this is currently responding to
crisis conditions by taking more than 2 years to make any meaningful
changes.

MA>   If getting a breadth of
MA> early reviews was self-evidently good then I'd have expected more.

You shouldn't have.

As I said, the problem is with the expectations.

d/

---

From: "Spencer Dawkins" <[email-masked]>
Date: Mon, 19 Apr 2004 20:37:18 -0500

From: "Dave Crocker" <[email-masked]>
To: "Mark Allman" <[email-masked]>

> alternative designs, that argue for superiority of results.

I may be incorrectly channeling Mark, but I took his comment about
doing things differently as having more to do with not punting after
four months than about changing SIR structure, etc.

Spencer

---

From: Mark Allman <[email-masked]>
Date: Mon, 19 Apr 2004 23:15:45 -0400

> > process that one believes should be different.  I do not recall
> > seeing arguments against points of the SIRS structure or in favor of

Yeah, Dave and I worked through this off-list and you're exactly right
Spencer.  I was not making a comment on the soundness of the SIRS
proposal, just on the actual experiment with it (with the major point
being that more experience would be better).  Thanks for making this
more clear Spencer.

allman