Subject: [Gen-art] Fwd: Re: tail-heavy process and early directorate reviews

From: "A. Jean Mahoney" <[email-masked]>
Date: Sat, 27 Jul 2013 11:10:51 -0500

Hi all,

We are moving forward with the early review experiment. For Gen-ART, an 
early review will replace the LC review, so a reviewer that is assigned 
an early review will perform the early review and the telechat review.

I will be monitoring the dir-coord mailing list and assigning reviews as 
necessary.

Check the email below for details.

Thanks!

Jean

-------- Original Message --------
Subject: 	Re: tail-heavy process and early directorate reviews
Date: 	Fri, 26 Jul 2013 20:35:03 +0300
From: 	Jari Arkko <[email-masked]>
To: 	Barry Leiba <[email-masked]>, Pete Resnick 
<[email-masked]>, Stephen Farrell <[email-masked]>, Sean 
Turner <[email-masked]>, joel jaeggli <[email-masked]>, Benoit 
Claise <[email-masked]>, "A. Jean Mahoney" <[email-masked]>, 
Tero Kivinen <[email-masked]>, Claudio Allocchio 
<[email-masked]>

Ok. So by now you should have seen that you've become a member of the 
dir-coord list (we changed the name). We've left TSV and OPS directorate 
coordinators out for now.

So here's the plan on how this will work. Please comment.

Internet Drafts sent for approval as RFCs are reviewed by individuals 
during the IETF Last Call, the Area Directors, IANA, as well as a number 
of volunteers in various directorates and review teams. The reviews from 
these teams has gained a significant role in ensuring that the IETF 
produces high-quality, understandable and implementable RFCs.

Yet, as discussed in 
http://www.ietf.org/blog/2013/05/balancing-the-process/ we have a 
general problem that quite a lot of the work around IETF documents 
happens at the end of the process. In particular, a number of the 
reviews during IETF last call point out issues that end up being raised 
by the IESG as comments when deciding for the final approval of the 
document. It is of course good that issues are caught, but raising them 
earlier would be better. And it would be better if the working groups - 
the intended focus point of work on a topic - would get to handle them, 
as opposed to raising these issues with the IESG. The IESG discussed 
these issues in its May 2013 retreat, and decided to experiment with 
three actions designed to move more work to the responsibility of the 
working groups:

(1) Perform some reviews that are now happening at IETF Last Call a bit 
earlier.
(2) Invite document shepherds on IESG telechats when there's a document 
that is likely to require discussion.
(3) When a document has a number of issues, hand over the process back 
to the working group, as opposed to the IESG tracking the issues. Among 
other things, this will ensure that changes are discussed in an open 
working group list and agreed through consensus.

Building quality and cross-area review to the process earlier is of 
course a big effort. For the first item above, we are launching an 
experiment to make a small change to current directorate review 
procedures to learn if we can move reviews a little bit earlier. If 
successful, this experiment will enable working groups to deal with 
issues before IETF Last Call and IESG review and empower the working 
groups to be in charge of the documents throughout their life cycle. We 
are also hoping that document quality will improve and number of issues 
discussed in the IESG will be lower.

While the number of reviews as such is not changed, some additional 
effort and care will however be required from the reviewers, directorate 
coordinators/secretaries, the working group chairs, and other 
participants. The experiment will show us whether this effort is 
reasonable and if there are any unexpected effects. The experiment is 
performed on a voluntary basis by each directorate. As early review 
requests come in, the directorates can throttle workload by either 
processing the requests or reverting to the existing procedures. 
Initially, the Gen-ART, Security Directorate, and Applications Area 
Directorate are included in this effort. Other directorates may be added 
later. The experiment will be reviewed after six months.

Care must be taken to avoid a number of possible drawbacks. There may be 
problematic documents that would require much re-review and effort. 
Similarly, working groups often perform a number of working group last 
calls on a document, and it would be undesirable to engage the reviewer 
before the document was really ready to be sent forward. And when 
reviewers send comments, it is important that the group listens to the 
comments in the right way, like you would for a review from a security 
expert or a general networking expert. E-mail practices around sending 
and responding to comments have to be carefully managed, as the outside 
reviewers are typically not list members.

For the *working group chairs*, you can request a review by sending mail 
to [email-masked] <mailto:[email-masked]>, indicating the name 
of the document. You can ask for a review as soon as a working group 
last call has ended successfully and the document edits are done. 
Initially, it would make sense to ask for reviews on documents that you 
feel are likely the most stable ones. To avoid congesting the reviewer 
resources, ask for this service only for some documents, not as a 
wholesale service on every document you submit for publication.

You can ask for the review in parallel with ongoing AD reviews, filling 
out the shepherd questionnaire, etc. The hope is that reviews can come 
in in the same time frame as other tasks in this stage, and that you can 
take the comments into account and revise the document before finally 
sending it out for IETF Last Call. Reviews will be sent to you and 
potentially the working group mailing list. You may need to forward 
comments and/or approve new posters to the list. Discussion on the 
mailing list is encouraged, but please try to ensure that the reviewer 
is kept on the Cc line, as he or she may not be on the list itself. 
Treat the reviews with respect and keep it in mind that outside experts 
may have opinions that need to be taken into account, even if the 
working group had not considered those aspects before. Mediate and 
monitor the discussion actively. Try to keep the reviewer out of e-mail 
storms and work out solutions separately and then engage the reviewer again.

For the *directorate coordinators/secretaries*, please monitor the 
[email-masked] <mailto:[email-masked]>. If there is a request 
for a review, dispatch the task to a reviewer. Managing overload is your 
task. Please acknowledge requests and whether you can accommodate them. 
Directorates that today review documents twice, at IETF Last Call and 
then before entering the IESG telechat should continue this practice by 
doing the early review and then the IESG telechat review. Existing 
practices such as retaining the same reviewer for checking the same 
version is useful. Your review tools and practices may need adjustment 
to accommodate reviews happening at different times.

For the *reviewers*, take into account the context. Post your reviews to 
the working group chairs, authors, and Cc the working group mailing list 
if necessary. If your review team uses a template for these e-mails, 
some changes may be necessary in the template for the early review. Some 
of those changes have been discussed, e.g., in the Gen-ART list.

For everyone, please collect your experiences so that in six months we 
can evaluate how you liked the experience. And thank you for your 
participation in this effort!

Jari