Subject: [Gen-art] Timing of reviews: conclusion

From: Jari Arkko <[email-masked]>
Date: Wed, 10 Jul 2013 19:53:26 +0300

I wanted to conclude our earlier discussion on the timing of directorate reviews. Overall I read the thread as cautiously positive for experimenting if this helps or not. But we need to take care of a multiple issues:

- Figure out the best way for the reviewers to be able to send mail to WG lists
- Limit the time/scope of the experiment in order to find out what the real effect is on, e.g., reviewer workload
- Determine whether to review at WG time, IETF last call time, and IESG review time. Right now we are doing the last two.
- Determine how to start the review, and what the timing is.
- Carefully select review timing and documents so that they are ready for an external review
- Shield reviewers from being shouted down by a large group of WG participants

The suggestions for resolving these items is:

- We can start the effort by having the WG chairs able to ask for this service. The team can then throttle the requests to a level they see they can handle.
- Start the experiment now and review in six months, determine success based on interviewing participating reviewers and WG chairs
- Perform reviews after the WGLC (in parallel with AD reviews)
- Review documents while they are in the working group, and attempt to have the same person do a review of the new version that is entering the IESG telechat (similar to what we are doing now, but replacing IETF last call review with WGLC review)
- Require working group chairs to mediate and monitor discussions between the reviewer and the working group
- Whitelisting might be a feasible solution for making the reviewers able to post to the WG list. Or is there something else that we could do? Rely on the WG chairs approving posts?

Comments?

Jari

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 11 Jul 2013 08:12:00 +1200

Hi Jari,

Personally, I'm willing to try this. I was just wondering what our
"standard" template would be for this. Something like this perhaps:

The WG Chairs requested a Gen-ART review for this draft. For background on
Gen-ART, please see the FAQ at
<http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Please resolve these comments along with any Area Director comments
you may receive.

Document: draft-ietf-foobar...
Reviewer: Brian Carpenter
Review Date: 2013-07-10
IETF LC End Date: TBD
IESG Telechat date: TBD

Summary:  Needs work...

Also: we would need to update the GenART FAQ slightly. The answer
to "Q: Why is my document being reviewed?" needs changing.

Regards
   Brian

On 11/07/2013 04:53, Jari Arkko wrote:
> _______________________________________________
> Gen-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/gen-art
>

---

From: Jari Arkko <[email-masked]>
Date: Wed, 10 Jul 2013 23:23:24 +0300

Brian:

Good points. And I like the template. Thanks.

Jari

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Thu, 11 Jul 2013 09:11:00 +0000

I am also in favor of the experiment and willing to participate. Besides the changes pointed to by Brian we may also need to make some adjustments to the Gen-ART tools (new column in the table at http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue to mark the WG review due date) and the process used by the secretary (separate mail for WG early review?). 

Regards,

Dan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Jari Arkko
> Sent: Wednesday, July 10, 2013 7:53 PM
> To: <[email-masked]> Team
> Subject: [Gen-art] Timing of reviews: conclusion
> 
> I wanted to conclude our earlier discussion on the timing of directorate
> reviews. Overall I read the thread as cautiously positive for
> experimenting if this helps or not. But we need to take care of a
> multiple issues:
> 
> - Limit the time/scope of the experiment in order to find out what the
> real effect is on, e.g., reviewer workload
> - Determine whether to review at WG time, IETF last call time, and IESG
> review time. Right now we are doing the last two.
> - We can start the effort by having the WG chairs able to ask for this
> service. The team can then throttle the requests to a level they see
> they can handle.
> - Start the experiment now and review in six months, determine success
> based on interviewing participating reviewers and WG chairs
> - Review documents while they are in the working group, and attempt to
> have the same person do a review of the new version that is entering the
> IESG telechat (similar to what we are doing now, but replacing IETF last
> call review with WGLC review)
> - Require working group chairs to mediate and monitor discussions
> between the reviewer and the working group
> - Whitelisting might be a feasible solution for making the reviewers
> able to post to the WG list. Or is there something else that we could
> do? Rely on the WG chairs approving posts?
> 
> Comments?
> 
> Jari
>

---

From: Jari Arkko <[email-masked]>
Date: Thu, 11 Jul 2013 18:23:11 +0300

> I am also in favor of the experiment and willing to participate. Besides the changes pointed to by Brian we may also need to make some adjustments to the Gen-ART tools (new column in the table at h…

That's a good point. Thanks.

Jari

---

From: Robert Sparks <[email-masked]>
Date: Thu, 11 Jul 2013 13:46:27 -0500

I would like start discussing a more general way to track reviews when 
they are needed. Dan's new-column tweak to the existing tool makes me a 
little nervous in the long run - the column won't apply to all 
documents, and other types will come in making new columns leading to a 
painfully sparse matrix.

Rather, I would like to work towards a system where we can request 
reviews on a document at any time, for any reason.

Instead of scraping last calls to identify documents that need review, 
we can make the parts of the system that issue last calls automatically 
make a review request, and that request can tailor the data we need to 
track for that particular review (like any constraining dates such as WG 
LC or IETF LC completion).

RjS

---

From: "A. Jean Mahoney" <[email-masked]>
Date: Thu, 11 Jul 2013 13:49:01 -0500

I'm also willing to help with the experiment, but I suggest that the 
review tool be updated later, after we have a better idea of the new 
requirements. Updating the review tool would probably require updating 
datatracker also.

I can easily capture early review assignments in the spreadsheets for now.

Jean

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Jul 2013 08:03:57 +1200

> requirements. Updating the review tool would probably require updating

Yes. The following is not a complaint, since the review tool is a
much-appreciated volunteer effort, but IMHO it would be better
if we submitted our reviews via the review tool, which could then
send them on as email automatically. At the moment I usually forget
to enter the review URL into the tool until some days or weeks later!

(People who've used EDAS know what I mean.)

It would also make sense for reviewers to be remembered by the tracker,
so that they can get status updates automatically.

Since the tracker already knows who is interested in each draft,
integrating the two would make a lot of sense.

   Brian

>>> I am also in favor of the experiment and willing to participate.
>>> Besides the changes pointed to by Brian we may also need to make some
>>> adjustments to the Gen-ART tools (new column in the table at
>>> http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue
>>> to mark the WG review due date) and the process used by the secretary
>>> (separate mail for WG early review?).

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Fri, 12 Jul 2013 09:35:05 +0000

+1

> Behalf Of Brian E Carpenter
> Sent: Thursday, July 11, 2013 11:04 PM
> To: A. Jean Mahoney
> Cc: [email-masked]
> Subject: Re: [Gen-art] Timing of reviews: conclusion
> 
> Yes. The following is not a complaint, since the review tool is a much-
> appreciated volunteer effort, but IMHO it would be better if we
> submitted our reviews via the review tool, which could then send them on
> as email automatically. At the moment I usually forget to enter the
> review URL into the tool until some days or weeks later!
> 
> >>> Besides the changes pointed to by Brian we may also need to make
> >>> some adjustments to the Gen-ART tools (new column in the table at
> >>> http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueu
> >>> e to mark the WG review due date) and the process used by the
> >>> secretary (separate mail for WG early review?).

---

From: Martin Thomson <[email-masked]>
Date: Fri, 12 Jul 2013 09:14:08 -0700

On 11 July 2013 13:03, Brian E Carpenter <[email-masked]> wrote:
> but IMHO it would be better
> send them on as email automatically.

I have all the same problems as Brian.  And this might also make
access control easier, such that reviews could be sent to WG lists
without requiring sign-up (though on reflection, that's a little
optimistic).