Subject: Draft review experiment

From: Kathleen Moriarty <[email-masked]>
Date: Wed, 25 May 2016 13:13:42 -0400

Hi,

At the IESG retreat, we were chatting about participation and draft
reviews.  Since many don't realize they should be reviewing drafts in
their own working group and others, we thought an easy to execute
experiment might help a little bit.  I do nudge people individually
and explain that if you want your drafts reviewed by others, you
should review other drafts as well.  A few of us doing that downs
scale.

If we add one slide to each WG meeting with this reminder (or for some
new information), the hope is that we would get some more reviews.

I put together this text, with slide attached.  Edit suggestions are
welcome, but let's keep the text short.

Thank you!

--------
Document Review Request

o Document quality relies on reviews, please review documents in your
working group and at least one other document from another working
group.

o If you’d like your documents reviewed, put the effort in to review
other documents
--------
-- 

Best regards,
Kathleen

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 25 May 2016 17:38:37 +0000

It seems like a fair observation.

> On May 25, 2016, at 10:13 AM, Kathleen Moriarty <[email-masked]> wrote:
> 
> Hi,
> 
> <DraftReviewSlide.pdf>

---

From: Melinda Shore <[email-masked]>
Date: Wed, 25 May 2016 09:49:05 -0800

I think there's a lot of value in highlighting this at
the beginning of each session.  That said, we do ask
people to commit to review documents and generally will
not put a document into working group last call without
thorough review from multiple participants (that is to
say, we don't treat silence as an indication that people
think a document's in good shape).  It's still the case
that there tends to be poor follow-through on reviews,
even among people who raised their hands at meetings or
otherwise committed to review a specific document.  So,
together with this slide, it might be worth thinking
about what the incentives are for reviews, and what
can be done to provide more motivation.

Melinda

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 26 May 2016 11:02:00 +1000

What would help me in this more than anything would be tools that help identify drafts that I *should* review. 

E.g., lots of drafts use HTTP, but it's hard to find them among the flood. Something that alerted me when a new draft references one of a set of RFCs that I nominate, for example, would be most helpful.

Cheers,

> On 26 May 2016, at 3:13 AM, Kathleen Moriarty <[email-masked]> wrote:
> 
> Hi,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Tomek Mrugalski <[email-masked]>
Date: Fri, 27 May 2016 13:20:34 +0200

On 26.05.2016 03:02, Mark Nottingham wrote:
> What would help me in this more than anything would be tools that
> help identify drafts that I *should* review.
> 
> E.g., lots of drafts use HTTP, but it's hard to find them among the
> flood. Something that alerted me when a new draft references one of a
> set of RFCs that I nominate, for example, would be most helpful.
You're not the first one with such requirements. DHC chairs use this
tool: https://github.com/tomaszmrugalski/ietf-tech-report
This is a slightly tweaked version. The original author is Alan DeKok.
You may try to ping him if there's newer version. I branched this one in
2013 and added -p option which allows you to catch individual drafts.

There's a text file that specifies RFC numbers and keywords that should
ring the bell. You may exclude specific WGs if you want to. The script
downloads all I-Ds, greps through them and then prints a list of drafts
that match your criteria. Once the script is done, you get a list of drafts.

Tomek

---

From: Daniel Migault <[email-masked]>
Date: Fri, 27 May 2016 12:52:52 -0400

Hi Kathleen,

That this is a good to remind this at WG sessions. I am fine with the
wording.
However, the second item seems to be restricted to authors of drafts - so
maybe excluding non draft authors. In addition, it seems also to state that
there is little interest for individual to review.

To encourage people that do not necessarily have a document to review like
new comers or people with limited involvement in the IETF maybe we should
maybe also add something that any review has value and should be
respectfully responded. In addition, reviewing a draft is a good way to get
people involved in the IETF and receive interesting feed backs. Some
mentoring initiatives are going into this direction and I see that as a
very good initiative.
In addition, to encourage people to review, maybe we should also provide
more visibility to the reviewers. I think this could be done be providing
guide lines on writing the acknowledgment section - if I recall properly
Brain did write some recommendations.

BR,
Daniel

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 27 May 2016 13:22:58 -0400

On 5/27/16 12:52 PM, Daniel Migault wrote:
> However, the second item seems to be restricted to authors of drafts -
> so maybe excluding non draft authors. In addition, it seems also to
> state that there is little interest for individual to review.
>
> To encourage people that do not necessarily have a document to review
> like new comers or people with limited involvement in the IETF maybe we
> should maybe also add something that any review has value and should be
> respectfully responded.

Yes! Reviews by people who have not participated in the development of a 
document are especially useful because they can identify things that are 
not written down but are "assumed context" by all those working on the 
document.

This is most helpful if the new reviewer falls within the intended 
audience for the document. Those who don't fall within the intended 
audience might not have the necessary prerequisite knowledge to 
understand it. But even reviews by those people can sometimes be helpful 
if they bring some useful perspective.

(Unfortunately we don't often explicitly identify the intended audience, 
or even actively think about it.)

> In addition, reviewing a draft is a good way to
> get people involved in the IETF and receive interesting feed backs. Some

In the IETF this is one of the best ways to become "known". If you get a 
reputation for giving good, insightful, comments then you are likely to 
be recognized and taken seriously on other matters.

	Thanks,
	Paul

> more visibility to the reviewers. I think this could be done be
> providing guide lines on writing the acknowledgment section - if I
> recall properly Brain did write some recommendations.
>
>
> BR,
> Daniel
>
> On Wed, May 25, 2016 at 1:13 PM, Kathleen Moriarty
> <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi,
>

---

From: [email-masked]
Date: Fri, 27 May 2016 13:32:46 -0400

Great feedback, thank you.  I'll work this into an updated version.

Kathleen 

Sent from my iPhone

> On May 27, 2016, at 12:52 PM, Daniel Migault <[email-masked]> wrote:
> 
> Hi Kathleen,
> 
> That this is a good to remind this at WG sessions. I am fine with the wording.
> However, the second item seems to be restricted to authors of drafts - so maybe excluding non draft authors. In addition, it seems also to state that there is little interest for individual to review. 
> 
> To encourage people that do not necessarily have a document to review like new comers or people with limited involvement in the IETF maybe we should maybe also add something that any review has value and should be respectfully responded. In addition, reviewing a draft is a good way to get people involved in the IETF and receive interesting feed backs. Some mentoring initiatives are going into this direction and I see that as a very good initiative. 
> In addition, to encourage people to review, maybe we should also provide more visibility to the reviewers. I think this could be done be providing guide lines on writing the acknowledgment section - if I recall properly Brain did write some recommendations. 
> 
> 
> BR, 
> Daniel
> 
>> On Wed, May 25, 2016 at 1:13 PM, Kathleen Moriarty <[email-masked]> wrote:
>> Hi,
>>

---

From: [email-masked]
Date: Fri, 27 May 2016 13:34:06 -0400

Sent from my iPhone

> On May 27, 2016, at 1:22 PM, Paul Kyzivat <[email-masked]> wrote:
> 
> Yes! Reviews by people who have not participated in the development of a document are especially useful because they can identify things that are not written down but are "assumed context" by all those working on the document.
> 
> This is most helpful if the new reviewer falls within the intended audience for the document. Those who don't fall within the intended audience might not have the necessary prerequisite knowledge to understand it. But even reviews by those people can sometimes be helpful if they bring some useful perspective.
> 
> (Unfortunately we don't often explicitly identify the intended audience, or even actively think about it.)
> 
> In the IETF this is one of the best ways to become "known". If you get a reputation for giving good, insightful, comments then you are likely to be recognized and taken seriously on other matters.

That might be good talking points when the slide is up.

Thanks,
Kathleen

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 28 May 2016 15:33:53 +1000

Thanks, will give it a try!

> On 27 May 2016, at 9:20 PM, Tomek Mrugalski <[email-masked]> wrote:
>

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 31 May 2016 21:45:45 +0000

Tomek, 

The tool: https://github.com/tomaszmrugalski/ietf-tech-report 
needs a local copy of all of the internet drafts. 

That is not easy to use. Possible to have a web portal for people to enter the key fields, and sent periodical messages to alert people of the relevant drafts?

Linda

-----Original Message-----
From: WGChairs [mailto:[email-masked]] On Behalf Of Tomek Mrugalski
Sent: Friday, May 27, 2016 6:21 AM
To: [email-masked]
Subject: Re: Draft review experiment

There's a text file that specifies RFC numbers and keywords that should ring the bell. You may exclude specific WGs if you want to. The script downloads all I-Ds, greps through them and then prints a list of drafts that match your criteria. Once the script is done, you get a list of drafts.

Tomek

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 31 May 2016 21:50:09 +0000

I agree with Melinda to give "incentives for reviews". Here are a couple of incentives to consider:

- People who don't provide review to other drafts can't get their own documents adopted to WG drafts.
- In IEEE802, all people who provided comments are listed in the contributor section. Maybe IETF can consider this approach. 

Thanks, Linda 

-----Original Message-----
From: WGChairs [mailto:[email-masked]] On Behalf Of Melinda Shore
Sent: Wednesday, May 25, 2016 12:49 PM
To: [email-masked]
Subject: Re: Draft review experiment

I think there's a lot of value in highlighting this at the beginning of each session.  That said, we do ask people to commit to review documents and generally will not put a document into working group last call without thorough review from multiple participants (that is to say, we don't treat silence as an indication that people think a document's in good shape).  It's still the case that there tends to be poor follow-through on reviews, even among people who raised their hands at meetings or otherwise committed to review a specific document.  So, together with this slide, it might be worth thinking about what the incentives are for reviews, and what can be done to provide more motivation.

Melinda

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Tue, 31 May 2016 22:00:04 +0000

> On May 31, 2016, at 5:50 PM, Linda Dunbar <[email-masked]> wrote:
> 

I normally do this for substantive comments on drafts on which I am the editor. I even include comments beyond nits from IESG members. 

Thanks,
Acee

---

From: Daniele Ceccarelli <[email-masked]>
Date: Wed, 1 Jun 2016 05:35:34 +0000

The second one (which in many cases is a common practice) could be a good incentive, while I see the first one more like a threat...and some people don't react well to threats.

The review is a contribution to the document but it could be a bit risky because I see the contributor as someone that in a way or another gives his/her support to the draft because is useful, because there is willingness to do some work around that. The reviewer on the other side is someone that helps improving the quality of the document, readability, but that probably doesn't care about the topic at all. Maybe an ACK would be more appropriate...but then the incentive becomes significantly lower.
One possibility could be to add a sections like "formally reviewed by" ? I'm not sure this could make much sense but I wanted to put the idea on the table.

BR
Daniele  

> From: WGChairs [mailto:[email-masked]] On Behalf Of Acee
> Lindem (acee)
> Sent: mercoledì 1 giugno 2016 00:00
> To: Linda Dunbar <[email-masked]>
> Cc: [email-masked]
> > - In IEEE802, all people who provided comments are listed in the
> contributor section. Maybe IETF can consider this approach.
> 
> I normally do this for substantive comments on drafts on which I am the
> editor. I even include comments beyond nits from IESG members.
> 
> Thanks,
> Acee
> 
> 
> >
> > Thanks, Linda
> >
> > I think there's a lot of value in highlighting this at the beginning of each
> session.  That said, we do ask people to commit to review documents and
> generally will not put a document into working group last call without
> thorough review from multiple participants (that is to say, we don't treat
> silence as an indication that people think a document's in good shape).  It's
> still the case that there tends to be poor follow-through on reviews, even
> among people who raised their hands at meetings or otherwise committed
> to review a specific document.  So, together with this slide, it might be worth
> thinking about what the incentives are for reviews, and what can be done to
> provide more motivation.
> >
> > Melinda
> >