Subject: PROTO Process

From: Russ Housley <[email-masked]>
Date: Tue, 12 May 2009 03:12:04 -0400

During the IESG Retreat, they was a discussion of the effectiveness 
of the PROTO process.  Our view is that the document write-up is the 
only part of the envisioned process that is really providing 
benefit.  We are seeking your input on the situation.  Can the 
community get the envisioned benefits?  What changes are necessary to get them?

On behalf of the IESG,
   Russ Housley

---

From: Jari Arkko <[email-masked]>
Date: Tue, 12 May 2009 12:14:28 +0200

I wanted to expand a little bit on what Russ was saying.

In my experience, the part of PROTO that is currently working is the 
write-ups. When it comes to the shepherding of the document through 
issues raised in last call, directorate, and IESG reviews, the authors 
and the responsible/discussing ADs seem to be most active in responding, 
explaining, suggesting text changes, etc. There are notable exceptions 
-- you know who you are -- but in many cases the shepherds are 
relatively silent. Is this a correct perception of the reality? And if 
it is, is it a desirable state of affairs or something that should be 
changed?

Jari

---

From: Tom Taylor <[email-masked]>
Date: Tue, 12 May 2009 12:32:43 +0200

I occasionally chide myself for not helping a discussion along. I simply haven't 
gotten into the habit of taking an active role. A complication in AVT's case is 
that we have so many documents, I forget and don't get around to checking 
whether I am or my co-chair is shepherd for a particular document.

---

From: "Bert Wijnen \(IETF\)" <[email-masked]>
Date: Tue, 12 May 2009 12:36:28 +0200

So Jari, do you see it a failure if authors/editors interact with the ADs and the
people who raise issues concerns? As far as I am concerned, when that 
happens in a timely fashion, then I am happy when shepherding.
If it does not happen quick enough, then I try to stimulate interaction.
When shepherding, I also keep an eye on the discussion to try and
determine if the resulting edits are changing what the WG had consensus
on, and in that case I would steer that discussion to the WG to make
sure that the WG is in agreement.

Don't overload the shepherd with things that seem to be working.

Bert
  ----- Original Message ----- 
  From: Jari Arkko 
  To: Russ Housley 
  Cc: [email-masked] 
  Sent: Tuesday, May 12, 2009 12:14 PM
  Subject: Re: PROTO Process

  Jari

---

From: Henk Uijterwaal <[email-masked]>
Date: Tue, 12 May 2009 12:44:43 +0200

Jari,

> write-ups.

Perhaps, though for the 6 to 10 notes that I've written, I cannot recall
any question or feedback on them.

> When it comes to the shepherding of the document through 

Most, if not all, issues that I see raised are detailed questions on
the contents of the document where the authors are probably in the
best position to answer them.  These questions are sent to the authors
directly, I don't see what the sheperd can add here other than monitoring
that the authors respond within a reasonable timeframe.

Henk

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Belgium: an unsolvable problem, discussed in endless meetings, with no
          hope for a solution, where everybody still lives happily.

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Tue, 12 May 2009 12:50:30 +0200

In my case it is a perceived inability to have any impact at this point in the process.

1)	In a number of cases the AD requests changes during the AD evaluation stage and the shepherd does not get informed of that. Perhaps you should do a survey of the number of instances were a "revised version required" is recorded

2)	Many of the issues do not get recorded in the drafts tracker tool, and the shepherds cannot actually update the tracker, so they are unable to use this as a tool to help them in the work.

Perhaps you should do a survey of the number of instances were a "revised version required" is recorded in the drafts tracker, against where tracker provides sufficient information to track down why a new version is needed. In any case, in an open process this information should be recorded and available to all.

regards

Keith 

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Jari Arkko
> Sent: Tuesday, May 12, 2009 11:14 AM
> To: Russ Housley
> Cc: [email-masked]
> In my experience, the part of PROTO that is currently working 
> is the write-ups. When it comes to the shepherding of the 
> document through issues raised in last call, directorate, and 
> IESG reviews, the authors and the responsible/discussing ADs 
> seem to be most active in responding, explaining, suggesting 
> text changes, etc. There are notable exceptions
> -- you know who you are -- but in many cases the shepherds 
> are relatively silent. Is this a correct perception of the 
> reality? And if it is, is it a desirable state of affairs or 
> something that should be changed?
> 
> Jari
> 
>

---

From: Ross Callon <[email-masked]>
Date: Tue, 12 May 2009 08:03:42 -0400

As an AD, I agree that the PROTO write-ups have been quite useful, for
example when filling out the ballot, during AD review of the document,
as well as during IESG discussions. 

I see quite a few cases where WG chairs are helpful and participate in
the discussions helping to resolve issues (along with the authors,
discussing ADs, and shepherding ADs). In some cases the WG chairs have
been helpful when the authors have been hard to get to pay attention to
the document (eg, in a few cases finding an alternate editor to finish
the work when the original author had retired or otherwise couldn't get
to it).  

I can't recall cases where the document shepherd has been helpful where
the shepherd was neither author nor WG chair. I can't say whether this
was because I haven't noticed any cases where the PROTO shepherd was
anyone other than a WG chair, or because we forgot to CC the PROTO
shepherd on the emails, or for some other reason. 

Ross 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jari Arkko
Sent: 12 May 2009 06:14
To: Russ Housley
Cc: [email-masked]
Subject: Re: PROTO Process

Jari

---

From: "David Harrington" <[email-masked]>
Date: Tue, 12 May 2009 08:43:12 -0400

Hi,

Maybe for the IESG, the writeups are the most important and useful
aspect. As an author and chair, I think the PROTO process is working
just fine.

As an author, I expect to deal directly with ADs and reviewers who
make comments or DISCUSSes. I think that works just fine most of the
time.

However, sometimes it does not work that well if the author thinks the
change should not be made, and a DISCUSS insists it must. Sometimes, a
WG totally overlooks some aspect, such as congestion control or
operational issues, and the IESG rightfully thinks the aspect should
be addressed. Sometimes the WG does not want to deal with delay that
would occur from addressing the issue. Sometimes addressing the aspect
can have significant effect on the protocol design. The shepherd can
help to determine whether WG pushback is to avoid doing the extra
work, or because there is a real technical reason for not doing the
change. The shepherd can help to ensure that the WG is kept in the
loop on the proposed changes, and not just the IESG and the authors.
As an editor, I like knowing the shepherd (and chairs) are there to
make sure all the parties have adequate opportunities for input to the
process. As a chair, I know it can be easy to be driven to get the
milestones completed. Having a shepherd (even if it is one of the
chairs) means that somebody has the explicit job of ensuring fairness
and transparency during the approval process. I appreciate that the
shpeherd is there for that.

IESG members typically get involved in the process very late, and do
not always understand what debates have already occurred in the WG and
were settled only by rough consensus. Those who "lost" the debate can
raise the same issues during IETF last call or IESG review to try once
again to force their viewpoint through. If the requested change is not
consistent with WG rough consensus, then the shepherd can serve as the
arbiter. The shepherd can lead the discussion to understand the
viewpoints and the history of the two sides, and to help educate the
IESG about the history of the issue. Of course the WG chairs also
serve that function, and the shepherd should certainly call upon the
chairs as needed. As an editor, I appreciate knowing that such
arbiters are available to help settle disputes that might arise
(again) during IESG reviews.

By the time many documents reach the IESG, the chairs and authors are
often tired of the document. They just want it done. And they can be
frustrated by delays late in the process. The shepherd helps to keep
driving the approval process forward, making sure it is clear what and
who are in the critical path - whose court is the ball in? What needs
to be done to keep things moving. I appreciate knowing that the
shepherd is doing this job. I do think this is most helpful when the
shepherd is not one of the WG chairs, so they can help to motivate the
chairs to get things done on a timely basis.

So, personally, as an author and chair, I think the PROTO process is
working effectively as designed. I think it may offer more visible
benefits to authors and chairs than to IESG members.

David Harrington
[email-masked]
[email-masked]
[email-masked]

> Sent: Tuesday, May 12, 2009 6:14 AM
> To: Russ Housley
> Cc: [email-masked]

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 12 May 2009 14:57:07 +0200

The issues raised by David are exactly the added value that I would
expect from the PROTO shepherds (WG chairs or not) beyond the write-up.
The issue is that this happens so rarely. The question is why.

Dan

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of David Harrington
> Sent: Tuesday, May 12, 2009 3:43 PM
> To: 'Jari Arkko'; 'Russ Housley'
> Cc: [email-masked]
> Subject: RE: PROTO Process
> 
> Hi,
> 
> Maybe for the IESG, the writeups are the most important and 
> useful aspect. As an author and chair, I think the PROTO 
> process is working just fine.
> 
> As an author, I expect to deal directly with ADs and 
> reviewers who make comments or DISCUSSes. I think that works 
> just fine most of the time.
> 
> However, sometimes it does not work that well if the author 
> thinks the change should not be made, and a DISCUSS insists 
> it must. Sometimes, a WG totally overlooks some aspect, such 
> as congestion control or operational issues, and the IESG 
> rightfully thinks the aspect should be addressed. Sometimes 
> the WG does not want to deal with delay that would occur from 
> addressing the issue. Sometimes addressing the aspect can 
> have significant effect on the protocol design. The shepherd 
> can help to determine whether WG pushback is to avoid doing 
> the extra work, or because there is a real technical reason 
> for not doing the change. The shepherd can help to ensure 
> that the WG is kept in the loop on the proposed changes, and 
> not just the IESG and the authors.
> As an editor, I like knowing the shepherd (and chairs) are 
> there to make sure all the parties have adequate 
> opportunities for input to the process. As a chair, I know it 
> can be easy to be driven to get the milestones completed. 
> Having a shepherd (even if it is one of the
> chairs) means that somebody has the explicit job of ensuring 
> fairness and transparency during the approval process. I 
> appreciate that the shpeherd is there for that.
> 
> IESG members typically get involved in the process very late, 
> and do not always understand what debates have already 
> occurred in the WG and were settled only by rough consensus. 
> Those who "lost" the debate can raise the same issues during 
> IETF last call or IESG review to try once again to force 
> their viewpoint through. If the requested change is not 
> consistent with WG rough consensus, then the shepherd can 
> serve as the arbiter. The shepherd can lead the discussion to 
> understand the viewpoints and the history of the two sides, 
> and to help educate the IESG about the history of the issue. 
> Of course the WG chairs also serve that function, and the 
> shepherd should certainly call upon the chairs as needed. As 
> an editor, I appreciate knowing that such arbiters are 
> available to help settle disputes that might arise
> By the time many documents reach the IESG, the chairs and 
> authors are often tired of the document. They just want it 
> done. And they can be frustrated by delays late in the 
> process. The shepherd helps to keep driving the approval 
> process forward, making sure it is clear what and who are in 
> the critical path - whose court is the ball in? What needs to 
> be done to keep things moving. I appreciate knowing that the 
> shepherd is doing this job. I do think this is most helpful 
> when the shepherd is not one of the WG chairs, so they can 
> help to motivate the chairs to get things done on a timely basis.
> 
> So, personally, as an author and chair, I think the PROTO 
> process is working effectively as designed. I think it may 
> offer more visible benefits to authors and chairs than to 
> IESG members.
> 
> David Harrington
> [email-masked]
> [email-masked]
> [email-masked]
> 
> 
> > and the responsible/discussing ADs seem to be most active in 
> > responding, explaining, suggesting text changes, etc. There are 
> > notable exceptions

---

From: "David Harrington" <[email-masked]>
Date: Tue, 12 May 2009 09:28:42 -0400

Hi,

I think the why is simple - if the normal process works adequately,
the shepherd has very little work to do other than monitor the process
for problems.

If DISCUSSes are documented and communicated on a timely basis to the
authors and chairs, then the shpeherd has no work to do there.

If the authors respond quickly and effectively to the DISCUSSes and
COMMENTs, then the shpeherd has no work to do there.

If the IESG and the authors/chairs reach satsifactory agreements on
requested changes, in a timely manner, then the shepherd has no work
to do. 

I would be more concerned if these "issues" did not happen rarely
(where "issues" and "rarely" depends on your interpretation). 

As co-chair, with co-chairs acting as shepherds, we recently had five
documents go through the approval process. We did run into issues when
(multiple) DISCUSSes were at odds with the WG consensus. The
chairs/shepherds worked with the ADs and the WG to resolve the
problems and get the documents modified on a timely basis and get them
approved (well, we have one still in process and the shepherd (me) is
being somewhat ineffective). It probably added an extra year of work
addressing the DISCUSSes, but the ADs, the chairs/shepherds, the
authors, and the WG together kept things moving forward. The shepherd
jumped in as needed, but to a large degree, while it was very
frustrating dealing with the amount of changes and constant delays,
the process was a great success because everybody helped keep things
moving.

As author, I have recently had three documents go through the process.
They drew multiple DISCUSSes, some new and some of the
here-we-go-again variety, and we worked them out quickly and, because
the process worked well, we didn't really need shepherd intervention.
I did look to see who the shepherd was at one point because I thought
we might need an arbiter. The IESG, the chair/shepherd, the authors,
and the WG worked together to resolve all the issues on a timely
basis. Again a nice successful process that did not require shepherd
intervention.

As contributor, I recently participated in documents going through the
approval process, and there were some issues raised, and the
(non-chair) shepherd was instrumental in keeping things moving forward
when they might have gotten sidetracked. Another process success.

So I'm thrilled that we do not see more issues that require a shepherd
to resolve. And when we have in my experience, the shepherds have
worked well.

I don't understand why the IESG thinks PROTO has problems. It seems to
me to be working well. I think this is a desirable state of affairs.

David Harrington
[email-masked]
[email-masked]
[email-masked]

> From: Romascanu, Dan (Dan) [mailto:[email-masked]] 
> Sent: Tuesday, May 12, 2009 8:57 AM
> To: David Harrington; Jari Arkko; Russ Housley
> Cc: [email-masked]
> > > it is, is it a desirable state of affairs or something that
should
> > be

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Tue, 12 May 2009 15:40:21 +0200

> From: David Harrington [mailto:[email-masked]] 

> 
> I don't understand why the IESG thinks PROTO has problems. It 
> seems to me to be working well. I think this is a desirable 
> state of affairs.
> 
> David Harrington
> [email-masked]
> [email-masked]
> [email-masked]
> 
> 

Let me answer shortly from the IESG member perspective. I need to
mention however that I was not the one who raised the issue originally.
I have seldom seen the shepherds playing that role that you mentioned of
bridging between the WGs and the IESG members and filling in missing
information. On the contrary, I have seen a number of documents in limbo
after the IESG entered some DISCUSSes and not making visible progress to
work the differences and issue revised ID's. When they did make progress
(late in some cases) it seemed to be more because of the AD's sending
reminders or in some cases the WG chairs (different that shepherds)
intervening. 

Dan

---

From: "Mary Barnes" <[email-masked]>
Date: Tue, 12 May 2009 08:51:12 -0500

I agree with Bert and I follow a very similar model as shepherd.  The
gap I see in the process is that the actual shepherd doesn't typically
appear on the IESG telechat agendas - i.e., chairs typically do the
PROTO write-up and then for my WG, the AD generally takes the lead from
there, so there is definitely the potential for the shepherds to get
left out of the loop. And, this lack of identification of the shepherd
applies to the majority of the documents on the IESG agendas.  However,
most folks are conscientious enough to copy WG chairs, who should
provide the info to the Shepherd, if it's someone other than the chairs.

I do ping the ADs as necessary to determine the status.  As Keith
suggests, there can be a loss of information once a doc leaves a WG and
often the only way to get the information is to ping the AD.   I would
certainly be willing to remain more actively involved if necessary and
like Bert do get involved when necessary. In particular, I consistently
ping authors to respond, making suggestions and providing guidance as
appropriate -  e.g., doc needs to come back to WG for re-reveiw, etc.
In my view, this model is quite effective as it helps editors/authors
that may not be as familiar with the process have a better understanding
of how their draft becomes an RFC, it removes an unnecessary
interaction/layer and is more transparent.   Also, as a document editor,
I see no problems with the shepherd being involved on an "as needed"
basis in resolving IESG concerns.  It might be useful to query document
editors (community in general) to see how this model works for them and
whether they feel the WG chair and/or shepherd does enough in helping
progress the documents. 
 
Mary. 

________________________________

From: [email-masked] [mailto:[email-masked]] On
Behalf Of Bert Wijnen (IETF)
Sent: Tuesday, May 12, 2009 5:36 AM
To: Jari Arkko; Russ Housley
Cc: [email-masked]
Subject: Re: PROTO Process

So Jari, do you see it a failure if authors/editors interact with the
ADs and the

Bert

	----- Original Message ----- 
	From: Jari Arkko <mailto:[email-masked]>  
	To: Russ Housley <mailto:[email-masked]>  
	Cc: [email-masked] 
	Sent: Tuesday, May 12, 2009 12:14 PM
	Subject: Re: PROTO Process

	In my experience, the part of PROTO that is currently working is
the 
	write-ups. When it comes to the shepherding of the document
through 
	issues raised in last call, directorate, and IESG reviews, the
authors 
	and the responsible/discussing ADs seem to be most active in
responding, 
	explaining, suggesting text changes, etc. There are notable
exceptions 
	-- you know who you are -- but in many cases the shepherds are 
	relatively silent. Is this a correct perception of the reality?
And if 
	it is, is it a desirable state of affairs or something that
should be 
	changed?
	
	Jari

---

From: "David Harrington" <[email-masked]>
Date: Tue, 12 May 2009 09:53:28 -0400

OK. 

I will observe that for both my chair and author experiences, Pasi was
the responsible AD, and he is a wonderul shepherd himself ;-)

dbh

> Sent: Tuesday, May 12, 2009 9:40 AM

---

From: Jari Arkko <[email-masked]>
Date: Tue, 12 May 2009 15:55:11 +0200

Mary,

> PROTO write-up and then for my WG, the AD generally takes the lead 
> from there, so there is definitely the potential for the shepherds to 
> get left out of the loop. And, this lack of identification of the 
> shepherd applies to the majority of the documents on the IESG 
> agendas.  However, most folks are conscientious enough to copy WG 
> chairs, who should provide the info to the Shepherd, if it's someone 
> other than the chairs. 
>

The tracker fills in the e-mail address for the WG chairs automatically 
in the e-mail notification field. This works well as long as the 
shepherd is a WG chair. As a result, we typically no longer label agenda 
entries with the shepherd's name. It is still recorded in the writeup 
(typically the first entry in the tracker log).

Jari

---

From: "Mary Barnes" <[email-masked]>
Date: Tue, 12 May 2009 09:14:18 -0500

If that's the model, then it would be good if it were consistent in the
agendas. There are still cases where the shepherd is specifically called
out in the "Notes" (even when the shepherd is a WG chair).   So, for
gen-art we've always interpreted the lack of indentification of a
shepherd to mean that the AD was the shepherd. 

Mary. 

-----Original Message-----
From: Jari Arkko [mailto:[email-masked]] 
Sent: Tuesday, May 12, 2009 8:55 AM
To: Barnes, Mary (RICH2:AR00)
Cc: Bert Wijnen (IETF); Russ Housley; [email-masked]
Subject: Re: PROTO Process

Mary,

> shepherd applies to the majority of the documents on the IESG agendas.

> However, most folks are conscientious enough to copy WG chairs, who 
> should provide the info to the Shepherd, if it's someone other than 
> the chairs.
>

Jari

---

From: Jari Arkko <[email-masked]>
Date: Tue, 12 May 2009 16:35:59 +0200

Mary,

> shepherd to mean that the AD was the shepherd. 

Point taken. I will take this up with the other ADs and see what we can 
do. Where does Gen-ART normally send its reviews? Do you Cc 
[email-masked]? The sponsoring AD?

Jari

---

From: "Mary Barnes" <[email-masked]>
Date: Tue, 12 May 2009 10:02:49 -0500

Thanks.  

Many of the gen-art reviewers use the WG chairs, document author and AD
email aliases. But, some just directly pull email addresses from the
documents (for authors) and only include the sponsoring AD (and not WG
chairs).   Some reviews are also sent to the WG mailing list. And, the
reviews may also be sent to IETF and IESG lists. It depends upon the
scope of the review and level of issues raised - i.e., if it's ready or
there are only nits, then the review is typically just sent to relevant
AD and authors.  But, I'm not certain many (if any) get sent to the
shepherd if they're not the WG chair (but I would think WG chairs would
forward). 

So, if the shepherd always needs to be in the loop, it seems gen-art
would need to make some adjustments here and the gen-art reviewers need
to look into the tracker and then follow the proto write-up link to get
the shepherd as there isn't room for that to get displayed in the
tracker itself for most documents. So, that's a bit of a pain itself and
I'm not seeing email addresses in some of these and actually there's one
that doesn't even have the proto author identified.  It would be much
easier if the shepherd were listed at the top of the tracker page, in
particular if we want folks to keep them in the loop on discussions. 

Mary. 

-----Original Message-----
From: Jari Arkko [mailto:[email-masked]] 
Sent: Tuesday, May 12, 2009 9:36 AM
To: Barnes, Mary (RICH2:AR00)
Cc: Bert Wijnen (IETF); Russ Housley; [email-masked]
Subject: shepherd e-mail addresses (was: Re: PROTO Process)

Mary,

> If that's the model, then it would be good if it were consistent in 
> the agendas. There are still cases where the shepherd is specifically
called

Jari

---

From: Olafur Gudmundsson <[email-masked]>
Date: Tue, 12 May 2009 11:16:30 -0400

>community get the envisioned benefits?  What changes are necessary to get them?

Well in my experience is that the general IETF community has no idea
what the role of the Doc Shep. is.
The ID-tracer hides who the shepherd is, I would suggest that his/her name
be listed on the top just like the AD name.

Based on what we have seen up to now I think we can put together a simple
guide to be doc. shepherd:
         do the WG Summary write up
         agree on a role with editors
                 who is in charge of answering questions
                         document contents
                         wg process/consensus
                         trouble makers
                 who suggests modified text
                 who is in charge of deciding changes

         Reach an understanding on what is expected by AD
                 how often to nag
                 ask if AD needs help or wait for instructions

         Subscribe to atom feed for draft.

         After LC double check all public comments have been answered
         Notify AD when all comments have been addressed

To help everyone it would be nice if the main page for the draft contained
a summary of the voting something like:
         Yes=2, NoObj=8, Dis=1, Abstain=1, Missing=1

The following idea may not fly but what the heck:

Create a short form for document shepherd to fill out to summarize the last
call similar to the working group summary.

         Q1: Have all public comments been addressed

         Q1.5: If NO List all unresolved comments (links to threads)

         Q1.7: Suggest a way forward

         Q2: What changes are needed in document before IESG processes
             the document?

         Q2.1: When will new version be available ?

         Q3: Any other information that might be helpful for IESG in reviewing
             the document

After this form is submitted the AD can do his write up.

In my experience the worst thing to happen is to discover the day before a
document is to be processed by the IESG that one important public comment
was not addressed and Russ puts a Discuss on the document.

         Olafur

---

From: Adrian Farrel <[email-masked]>
Date: Tue, 12 May 2009 14:34:11 +0100

David,

I think Dan's point is not that he is not seeing shepherd intervention, 
period.

The question arises because a bunch of I-Ds *are* getting stuck with 
unresolved and even undiscussed Discuss positions.
There are also a number of cases where the ADs (poor darlings) are 
effectively doing the shepherd work, or even doing the authors' work by 
taking on the effort of resolving the Discusses.

In these cases we are wondering whether/why the shepherds are not actively 
involved in the process.

Cheers,
Adrian
----- Original Message ----- 
From: "David Harrington" <[email-masked]>
To: "'Romascanu, Dan (Dan)'" <[email-masked]>; "'Jari Arkko'" 
<[email-masked]>; "'Russ Housley'" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, May 12, 2009 2:28 PM
Subject: RE: PROTO Process

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 12 May 2009 09:26:32 -0700

Hi -

> From: "DRAGE, Keith (Keith)" <[email-masked]>
> To: "Jari Arkko" <[email-masked]>; "Russ Housley" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, May 12, 2009 3:50 AM
...
> 1) In a number of cases the AD requests changes during the AD
> evaluation stage and the shepherd does not get informed of that.
> Perhaps you should do a survey of the number of instances were
> a "revised version required" is recorded

This has not been a problem in my limited experience.

> 2) Many of the issues do not get recorded in the drafts tracker tool,
> and the shepherds cannot actually update the tracker, so they are
> unable to use this as a tool to help them in the work.

In my WG, *everything* gets taken back to the WG for discussion,
even editorial / clarification comments.  This means that the issues
end recorded up in our tracker, if not the drafts tracker.

> Perhaps you should do a survey of the number of instances were
> a "revised version required" is recorded in the drafts tracker,
> against where tracker provides sufficient information to track
> down why a new version is needed. In any case, in an open
> process this information should be recorded and available to all.

I agree that the rationale for changes, especially those happening
late in the process, should be recorded *somewhere* publicly
accessible.  I'm not convinced that the drafts tracker is necessarily
the place to do it - pointers might suffice.

Randy

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Tue, 12 May 2009 18:28:40 +0200

Randy Presuhn wrote

> > Perhaps you should do a survey of the number of instances were a 
> > "revised version required" is recorded in the drafts 
> tracker, against 
> > where tracker provides sufficient information to track down 
> why a new 
> > version is needed. In any case, in an open process this information 
> > should be recorded and available to all.
> 
> I agree that the rationale for changes, especially those 
> happening late in the process, should be recorded *somewhere* 
> publicly accessible.  I'm not convinced that the drafts 
> tracker is necessarily the place to do it - pointers might suffice.

I used wording to allow it to be that. But if the "revised version required" tag exists in the tracker, I should be able to start with the draft tracker and find some content somewhere in an IETF archive as to what the expectations of that revised version are.

Keith 

> [mailto:[email-masked]] On Behalf Of Randy Presuhn
> Sent: Tuesday, May 12, 2009 5:27 PM
> To: [email-masked]
> > Perhaps you should do a survey of the number of instances were a 
> > "revised version required" is recorded
> 
> In my WG, *everything* gets taken back to the WG for 
> discussion, even editorial / clarification comments.  This 
> means that the issues end recorded up in our tracker, if not 
> the drafts tracker.
> 
> > Perhaps you should do a survey of the number of instances were a 
> > "revised version required" is recorded in the drafts 
> tracker, against 
> > where tracker provides sufficient information to track down 
> why a new 
> > version is needed. In any case, in an open process this information 
> > should be recorded and available to all.
> 
> I agree that the rationale for changes, especially those 
> happening late in the process, should be recorded *somewhere* 
> publicly accessible.  I'm not convinced that the drafts 
> tracker is necessarily the place to do it - pointers might suffice.
> 
> Randy
> 
>

---

From: Fred Baker <[email-masked]>
Date: Tue, 12 May 2009 10:25:39 -0700

I guess the key question is "what additional benefits did the IESG  
want besides streamlining its workload with help from the working  
group chairs"?

I gather that the IESG hopes that the shepherds will bird-dog the  
process better. Anything else?

---

From: Jari Arkko <[email-masked]>
Date: Wed, 13 May 2009 12:34:16 +0200

I wanted to provide some additional background on why we are asking 
about the PROTO process.

The IESG met this week and talked about a number of things related to 
our process. One of the things that we would like to improve upon is the 
progress of drafts through the IESG process. There are several problems 
in this, including:

- the fact that 82% of drafts have to be modified during AD review - 
IESG review (and the rest probably have RFC Editor notes)
- the process takes a relatively long time (avg 9 weeks for drafts that 
stay unchanged and 31 weeks for drafts that change)
- the timing is rather unpredictable, there is great variation among drafts
- the visibility to what is happening and who has the token could be 
improved
- there is at least anecdotal evidence that for many drafts, there is a 
long wait for new version or a response to review comments

Various statistics on these are available at 
http://www.arkko.com/tools/admeasurements/stat/base.html (but do not 
stare at the numbers too directly). The situation seems to have improved 
with regards to various metrics in recent times, but the IESG discussed 
a number of actions to make additional improvements. These include:

- Reminding ADs and shepherds to keep their working groups better 
informed about the ongoing reviews, raised issues, and suggested changes 
to their drafts.

- Starting a discussion in the wgchairs list about the roles of 
shepherds and whether there is something to improve.

- Create a "stall" state in the tracker. This would be used for 
documents where the wait for a response to an IESG or other review has 
taken a long a time. The fact that the draft has entered this state 
should also be communicated to the WG.

- Reminding everyone that if the holdup on a particular issue is due to 
an unresponsive AD, you can contact Russ to send a reminder.

Then I wanted to say a few things about the shepherd process. Please 
remember this works well in many cases, and we get great help from the 
shepherds in resolving issues. Or the authors and ADs converge quickly 
and things just move forward. However, there are cases where I would 
expect more action from the shepherds. For instance, Pasi often reports 
in his AD notes 
(http://www.ietf.org/mail-archive/web/saag/current/msg02643.html) about 
cases where he has raised a Discuss but has not heard from the authors. 
The shepherds and WG chairs should be active in such cases, reminding 
authors to move forward, finding new editors if needed, etc.

I'd also observe that generally speaking the chairs and shepherds and 
more experienced in the IETF process and have worked with more 
specifications than individual authors. This experience is very valuable 
when talking about issues raised during the Last Call and IESG reviews. 
It takes judgment and knowledge of the WG's area to determine whether a 
particular issue is truly a problem. The ADs make these calls, but it 
often comes down to a discussion between an individual AD having 
specific expertise (such as transport or security) and the author; I 
would like to see more shepherds participating in the discussions and 
weighing in. Pushback on Discusses is appreciated!

About the writeups: the ADs do read these, and I at least find them 
useful. I also often read writeups from other AD's documents in the IESG 
review stage, if something about the status or history of the draft is 
unclear. Of course, the writeups are truly useful only if the shepherd 
really did do a final review of the document, checked the nits, and 
provided more useful background information than simply copying the 
abstract from the document :-) But in most cases that I'm aware of they 
do contain useful information.

Jari

---

From: Henk Uijterwaal <[email-masked]>
Date: Wed, 13 May 2009 13:22:34 +0200

Jari,

> IESG review (and the rest probably have RFC Editor notes)

I think it'd be good if we could split the modifications into
"technical" and "editorial".  ID's are often written by engineers
who do not have English as their first language.  Reviewers (WG
chairs, sheperds, AD's) don't have that either and aren't trained
into reviewing documents without paying attention to the technical
content.   If we find that there is a large number of editorial
comments per draft, then it might make sense to do an editorial
review first.

> long wait for new version or a response to review comments
[...]
> an unresponsive AD, you can contact Russ to send a reminder.

I think that if we want to improve this, we should first define what an
acceptable time for a response from an author (or AD) is.

Henk

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 13 May 2009 10:17:37 -0400

> IESG review (and the rest probably have RFC Editor notes)

Surely that problem is not something that can be solved by procedural
rules?  If there is so much need for modification during AD review, it
suggests that documents are coming from WGs without being ready.  If
that is indeed the problem, I find it pretty hard to imagine the mere
procedural tweak that will solve it.

The alternative explanation is that the "have to be modified" could be
stated slightly differently as "have to be modified to get through" AD
review, which is the sort of thing I sometimes hear people saying.
This locution suggests that people do not regard the IETF and AD
reviews as useful tests that improve the technical quality of a
document, but that instead the reviews are regarded as hurdles to
clear on the way to publishing an RFC.  This might suggest a
reluctance to compromise that is not ideally matched for our
consensus-based approach.  If that's right (and I don't know whether
it is), then no procedural tweaks will fix it either, because the
problem is still the suitability of the documents for publication as
RFCs unless we abandon rough consensus as a test.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Ross Callon <[email-masked]>
Date: Wed, 13 May 2009 09:53:18 -0400

> - Reminding everyone that if the holdup on a particular issue is
> due to an unresponsive AD, you can contact Russ to send a reminder.

You can also prod the AD directly. There are some WG chairs who are
quite good at gently prodding ADs (which after some delay is appreciated
in the case that the AD has overlooked a task). 

Ross

---

From: Sean Turner <[email-masked]>
Date: Wed, 13 May 2009 11:24:27 -0400

> IESG review (and the rest probably have RFC Editor notes)

I think when you ask people to comment they're going to so I'm actually 
surprised the number isn't higher ;)

spt

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 13 May 2009 16:43:10 +0100

Sean Turner wrote:

> I think when you ask people to comment they're going to so I'm 
> actually surprised the number isn't higher ;)

Of course, our employers give us bonus based on the number of DISCUSSes 
filed ;-).

---

From: Jari Arkko <[email-masked]>
Date: Wed, 13 May 2009 21:34:40 +0300

Henk,

> acceptable time for a response from an author (or AD) is.

Yes. The IESG talked about new stretch goal times for the time between 
publication request and draft approval. These would be 6 weeks for 
drafts without changes and 8 weeks + any author time for others. In 
addition, when the draft enters a state where we are waiting for the AD 
to do something, we'd like to see some action within two weeks.

I think two weeks would be a good default maximum time for authors to 
respond or react with a new document version.

Of course, there are special cases that will go over these times for 
various reasons. Vacations and difficult document changes, for instance. 
And I hope most interaction actually happens immediately; same day 
responses seem to lead to immediate action on both sides. If everyone 
responds exactly in two weeks, the RTT in the discussion will be 1 month :-(

Jari

---

From: Jari Arkko <[email-masked]>
Date: Wed, 13 May 2009 21:38:36 +0300

Ross,

> in the case that the AD has overlooked a task). 

Indeed -- what I wrote above was only meant as a last resort, if other 
means of waking up the AD have failed.

Jari

---

From: Fred Baker <[email-masked]>
Date: Wed, 13 May 2009 14:05:24 -0700

On May 13, 2009, at 7:17 AM, Andrew Sullivan wrote:
> The alternative explanation is that the "have to be modified" could  
> be stated slightly differently as "have to be modified to get  
> through" AD review, which is the sort of thing I sometimes hear  
> people saying. This locution suggests that people do not regard the  
> IETF and AD reviews as useful tests that improve the technical  
> quality of a document, but that instead the reviews are regarded as  
> hurdles to clear on the way to publishing an RFC. This might suggest  
> a reluctance to compromise that is not ideally matched for our  

Having been on both sides of this divide, I think I can attest that  
both views hold water.

As IETF chair and with Scott Bradner as one of the ADs, I noted that  
any draft that used normative language and didn't cite RFC 2119 would  
get a "discuss" on the point, and included that in my (decade-old now)  
id nits bullet list simply to avoid the predictable revision cycle. As  
an author on what is now RFC 4192, I found a particular AD to be using  
his position on the IESG as leverage to force into the document a  
statement which was both vacuous and untrue - and which the working  
group had not discussed. I asked him for text supporting his position,  
which he didn't supply, and his "discuss" was removed when he left the  
IESG. Those are both cases of "what it took to get past the IESG".

On the other hand, at least in my opinion, the original L2TP draft  
sent to the IESG was largely incomprehensible. Tom Narten's efforts  
with the authors, centered around "I'm not removing my 'discuss' until  
I can understand your document", resulted in a complete rewrite, which  
was a vast improvement. While that was an extreme case, I can think of  
many cases in which IESG comments have improved documents in real ways.

As an author, I sometimes get the feeling that ADs place discusses  
because they feel honor-bound to find something wrong with a document.  
I personally would appreciate getting those remarks during the working  
group process rather than having the AD wait for last call to place  
them. I appreciated Magnus' detailed comments on a draft I am working  
on in behave coming now rather than later, as they constitute a lot of  
additional effort - effort that was not deemed necessary when SIIT,  
which the draft updates, was first proposed. In general, I think that  
directorate review and AD review works best if it happens during  
working group discussion rather than as a remedial effort after the  
fact.

---

From: Jari Arkko <[email-masked]>
Date: Thu, 14 May 2009 11:04:30 +0300

Henk,

>> Yes. The IESG talked about new stretch goal times for the time 
>> between publication request and draft approval. These would be 6 
>> weeks for drafts without changes and 8 weeks + any author time for 
>> others. In addition, when the draft enters a state where we are 
>> waiting for the AD to do something, we'd like to see some action 
>> within two weeks.
>>
> Can we make this two-way?  "author or AD is expected to respond in 2
> weeks".

Yes -- that's why the first paragraph above said "some action within two 
weeks".

Jari

---

From: Henk Uijterwaal <[email-masked]>
Date: Thu, 14 May 2009 07:58:57 +0200

Jari,

Can we make this two-way?  "author or AD is expected to respond in 2
weeks".  I recently had a draft stuck for almost 8 weeks due to an
unresponsive AD :-(

Henk

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 14 May 2009 12:49:30 -0400

> respond or react with a new document version.

Or what?  I worry that this is going to turn into another procedural
rule that will just frustrate well-meaning people who for some reason
can't meet it, while others who are obviously gaming the system will
sail through because they "followed the rules".  

I'm still wondering, however, whether we're tinkering with the wrong
variable.  If documents really need that much work when they come out
of WGs, does that mean that they're really in bad shape?  If so, I
argue we ought to fix that, rather than trying to come up with more
process rules.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Peter Koch <[email-masked]>
Date: Thu, 14 May 2009 18:59:03 +0200

> argue we ought to fix that, rather than trying to come up with more

seconded.  Some cases I remember were clearly cross-area or cross-wg issues
that are less about "bad shape" but some aspect of another protocol being
used in undesirable ways.  IESG review is a bad time to discover this, so
response timers won't help solving the lock/DISCUSS. Early review is key,
but I'm not sure that's part of the PROTO process.

-Peter

---

From: Fred Baker <[email-masked]>
Date: Thu, 14 May 2009 10:15:40 -0700

On May 14, 2009, at 9:49 AM, Andrew Sullivan wrote:

> of WGs, does that mean that they're really in bad shape?

That's worth looking at, but as I said yesterday, I think there is  
also room for concern on the IESG side.

On May 13, 2009, at 2:05 PM, Fred Baker wrote:
> because they feel honor-bound to find something wrong with a  
> document. I personally would appreciate getting those remarks during  
> the working group process

To be honest, getting solid reviews in the WGs can be a pain, and I  
think that's what you're saying is needed.

---

From: Ben Campbell <[email-masked]>
Date: Thu, 14 May 2009 12:31:39 -0500

> sail through because they "followed the rules".

I've had similar thoughts. Let's not forget this is a volunteer  
organization. It's hard to hold volunteer's feet to the fire. For some  
people (and I'm not talking about myself) IETF participation is a  
sideline to their day jobs. If we up the minimum required effort level  
too much, we may find ourselves limited to participants that are paid  
to participate.

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 14 May 2009 13:45:48 -0400

> think that's what you're saying is needed.

Right, it sure _sounds_ like that's the problem.  I guess I'm
sceptical that this problem can really be resolved by more process
rules.

There are three possibilities here.  

The first is that the documents really are getting to the IESG in need
of additional work -- sometimes because the WG didn't tackle document
clarity issues (Fred suggested one such case yesterday), sometimes
because of inter-WG issues (like Peter mentioned), or sometimes
because WG review was just not good enough (as Fred suggests above).
In this case, I have doubts that additional process rules will help.
For in this case, the problem is less the turn-around time, I think,
and more the substantive disagreements that need to be addressed.
It's a bad thing that they come so late, and I don't think new IESG
processing rules will help this core issue of the disagreements
turning up when the document comes out of the WG.

The second is that some IESG members decide to use the position to
insist upon positions contrary to WG consensus (Fred mentioned being
aware of this case yesterday).  In this case, I also doubt process
rules will help.  The rules would of course govern the way that the
dispute would play out, but it wouldn't actually change the result:
either someone would back down, or else the document would go into
limbo forever.  We can't fix this problem with a process rule, in
fact, because sometimes we want IESG members to be able to use their
judgement and to be able to insist on changes in exactly this way.
Unfortunately, that means that individuals with a very strongly-held
belief and who are not amenable to compromise can hold up the works.
This is a problem of human judgement, and I have yet to see a system
where such positions of trust cannot be abused.  If this is the
problem we want to solve, then I think we need to give direction to
the NomCom to the effect that positive evidence of a willingness to
compromise is a requirement of serving on the IESG.  (Yes, I can think
of about a million objections to that, too.)

The third is that almost all of the delay is due to the process,
because people aren't responding fast enough.  Given the number of
documents that actually do get changed, however, it seems possible
that any delays that are just from slow response times are themselves
attributable to the number of documents that need to be changed: a
single person can only process so much, and the volume might simply be
too great if 80% of them really do need work.

All of that sends me back to the problem of getting documents from WGs
that are almost always ready for publication, so that IETF- and
IESG-period changes become the exception rather than the rule.  I
wonder whether there might in fact be a procedural change we can make.

Would better scheduling of document flow from WGs be helpful?  If we
had clear, author/editor-generated dates at which things are projected
to be ready, then we could announce IETF-wide "Document _n_ will be
available on $date, and it deals with {$topic1..$topic_t_}."  Would
such announcements do anything more for review than the current RFC
Editor postings on the arrival of new I-Ds?

Note that none of the above is to suggest automatically that things
like two-week guidelines are obviously bad.  But if 80% of documents
coming from WGs really do need to be fixed on their way through to
publication, no amount of tinkering will cure that bad illness.  We
need to address the basic problem.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Eric Burger <[email-masked]>
Date: Thu, 14 May 2009 13:28:57 -0400

+1.  On the one hand, most of the documents I have been shepherding  
have not had the "lost in IESG" issue Jari mentions.  However, I have  
on numerous occasions had the impression that some ADs really wanted  
to "add value" to a document.  If there was a procedure to correct  
this, I am all for it!

On May 13, 2009, at 5:05 PM, Fred Baker wrote:

>> hurdles to clear on the way to publishing an RFC. This might  
>> suggest a reluctance to compromise that is not ideally matched for  
>> our consensus-based approach. If that's right (and I don't know  
>> whether it is), then no procedural tweaks will fix it either,  
>> because the problem is still the suitability of the documents for  
>> publication as RFCs unless we abandon rough consensus as a test.
>
> any draft that used normative language and didn't cite RFC 2119  
> would get a "discuss" on the point, and included that in my (decade- 
> old now) id nits bullet list simply to avoid the predictable  
> revision cycle. As an author on what is now RFC 4192, I found a  
> particular AD to be using his position on the IESG as leverage to  
> force into the document a statement which was both vacuous and  
> untrue - and which the working group had not discussed. I asked him  
> for text supporting his position, which he didn't supply, and his  
> "discuss" was removed when he left the IESG. Those are both cases of  
> "what it took to get past the IESG".
>
> with the authors, centered around "I'm not removing my 'discuss'  
> until I can understand your document", resulted in a complete  
> rewrite, which was a vast improvement. While that was an extreme  
> case, I can think of many cases in which IESG comments have improved  
> documents in real ways.
>
> the working group process rather than having the AD wait for last  
> call to place them. I appreciated Magnus' detailed comments on a  
> draft I am working on in behave coming now rather than later, as  
> they constitute a lot of additional effort - effort that was not  
> deemed necessary when SIIT, which the draft updates, was first  
> proposed. In general, I think that directorate review and AD review  
> works best if it happens during working group discussion rather than  
> as a remedial effort after the fact.

---

From: Jari Arkko <[email-masked]>
Date: Thu, 14 May 2009 21:38:49 +0300

Andrew,

> I guess I'm

I think we need to separate a few different problems and what the remedy 
for them might be.

1) The ball is simply dropped. Do we have this problem? Yes we do, at 
least for some number of drafts. For an example, look at the link I 
provided earlier to Pasi's AD notes. Possible remedies: more active 
shepherds in WGs that have this problem, creating a stall state to more 
accurately reflect where the document is, sending more mail to the WG 
list to make sure everyone knows what the situation is, and escalation 
path via Russ in cases ADs are not responding. This is all relatively 
easy to do.

2) The quality at Publication Request is too low. Do we have this 
problem? I think we do. Clearly, more work would have to be done at the 
WG stage, or perhaps some late effort should be moved earlier.

3) Discusses that raise non-issues. There are some of these. Probably 
one third or so of all Discusses are cleared in the first telechat 
without any changes, after some misunderstanding or lack of information 
is clarified. But that's fine. But then there are some cases where a 
non-issue Discuss survives further and causes an unnecessary change in 
the document. IMHO a small number, but anyway. The only way to deal with 
this, however, is the constant discussion between the IESG and the WG 
(or the greater community) about what issues are significant enough to 
warrant a mandatory fix. Pushback from the shepherd and the sponsoring 
AD would be helpful here, obviously only when the pushback really is 
warranted.

4) Pace of progress through the process is slow. Do we have this 
problem? Yes we do. Possible remedies include fixes to the specific 
problems, such as the three ones above, but also setting expectations. 
Hence the suggested target times. I do realize that we have no means to 
enforce these in a volunteer organization. And that there are legitimate 
cases where both ADs and authors may be unable to follow the timelines. 
However, this does not mean that we should forget setting goals or 
remind people if they don't reach them.

I realize that #2 is probably the biggie. But lets not disregard the 
other ones either, particularly the ones that are easy to do...

Jari

---

From: "Mary Barnes" <[email-masked]>
Date: Thu, 14 May 2009 13:41:36 -0500

Andrew, 

We have tried something similar to the approach you mention in the
SIPPING WG over the past 3 years - i.e., publishing deadlines for
document reviews and subsequent updates post review, etc. (within the
WG).   In the end, one of the problems we commonly encountered was
authors not having enough cycles to do the updates in a timely manner
(at least within the WG).  The WGLC and IESG target timeframes that we
were publishing kept having to expand (i.e., originally we had thought 2
weeks was reasonable to get an update post review, but in the end that
got to be more like 8 weeks) and it was quite rare that a document was
ready in what one would consider a reasonable timeframe.  However, I do
think that our requirement of ensuring that we had 3 to 4 people doing
very detailed reviews of the documents did improve the overall quality
and the changes for the majority of the documents once they left the WG
were generally handled in a fairly timely manner (although, there are
some documents that have languished - both due to lack of author cycles
and some IESG concerns over the solution approach).  We also had
multiple types of reviews - "Initial WG reviews", "pre-WGLC", "WGLC"
(sometimes multiple), "post-WGLC" as a document was progressed to try to
get as much feedback as possible earlier in the process.

As others have suggested, however, even a process like this doesn't
really keep folks from coming in at the last minute and disagreeing with
the work item. But, it did seem to help in getting these concerns in a
bit earlier rather than later.   

We did cross area and cross WG reviews (within RAI) for documents as
appropriate before they left the WGs, which I think also helps with
resolving some of the potential IESG type concerns earlier.   

I would be concerned about trying to take this sort of process IETF wide
(i.e., announcing doc deadlines for WG documents) due to the amount of
noise.   I think such announcements are certainly appropriate at the WG
level and maybe the area level.  But, I honestly have a hard enough time
just catching the documents published for the WGs that I chair, have
deliverables in and participate in. I've actually given up on documents
for some of the WGs that I participate in and now just rely on the tools
page to keep track of the relevant documents. One suggestion would be
for WG chairs to use the "Comments/issues" field on the tools page to
publish the internal WG deadlines. It seems that only a few chairs are
even taking advantage of that field right now.  That would make it
easier as right now, you often have to backtrack through WG minutes to
find out where a doc is internally in the WG. 

Regards,
Mary   

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Andrew Sullivan
Sent: Thursday, May 14, 2009 12:46 PM
To: Fred Baker
Cc: [email-masked]
Subject: Re: PROTO Process

.....

All of that sends me back to the problem of getting documents from WGs
that are almost always ready for publication, so that IETF- and
IESG-period changes become the exception rather than the rule.  I wonder
whether there might in fact be a procedural change we can make.

Would better scheduling of document flow from WGs be helpful?  If we had
clear, author/editor-generated dates at which things are projected to be
ready, then we could announce IETF-wide "Document _n_ will be available
on $date, and it deals with {$topic1..$topic_t_}."  Would such
announcements do anything more for review than the current RFC Editor
postings on the arrival of new I-Ds?

Note that none of the above is to suggest automatically that things like
two-week guidelines are obviously bad.  But if 80% of documents coming
from WGs really do need to be fixed on their way through to publication,
no amount of tinkering will cure that bad illness.  We need to address
the basic problem.

A

--
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Sean Turner <[email-masked]>
Date: Thu, 14 May 2009 14:48:46 -0400

> sail through because they "followed the rules".  

This was the first thing that went through my mind.

spt

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Fri, 15 May 2009 12:20:32 +0200

I's not sure what side of the fence Eric is sitting on here.

I have a view that in many cases the IESG is trying to do too much to a document. Certainly I do not believe it is an IESG function to add value. I suspect the process should be more clear cut of either approving the document warts and all, or deciding it is not publication ready and sending it back to the working group with a clear description of why the document could not be published.

That would solve the "buried in the IESG" problem in one quick process change, and encourage the issues with the documents to get out on the working group lists, which is what fails to happen at the moment.

If the document is really wrong, they can work on it again. 

For a number of the IESG comments I have seen have come from our own area directors, I believe they should really have been made as individuals during the consensus building process in the WG, rather than as IESG comments.

regards

Keith 

> [mailto:[email-masked]] On Behalf Of Eric Burger
> Sent: Thursday, May 14, 2009 6:29 PM
> To: Working Group Chairs
> have not had the "lost in IESG" issue Jari mentions.  
> However, I have on numerous occasions had the impression that 
> some ADs really wanted to "add value" to a document.  If 
> there was a procedure to correct this, I am all for it!
> 
> > get a "discuss" on the point, and included that in my (decade- old 
> > now) id nits bullet list simply to avoid the predictable revision 
> > cycle. As an author on what is now RFC 4192, I found a 
> particular AD 
> > to be using his position on the IESG as leverage to force into the 
> > document a statement which was both vacuous and untrue - 
> and which the 
> > working group had not discussed. I asked him for text 
> supporting his 
> > position, which he didn't supply, and his "discuss" was 
> removed when 
> > he left the IESG. Those are both cases of "what it took to get past 
> > the IESG".
> >
> > until I can understand your document", resulted in a 
> complete rewrite, 
> > which was a vast improvement. While that was an extreme case, I can 
> > think of many cases in which IESG comments have improved 
> documents in 
> > real ways.
> >

---

From: Lisa Dusseault <[email-masked]>
Date: Thu, 14 May 2009 11:58:53 -0700

> In this case, I have doubts that additional process rules will help.

Agreed.  To be blunt, my fellow ADs, past and present, are exceptional
reviewers,
aware of dozens of pitfalls with IANA process, various registry peculiarities,
reference issues, conflicting work, and most notably, experts in recognizing
when a statement is ambiguous even if the authors and even WG reviewers
automatically fill in the missing pieces.  Many of these are entered as
DISCUSS issues so that the AD who found the issue can confirm in
telechat time if their understanding is correct.  Some of these issues could be
re-entered as COMMENT but our tools encourage us to keep a DISCUSS
so that the AD who is sponsoring the document gets a reminder to check
that an issue really is cleared up.

> too great if 80% of them really do need work.

A lot of time is spent waiting for authors to make decisions or revisions.
Explicit author wait states account for an average of over 50 days delay!
In addition, some states like "AD Followup" include time passing while
authors or WG chairs respond and decide what to do.

My personal attitude towards this is that authors are volunteers who have
to pull together a few hours now and then in the interstices between a
full-time job commitment and the rest of their lives.  So if it takes three
months to get a document revision, so be it.  My time, on the other hand,
is committed to being an AD for at least some hours every week.  I make
sure every month that there are no documents that are waiting on my
next action, once identified that doc goes to the top of my priority queue.

I mention this because your wording of "single person" might be read as
the AD being the bottleneck, having at any time a dozen documents with
DISCUSS raised, and a dozen documents on which that AD has raised a
DISCUSS.  My experience is that's not the bottleneck -- communication
and editing wait-states are, and the bottleneck there is the difficulty of
making decision among 3 to 10 people (authors and chairs and shepherds)
with many other things to do.  I certainly can't complain about fellow
ADs being
unresponsive to clearing their DISCUSS (or updating it) when presented
with information and because it's their job I'm happy to be aggressive
bugging them until they do respond.

> Would better scheduling of document flow from WGs be helpful?

I don't think so.  I would much rather get docs from WGs as soon as
they're ready.  Most WG documents spend longer in the WG than in
processing after pub-request so this would be a bad fix anyway.

> need to address the basic problem.

Do we want less review from the IESG?  As many have pointed out,
it might be better to get Proposed Standards out faster and fix them
up in the Draft stage.  Currently our tools and processes are structured
to do more careful content review for PS docs, than to do any other
part of our work.  We must read nearly every PS proposal
because our vote processes require a minimum number of ADs to
pass a PS, and even "No Objection" implies that we've reviewed
the document and agree it passes muster.

I must be becoming a cynical "old" AD because I dislike much of our
processes and biases but see little to no hope in changing them in
ways that truly matter.  And sorry if jetlag is contributing to the
cynicism and bluntness!

Lisa

---

From: Eric Burger <[email-masked]>
Date: Fri, 15 May 2009 14:17:01 -0400

If this helps: "Adding Value" is a derogatory comment.

On May 15, 2009, at 6:20 AM, DRAGE, Keith (Keith) wrote:

> I have a view that in many cases the IESG is trying to do too much  
> to a document. Certainly I do not believe it is an IESG function to  
> add value. I suspect the process should be more clear cut of either  
> approving the document warts and all, or deciding it is not  
> publication ready and sending it back to the working group with a  
> clear description of why the document could not be published.
>
> That would solve the "buried in the IESG" problem in one quick  
> process change, and encourage the issues with the documents to get  
> out on the working group lists, which is what fails to happen at the  
> moment.
>
> For a number of the IESG comments I have seen have come from our own  
> area directors, I believe they should really have been made as  
> individuals during the consensus building process in the WG, rather  
> than as IESG comments.
>
> regards
>
> Keith
>

---

From: Fred Baker <[email-masked]>
Date: Fri, 15 May 2009 12:57:13 -0700

> individuals during the consensus building process in the WG, rather  

personally, I prefer such.

To Lisa's comment that the ADs are great reviewers, they are. I'm not  
complaining about getting good reviews, and I agree that the WGs often  
don't do an adequate job. I'm complaining about how late in the cycle  
they come, and as "QA" as opposed to "contribution".

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 18 May 2009 10:47:45 -0700

> Sent: Tuesday, May 12, 2009 3:14 AM
> To: Russ Housley
> Cc: [email-masked]
> relatively silent.

Document shepherds are not on the IESG call, so do not hear
any nuance of IESG discussion/comment -- all that is available
is what is published online (in the tracker) or by talking 1:1
with the IESG member(s) that raised a DISCUSS.

This limits the effectiveness of the document shepherd to help
the document through the IESG -- because the shepherd has the
same information as the authors (namely, the information in the
tracker and the ability to directly contact IESG members raising
a DISCUSS).

-d

> Is this a correct perception of the 
> reality? And if

---

From: Dean Willis <[email-masked]>
Date: Wed, 20 May 2009 13:41:11 -0500

> Perhaps you should do a survey of the number of instances were a "revised version required" is recorded in the drafts tracker, against where tracker provides sufficient information to track down wh…

Keith is spot-on. Frequently, the shepherd gets left out of the loop,
and can serve only as a watchdog timer that pops up occasionally and
says "Is this process stalled?"

Putting the feedback process into the tracker would be a big help.

Making the shepherd an active participant in the "use of tools" would be
even bigger.

--
Dean

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 20 May 2009 13:50:41 -0500

Just to follow-on Dean and Keith, this is also an interesting problem for 
the review teams.

At least on Gen-ART, it's helpful to look back at what's been happening to a 
document since pub-req - and information about the issues that came up in 
Last Call, or in AD review, is often missing from the tracker. We're just 
looking at "revised version required".

I'm still seeing tracker entries that don't include shepherd writeups, 
although this is getting rarer.

So, yeah. What they said.

Thanks,

Spencer

>> 2) Many of the issues do not get recorded in the drafts tracker tool, and 
>> the shepherds cannot actually update the tracker, so they are unable to 
>> use this as a tool to help them in the work.
>>
>> Perhaps you should do a survey of the number of instances were a "revised 
>> version required" is recorded in the drafts tracker, against where 
>> tracker provides sufficient information to track down why a new version 
>> is needed. In any case, in an open process this information should be 
>> recorded and available to all.

... deleted down to