Subject: NUDGE: WG milestones tool requirements: draft-ietf-genarea-milestones-tool-00.txt

From: Paul Hoffman <[email-masked]>
Date: Wed, 4 May 2011 10:10:20 -0700

[[ Did I really get all the requirements right on the first try? :-) ]]

Greetings again. Following the lively discussion we had here a few weeks ago, the IAOC asked me to produce a new Internet Draft specifically about a tool for chairs to update their milestones. This has now been separated from the earlier work on updating charters.

After talking to some ADs and reading the earlier threads here, I put together <http://tools.ietf.org/html/draft-ietf-genarea-milestones-tool-00>. This is explicitly a first draft and is open for lots of changes, clarifications, and so on. In the earlier thread, there was wide disagreement on who should approve milestone additions and changes, so section 4 of the -00 is my first guess about what might work for most people, but I could have missed the mark.

I am Cc'ing the IESG on this message, but the forum for discussion of the draft (at this stage) is the [email-masked] mailing list.

The sooner we come to resolution on these requirements, the sooner the IAOC can put out an RFP to make the tool become real.

--Paul Hoffman

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Wed, 4 May 2011 19:28:15 +0200

I think part of the issue regarding the level of approval of changes is the length of time that this sometimes takes. If the AD is absent, then certainly a new milestone is not going to get approved until they return. In the distant past it sometimes took me several months to set a milestone, because the AD was unresponsive, and when they did finally respond, I had to find time to go and remind myself what the issue was.

I guess the proposal in section 4 at least gives us the opportunity for trialling a tool and seeing what works. Of your 5 bullets, I would note that in my understanding, 3 and 5 do not currently require AD approval, and I would prefer the default to retain this.

In section 5, I would like the linkage to i-ds to be linked to completion of milestones as well, i.e. that if we have a milestone of the form "Submit xxxx to IESG as proposed standard", and that is associated with one or more i-ds, then when the appropriate state change or changes occurs in the tracker, then the milestone is automatically marked as complete (subject to anything in section 4 of course). An alternative would be to prompt the WG chair for completion.

Keith

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Paul Hoffman
> Sent: 04 May 2011 18:10
> To: IETF WG chairs
> Subject: NUDGE: WG milestones tool requirements: draft-ietf-genarea-
> milestones-tool-00.txt
> 
> Greetings again. Following the lively discussion we had here a few weeks
> ago, the IAOC asked me to produce a new Internet Draft specifically about
> a tool for chairs to update their milestones. This has now been separated
> from the earlier work on updating charters.
> 
> After talking to some ADs and reading the earlier threads here, I put
> together <http://tools.ietf.org/html/draft-ietf-genarea-milestones-tool-
> 00>. This is explicitly a first draft and is open for lots of changes,
> clarifications, and so on. In the earlier thread, there was wide
> disagreement on who should approve milestone additions and changes, so
> section 4 of the -00 is my first guess about what might work for most
> people, but I could have missed the mark.
> 
> I am Cc'ing the IESG on this message, but the forum for discussion of the
> draft (at this stage) is the [email-masked] mailing list.
> 
> The sooner we come to resolution on these requirements, the sooner the
> IAOC can put out an RFP to make the tool become real.
> 
> --Paul Hoffman

---

From: Adrian Farrel <[email-masked]>
Date: Wed, 04 May 2011 18:42:18 +0100

> I think part of the issue regarding the level of approval of changes is the
length of
> time that this sometimes takes. If the AD is absent, then certainly a new
> milestone is not going to get approved until they return. In the distant past
it
> sometimes took me several months to set a milestone, because the AD was
> unresponsive, and when they did finally respond, I had to find time to go and
> remind myself what the issue was.

There is an escalation process for unresponsive ADs.
- The co-AD can usually attract their attention.
- The IETF chair should be quite interested in helping.
- A Recall petition may wake them up.

Let us not over-engineer the process for an error case that already has code to
handle it.

This is not to say that I favor the ADs having absolute control over everything,
and I actually think that the WG chairs should be able to give strong input on
their milestones. I have no problem with chairs being able to set their own
milestones. But I do believe that the system we have places the responsibility
for guiding WG charters with the ADs: unless we plan to change that, the ADs
have to have the power to say "no" to anything in the charter (consensus,
NomCom, etc. permitting).

IMHO, it is far easier for everyone if the milestones published in a charter are
not retracted two weeks later when the AD notices them. Better to get them
discussed and approved up front.

Cheers,
Adrian

PS I wish *my* problem was that the WG chairs had milestone changes they wanted
to introduce to help them move their charters along more quickly :-)

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 6 May 2011 14:27:48 -0700

> I think part of the issue regarding the level of approval of changes is the length of time that this sometimes takes. If the AD is absent, then certainly a new milestone is not going to get approve…

This can probably be dealt with by a new requirement: that requests that are not accepted or rejected cause a (weekly?) reminder being sent to the chairs and AD that there pending requests, and list them.

Does that work for you?

> I guess the proposal in section 4 at least gives us the opportunity for trialling a tool and seeing what works.

It sounds like you are wanting to see what doesn't work. :-)

> Of your 5 bullets,
   o  create new milestones

   o  change milestone descriptions

   o  change milestone due dates

   o  change which Internet-Drafts are associated with a milestone

   o  assert that a milestone is completed
> I would note that in my understanding, 3 and 5 do not currently require AD approval, and I would prefer the default to retain this.

Our impressions differ. I do not think and a WG chair can make changes to an of those without an AD agreeing to it. Further, even if you are right, I suspect some ADs would find that to be an oversight, not a feature.

> In section 5, I would like the linkage to i-ds to be linked to completion of milestones as well, i.e. that if we have a milestone of the form "Submit xxxx to IESG as proposed standard", and that is…

This makes a faulty assumption, that milestone text like "Submit xxxx to IESG as proposed standard" is machine-interpretable. Today, it is not, and probably cannot be done easily. In some WGs I have been in, there has been disagreement about whether such a milestone is met when the submission starts, or when the the last @#$% DISCUSS is finally cleared.

Is such automation really needed? Shouldn't WG chairs be looking at their milestones every month or so to see if any need to be dispatched? A middle ground might be that when a draft that is listed in a milestone moves between post-WG states, mail is sent to the WG chairs noting that fact. You can then decide when to propose a milestone change.

--Paul Hoffman

---

From: "David Harrington" <[email-masked]>
Date: Fri, 6 May 2011 19:08:37 -0400

Hi, 

> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Paul Hoffman
> Sent: Friday, May 06, 2011 5:28 PM
> To: DRAGE, Keith (Keith)
> Cc: IETF WG chairs
> Subject: Re: NUDGE: WG milestones tool 
> requirements:draft-ietf-genarea-milestones-tool-00.txt
> 
> > I think part of the issue regarding the level of approval 
> of changes is the length of time that this sometimes takes. 
> If the AD is absent, then certainly a new milestone is not 
> going to get approved until they return. In the distant past 
> it sometimes took me several months to set a milestone, 
> because the AD was unresponsive, and when they did finally 
> respond, I had to find time to go and remind myself what the 
> issue was.
> 
> This can probably be dealt with by a new requirement: that 
> requests that are not accepted or rejected cause a (weekly?) 
> reminder being sent to the chairs and AD that there pending 
> requests, and list them.
> 
> Does that work for you?

This can be addressed by the chair sending an email to the AD.
If the AD is unresponsive to personal emails from his chairs, why
would he be responsive to machine-generated spam?
I recommend you find out his IM address or his phone, and call him
personally.

> 
> > I guess the proposal in section 4 at least gives us the 
> opportunity for trialling a tool and seeing what works.
> 
> > I would note that in my understanding, 3 and 5 do not 
> currently require AD approval, and I would prefer the default 
> to retain this.
> 
> Our impressions differ. I do not think and a WG chair can 
> make changes to an of those without an AD agreeing to it. 
> Further, even if you are right, I suspect some ADs would find 
> that to be an oversight, not a feature.

I believe an AD currently needs to approve milestone changes.
Assuming the changes are non-controversial, the AD typically approves
quickly.

> 
> > In section 5, I would like the linkage to i-ds to be linked 
> to completion of milestones as well, i.e. that if we have a 
> milestone of the form "Submit xxxx to IESG as proposed 
> standard", and that is associated with one or more i-ds, then 
> when the appropriate state change or changes occurs in the 
> tracker, then the milestone is automatically marked as 
> complete (subject to anything in section 4 of course). An 
> alternative would be to prompt the WG chair for completion.
> 
> 
> This makes a faulty assumption, that milestone text like 
> "Submit xxxx to IESG as proposed standard" is 
> machine-interpretable. Today, it is not, and probably cannot 
> be done easily. In some WGs I have been in, there has been 
> disagreement about whether such a milestone is met when the 
> submission starts, or when the the last @#$% DISCUSS is 
> finally cleared.

My impression is that this is met when the doc becomes offically
"publication requested".
That's when it is entered into iesg database tracking. YMMV.

> 
> Is such automation really needed? Shouldn't WG chairs be 
> looking at their milestones every month or so to see if any 
> need to be dispatched? A middle ground might be that when a 
> draft that is listed in a milestone moves between post-WG 
> states, mail is sent to the WG chairs noting that fact. You 
> can then decide when to propose a milestone change.

I think prompting the chair to mark the milestone complete provides
more flexibility and awareness.

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Sat, 7 May 2011 19:20:30 +0200

As regards the milestone changes, I've been sending date changes and indicates that the milestone has been completed direct to the IESG secretary for the last 6/7 years and there have never been queries as requiring AD approval.

All changes of the milestone text, all new milestones, and all deletions of milestones have been to the IESG secretary copied to the AD and the AD has approved.

As regards the automatic completions of milestones, I see two options.

1)	If adding the relationship between the deliverable and milestone means the milestone database has to be amended, then I see no real issue with adopting a specific type of milestone that in format that is tied to that deliverable and has a specific fields that relations to indicate the type of deliverable and the state the deliverable has to be in to allow completion to be declared. 

2)	If the structure of the existing milestone database is sacrosanct, then at least automatically give the chairs a notification and a link to update the linked milestone when the deliverable enters the publication request state.

As regards other comments from Paul, I fail to see why anybody would want a working group milestone to be tied to completion of any IESG action after a deliverable had been publication requested. Such actions are built into the ID tracker, are completely outside the control of the working group, and the ID tracker prompts the AD to make sure they are complete, and tells everyone how long the AD is taking to do that task. The milestones represent the actions the working group can be in control of. There are some valid arguments that there might be separate deliverable milestones respresenting getting the document ready for working group last call.

> From: David Harrington [mailto:[email-masked]]
> Sent: 07 May 2011 00:09
> To: 'Paul Hoffman'; DRAGE, Keith (Keith)
> Cc: 'IETF WG chairs'
> Subject: RE: NUDGE: WG milestones tool requirements:draft-ietf-genarea-
> milestones-tool-00.txt
> 
> Hi,
>

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Sat, 7 May 2011 20:27:27 +0100

Paul,

>>> be done easily. 

The current draft states

   The new tool is required to make mappings between milestones and
   Internet-Drafts explicit, and those drafts must be listed in views of
   the milestone.  This change will require a change to the Datatracker
   database to make such an association.

I don't expect that all possible variations of milestones and how they map to internet-drafts are machine interpretable but I my read of Keith's requirement is that many milestones are of the general form:

   Submit X specification to IESG as Y (draft-abc)

And if the database will have (when your draft is implemented) an association between a milestone and its associated drafts it seems a simple addition to have an associated property of "automatically mark milestone as DONE when all associated drafts have reached state publication requested". I guess the tool would also have to validate that the intended status of the submitted draft matched the intended status recorded in the milestone so the database would need another property for that.

It won't fully automate marking milestones as DONE and I don't think anyone is suggesting we need a milestone management tool that's capable of that but it would appear to be low hanging fruit that could be implemented at the same time as the other work your draft proposes.

>>> can then decide when to propose a milestone change.

Re: is such automation really needed. Personally I'm agnostic, having that automation would be useful for me as in my WG all milestones past & present are of the form described above and so I wouldn't ever have to send mail to the secretariat for milestone completions :-) Would it save me a bunch of work? not really, sending an e-mail is cheap but it is one of those things I tend to forget about.

Another thing I would find useful if it was there but wouldn't say was critical would be a reminder e-mail for an impending/overdue milestone. I wouldn't want to be nagged as milestones are often missed for good reasons but I also tend to be interrupt-driven to a certain extent and a gentle reminder would act as a nice interrupt for me.

Ben

---

From: Scott Brim <[email-masked]>
Date: Sat, 7 May 2011 16:15:21 -0400

This thread leads me to believe firmly that there will be many exception
states that cannot be covered by software or even operating procedures.
 Let's be sure that these remain tools for us, and not jailers.  To the
extent that they can make life easier, good, but if they start to get in the
way and make us less productive, it should be possible to revert to actual
human interaction, right?

---

From: John C Klensin <[email-masked]>
Date: Sat, 07 May 2011 17:35:04 -0400

--On Saturday, May 07, 2011 16:15 -0400 Scott Brim
<[email-masked]> wrote:

> This thread leads me to believe firmly that there will be many
> exception states that cannot be covered by software or even
> operating procedures.  Let's be sure that these remain tools
> for us, and not jailers.  To the extent that they can make
> life easier, good, but if they start to get in the way and
> make us less productive, it should be possible to revert to
> actual human interaction, right?

+1

As long as different WGs, and even different ADs or different
areas, use milestones differently, cases and exceptions will
arise that don't fit any particular automation model.
Automation of things than can conveniently be automated is fine,
but steps that reduce flexibility, force everything into the
same mold, or encourage substitution of rules and procedures for
thinking about what is happening are not, IMO, in the IETF's
best long-term interests.

   john

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 9 May 2011 19:32:17 -0700

> It won't fully automate marking milestones as DONE and I don't think anyone is suggesting we need a milestone management tool that's capable of that but it would appear to be low hanging fruit that…

Keith: does that work for you? Others: does this sound like something y'all want?

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 9 May 2011 19:33:58 -0700

> Another thing I would find useful if it was there but wouldn't say was critical would be a reminder e-mail for an impending/overdue milestone. I wouldn't want to be nagged as milestones are often m…

I remember hearing people argue vociferously in both directions on this in the past, but now that we have a tool update coming let's run at it again.

Do chairs want this? Hate this? Wouldn't care much either way?

--Paul Hoffman

---

From: Barry Leiba <[email-masked]>
Date: Mon, 9 May 2011 22:37:12 -0400

> Keith: does that work for you? Others: does this sound like something y'all want?

I'll be happy to have a tool that allows me to mark milestones as
complete.  I don't need it to be automatically tied to any particular
state change in a document.  *IF* we were to have that, the chair
would have to be able to specify which state change (in which
document) should trigger it.  That'd be fine, but it seems
unnecessary.

Barry

---

From: Barry Leiba <[email-masked]>
Date: Mon, 9 May 2011 22:42:02 -0400

> Do chairs want this? Hate this? Wouldn't care much either way?

If we can (have to) turn it on (and off), I don't mind it at all.
Chairs who want it can turn it on.

If it's there, it should probably be turnable onable in several stages:

   Notify of milestones:
   X  Due within one month.
   X  Due now.
   X  Overdue by one month.
   X  Overdue, repeat notice every month.

(And we can turn on any of the "X" marks we like; all are off by default.)

Barry

---

From: "David Harrington" <[email-masked]>
Date: Tue, 10 May 2011 07:58:30 -0400

Maybe an option that each chair can enable/disable, possibly on a
per-milestone basis? 

dbh

> Sent: Monday, May 09, 2011 10:34 PM
> To: Ben Niven-Jenkins
> Cc: IETF WG chairs
> Subject: Reminder email for impending/overdue milestones
> 
> > Another thing I would find useful if it was there but 
> wouldn't say was critical would be a reminder e-mail for an 
> impending/overdue milestone. I wouldn't want to be nagged as 
> milestones are often missed for good reasons but I also tend 
> to be interrupt-driven to a certain extent and a gentle 
> reminder would act as a nice interrupt for me.
> 
> 
> I remember hearing people argue vociferously in both 
> directions on this in the past, but now that we have a tool 
> update coming let's run at it again.
>

---

From: "David Harrington" <[email-masked]>
Date: Tue, 10 May 2011 08:00:43 -0400

I agree that a manual button to mark the milestone done would seem
sufficient.
Having it automated opens up a plethora of exceptions to deal with.

dbh

> [mailto:[email-masked]] On Behalf Of Barry Leiba
> Sent: Monday, May 09, 2011 10:37 PM
> To: Paul Hoffman
> Cc: IETF WG chairs
> Subject: Re: Adding automation for a common milestone
>

---

From: "David Harrington" <[email-masked]>
Date: Tue, 10 May 2011 08:06:33 -0400

I will observe that we have a certain amount of money to spend to
build these tools.
Is milestone automation and all this optional behavior the most
important thing to spend money on, or are there other more important
things to get chair-tools to support?

dbh 

> Sent: Monday, May 09, 2011 10:42 PM
> To: Paul Hoffman
> Cc: IETF WG chairs
> Subject: Re: Reminder email for impending/overdue milestones
>

---

From: Scott Brim <[email-masked]>
Date: Tue, 10 May 2011 09:41:53 -0400

+1

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 May 2011 09:42:14 -0400

--On Tuesday, May 10, 2011 08:06 -0400 David Harrington
<[email-masked]> wrote:

> I will observe that we have a certain amount of money to spend
> to build these tools.
> important thing to spend money on, or are there other more
> important things to get chair-tools to support?

Dave,

I read several of the comments, including mine, as saying:

	* Manual button almost certainly useful if it can be
	done without great cost.
	
	* More complexity and fine-tuning leads to more
	complexity and need for fine-tuning, i.e., we should not
	go down the path of trying to make this very automated
	and sophisticated unless we believe that such
	resource-intensive work it worth it... and that halfway
	is bad.

The case for doing the second still seems weak at best, at least
to me.

Is that consistent with what you are suggesting?

   john

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 May 2011 03:49:22 -0400

+1

--On Monday, May 09, 2011 22:42 -0400 Barry Leiba
<[email-masked]> wrote:

> If we can (have to) turn it on (and off), I don't mind it at
> all. Chairs who want it can turn it on.
>

---

From: Glen Zorn <[email-masked]>
Date: Wed, 11 May 2011 11:03:06 +0700

>> Having it automated opens up a plethora of exceptions to deal with.

I agree.

---

From: <[email-masked]>
Date: Fri, 13 May 2011 01:24:01 -0400

Paul,

I think a means of notifying WG chairs and ADs of changes is needed.
Is the atom feed sufficient for notification?  I would probably prefer to receive email.

It may be useful to allow WG secretaries to access the tool.  Making milestone
updates as part of putting out meeting minutes seems like a logical thing to do.

Section 4 lists 5 changes:

   o  create new milestones

And goes on to suggest that the default is that all require AD approval.

IMHO, marking a milestone complete should default to notifying the AD, not waiting for approval.  I routinely send these completion requests directly to ietf-action.  I would suggest that due date changes and Internet-Draft changes default similarly.  While I routinely check date with my AD before submitting, I think a notification to the AD would be sufficient to trigger a WTF??? inquiry to WG chairs who do something unexpected.  Regarding Internet-Drafts, at a minimum, I should not need to ask an AD's permission to associate a newly-submitted -00 WG draft with a milestone.

I would not spend the resources on automating marking of milestones as complete in response to draft state changes in the draft tracker.  This sort of milestone completion update is not that much work, and ADs should be encouraged to delegate this task to the WG chairs and/or secretary as appropriate.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Hoffman
> Sent: Wednesday, May 04, 2011 1:10 PM
> To: IETF WG chairs
> Subject: NUDGE: WG milestones tool requirements: draft-ietf-genarea-milestones-tool-00.txt
> 
> Greetings again. Following the lively discussion we had here a few weeks ago, the IAOC asked me to
> produce a new Internet Draft specifically about a tool for chairs to update their milestones. This
> has now been separated from the earlier work on updating charters.
> 
> After talking to some ADs and reading the earlier threads here, I put together
> <http://tools.ietf.org/html/draft-ietf-genarea-milestones-tool-00>. This is explicitly a first draft
> and is open for lots of changes, clarifications, and so on. In the earlier thread, there was wide
> disagreement on who should approve milestone additions and changes, so section 4 of the -00 is my
> first guess about what might work for most people, but I could have missed the mark.
>

---

From: Robert Sparks <[email-masked]>
Date: Fri, 13 May 2011 11:10:03 -0500

Paul -

The current draft paints a history (or at least a current state of things) that is just not true - if we're going to leave the description of the current state  in the draft (my preference),
please capture it more accurately with the next update.

In particular, it has long been the case for me as a chair and as an AD that the creation and modification (which includes deletion) of milestone text requires AD approval.
Changing dates, and marking things done, has not.

I would argue that if the tool allows different policies (which I think  the threads support), that the defaults represent that policy - I would be disturbed by a policy change inflicted by a tool design.

As I noted above, sometimes milestones are deleted - that needs to be added to the set of things the tool will support.

Thanks,

RjS

---

From: Robert Sparks <[email-masked]>
Date: Fri, 13 May 2011 11:19:18 -0500

The section on mapping documents to milestones should include the possibility of one draft mapping to multiple milestones
(I've seen groups present one document satisfying multiple milestones in the past).
The tool should support a many-many relationship.

RjS

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 18 May 2011 20:29:54 -0600

> Changing dates, and marking things done, has not.

+1 Keith and Robert on this

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 19 May 2011 08:09:02 -0600

If we are going to have the discussion about emails reminders (which I don't care about either way), I'd rather have the conversation about what these dates mean, and if they do more harm than good.