Subject: Dated milestones considered... pointless?

From: Lucas Pardue <[email-masked]>
Date: Thu, 24 Feb 2022 03:07:47 +0000

Hello fellow WG chairs,

Some time back before I was a WG chair, the datatracker added a feature to
support a list of ordered but undated milestones.

For the QUIC QG, my co-chair and I have been discussing with our
responsible AD about possibly switching from the dated list to the undated
list. It's been interesting to see the different perspectives on this, even
in the small group. The official literature about IETF milestones seems
fairly scant and it seems likely that many interpretations of the purpose
of milestones exist.

Rather than focus on the philosophical nature of the matter. I thought it
might be interesting to just grab some numbers about WG milestones by
querying the Datatracker API [1] [2] and gathering some stats.

These are some preliminary results, so take them with a pinch of salt in
case my methodology is wrong. I queried all 123 active WGs listed
(including things like dispatch WGs), filtering for active milestones that
had no resolution. For groups that do use dated milestones, I compare the
due date to today's date and record the delta in days. If the due date
earlier than today, it is recorded as late.

Here is a summary of the findings:

total WGs 123
total WGs using dates 119
  - total WGs using dates but no active and unresolved milestones are
listed 18
total WGs not using dates 4
global total active and unresolved milestone count 502
global total active and unresolved late milestone count 408
global late milestone % 81.27490039840637

==Late milestone statistics (days)==
Min 24
p25 300.0
p50 939.0
p75 1669.0
p90 2581.0
p95 3161.0
Max 4287

Assuming my methodology is not totally broken. When > 80% of our collective
milestones are late, with a median lateness of ~2.5 years, I'm inclined to
think that dated milestones are pointless. This analysis of course also
overlooks all the milestones that *should* have been added but have not;
something we are guilty of in QUIC (but which we are deferring somewhat
based on the decision of the milestone format).

Curious to hear what the rest of WG chairs think!

Cheers,
Lucas

[1] - https://datatracker.ietf.org/api/
[2] - more specifically
https://datatracker.ietf.org/api/v1/group/groupmilestone/

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 23 Feb 2022 22:17:50 -0500

Thank you.  I thought I knew most of the branches of our tools.  I had 
not realized there was provision for undated milestones.

Your data matches my experience and my understanding.  The dates we put 
down are usually meaningless.  I have seen a couple of WGs where the 
dates were near term, intended, and at least close to met.  But as I 
understand, and from your report, is very rare.

The reasons for missing the dates are varied and complex.  Based on the 
dates my WGs have missed, the dates are set in good faith, and missed 
for good reasons.  But it seems to me that if they are almost always 
missed, what are we setting them for?

Yours,
Joel

On 2/23/2022 10:07 PM, Lucas Pardue wrote:
> Hello fellow WG chairs,
> 
> Some time back before I was a WG chair, the datatracker added a feature 
> to support a list of ordered but undated milestones.
> 
> responsible AD about possibly switching from the dated list to the 
> undated list. It's been interesting to see the different perspectives on 
> this, even in the small group. The official literature about IETF 
> milestones seems fairly scant and it seems likely that many 
> interpretations of the purpose of milestones exist.
> 
> Rather than focus on the philosophical nature of the matter. I thought 
> it might be interesting to just grab some numbers about WG milestones by 
> (including things like dispatch WGs), filtering for active milestones 
> that had no resolution. For groups that do use dated milestones, I 
> compare the due date to today's date and record the delta in days. If 
> the due date earlier than today, it is recorded as late.
> 
> Assuming my methodology is not totally broken. When > 80% of our 
> collective milestones are late, with a median lateness of ~2.5 years, 
> I'm inclined to think that dated milestones are pointless. This analysis 
> of course also overlooks all the milestones that *should* have been 
> added but have not; something we are guilty of in QUIC (but which we are 
> deferring somewhat based on the decision of the milestone format).
> 
> [1] - https://datatracker.ietf.org/api/ <https://datatracker.ietf.org/api/>
> [2] - more specifically 
> <https://datatracker.ietf.org/api/v1/group/groupmilestone/>

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Thu, 24 Feb 2022 09:24:57 +0000

>    missed, what are we setting them for?

Or rather, why are they not updated to have at least a vague idea of truth ? Which is what I suggest to the chair of "my" WG.

Bottom line: having hard milestones in a volunteer-driven organism is probably best-effort.

-éric

-----Original Message-----
From: WGChairs <[email-masked]> on behalf of "Joel M. Halpern" <[email-masked]>
Date: Thursday, 24 February 2022 at 04:18
To: Working Chairs <[email-masked]>
Subject: Re: Dated milestones considered... pointless?

    Yours,
    Joel

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 24 Feb 2022 11:51:55 +0100

> Or rather, why are they not updated to have at least a vague idea of truth ? Which is what I suggest to the chair of "my" WG.

I think we need to make this a scheduled, rhythmic activity.
E.g., all milestones are updated/confirmed three weeks after each IETF.
(Says he who no longer is a WG chair.  
But doing regular schedule checks for RG milestones would not be wrong, either.)

Grüße, Carsten

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 24 Feb 2022 12:39:57 +0000

Hey, my conjecture is that if the majority of milestones at a point in time
are late, the process is completely flawed. Since no one seems to be
complaining loudly that the milestones are on average 2.5 years late, I
just fail to see their relevance. What is their point or purpose?

Dates in a volunteer-led org are best effort. My own feeling is that the
effort it takes to pick dates and eternally move them around is
non-negligible and a waste of time and effort. Undated milestones remove
most of the pain in this respect.

I didn't take my analysis further but it would be possible it also look at
the pattern of how dated milestones are right-shifted. I suspect, based on
current evidence, most time estimates are picked according to some form of
game theory and then just punted or ignored and forgotten.

Cheers
Lucas

On Thu, 24 Feb 2022, 10:53 Carsten Bormann, <[email-masked]> wrote:

> > Or rather, why are they not updated to have at least a vague idea of
> truth ? Which is what I suggest to the chair of "my" WG.
>

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 24 Feb 2022 13:04:26 +0000

On Thu, 24 Feb 2022, 12:39 Lucas Pardue, <[email-masked]> wrote:

> Hey, my conjecture is that if the majority of milestones at a point in
> time are late, the process is completely flawed. Since no one seems to be

To add. I didn't deeply analyse what the milestone actually are. Since they
are a free form text field, analysis is a bit tricky.

My straw poll is that most milestones were either adopt a document, publish
a draft, or submit to IESG. We all know that there is a stark detachment
between submission to IESG and a document actually getting published. The
datatracker is actually pretty decent at tracking document stages and
timelines, so why do we feel compelled to set and forget milestones in some
other bucket?

Cheers
Lucas

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 24 Feb 2022 14:14:18 +0000

I believe the point of dated milestones is to prevent a WG from languishing and just going on forever without getting things done. I am sure we can each think of one or more examples on our own.   It seems to have been a pervasive problem, so much so that the Tao – a document we recommend to all newcomers – used to say “drag on for decades” (I removed that; waiting for publication on www.ietf.org<http://www.ietf.org> soon), and it’s RFC predecessor had the phrase “degenerative working group syndrome” [1]

“Once burned, twice shy” comes to mind. I don’t think dated milestones are an effective way of keeping a WG moving forward any more, and the IESG has enough oversite to handle things if they go awry (or, more accurately, don’t go at all :)

The IETF is, for the most part, software engineers. As a field, we are horrible at predicting schedules. Others have pointed out this is a volunteer activity which adds to the problem.

I am completely in favor of making undated be the default.

[1] https://datatracker.ietf.org/doc/html/rfc4677#section-5.1

From: Lucas Pardue <[email-masked]>
Date: Thursday, February 24, 2022 at 7:40 AM
To: Carsten Bormann <[email-masked]>
Cc: 'WG Chairs' <[email-masked]>
Subject: Re: Dated milestones considered... pointless?

Hey, my conjecture is that if the majority of milestones at a point in time are late, the process is completely flawed. Since no one seems to be complaining loudly that the milestones are on average 2.5 years late, I just fail to see their relevance. What is their point or purpose?

Dates in a volunteer-led org are best effort. My own feeling is that the effort it takes to pick dates and eternally move them around is non-negligible and a waste of time and effort. Undated milestones remove most of the pain in this respect.

I didn't take my analysis further but it would be possible it also look at the pattern of how dated milestones are right-shifted. I suspect, based on current evidence, most time estimates are picked according to some form of game theory and then just punted or ignored and forgotten.

Cheers
Lucas

On Thu, 24 Feb 2022, 10:53 Carsten Bormann, <[email-masked]<mailto:[email-masked]>> wrote:

Grüße, Carsten

---

From: Warren Kumari <[email-masked]>
Date: Thu, 24 Feb 2022 09:53:41 -0500

On Thu, Feb 24, 2022 at 9:14 AM Salz, Rich <rsalz=
[email-masked]> wrote:

> I believe the point of dated milestones is to prevent a WG from
> languishing and just going on forever without getting things done. I am
> sure we can each think of one or more examples on our own.   It seems to
> have been a pervasive problem, so much so that the Tao – a document we
> recommend to all newcomers – used to say “drag on for decades” (I removed
> that; waiting for publication on www.ietf.org soon), and it’s RFC
> predecessor had the phrase “degenerative working group syndrome” [1]
>
>
>
> “Once burned, twice shy” comes to mind. I don’t think dated milestones are
> an effective way of keeping a WG moving forward any more, and the IESG has
> enough oversite to handle things if they go awry (or, more accurately,
> don’t go at all :)
>
>
>
> The IETF is, for the most part, software engineers. As a field, we are
> horrible at predicting schedules. Others have pointed out this is a
> volunteer activity which adds to the problem.
>
>
>

Meh. I don't really care what the default is, but one of the strengths[0]
of the IETF is that our laws are more like guidelines.
“The code is more what you’d call ‘guidelines’ than actual rules.”
    – Barbossa, Pirates of the Caribbean
or https://i.kym-cdn.com/photos/images/original/001/342/616/3ba.jpg

Different areas and working groups have different ways of operating -- I
personally like undated milestones; or dated ones with the understanding
that they are aspirational, not deadlines[1], but I also understand that
some WGs really like using them to drive work forward, and that the chairs
/ ADs point and them and make grumpyface to more documents along.

So, meh on defaults, but I'd be sad if we removed the ability to set a
date...

[0]: When things are going well. When not, it's a liability :-P
[1]: “I love deadlines. I like the whooshing sound they make as they fly
by.” - Douglas Adams

> *From: *Lucas Pardue <[email-masked]>
> *Date: *Thursday, February 24, 2022 at 7:40 AM
> *To: *Carsten Bormann <[email-masked]>
> *Cc: *'WG Chairs' <[email-masked]>
> *Subject: *Re: Dated milestones considered... pointless?
>
>
>

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Thu, 24 Feb 2022 14:55:43 +0000

We have a similar problem with schedules in MEF. Projects in MEF are similar to WGs in IETF in that they have a charter that includes a schedule/milestones that are set at the time the project starts. As in IETF, the target dates are typically very optimistic/best case. In an attempt to address this, we have a brief status update on each project at each quarterly meeting. Anytime a project falls more than 3 quarters behind schedule, a re-charter is triggered. This results in a discussion of whether the project is off the rails or simply moving more slowly than expected. The project is either cancelled or rechartered with a new schedule.

We have been doing this for a little over a year now and it seems to be working. In addition to keeping schedules relatively up to date, it encourages more realistic schedules when (re)chartering.

Cheers,
Charles

On Feb 24, 2022, at 8:14 AM, Salz, Rich <rsalz=[email-masked]<mailto:rsalz=[email-masked]>> wrote:

I believe the point of dated milestones is to prevent a WG from languishing and just going on forever without getting things done. I am sure we can each think of one or more examples on our own.   It seems to have been a pervasive problem, so much so that the Tao – a document we recommend to all newcomers – used to say “drag on for decades” (I removed that; waiting for publication on www.ietf.org<http://www.ietf.org/> soon), and it’s RFC predecessor had the phrase “degenerative working group syndrome” [1]

From: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Date: Thursday, February 24, 2022 at 7:40 AM
To: Carsten Bormann <[email-masked]<mailto:[email-masked]>>
Cc: 'WG Chairs' <[email-masked]<mailto:[email-masked]>>
Subject: Re: Dated milestones considered... pointless?

Cheers
Lucas

Grüße, Carsten

---

From: Lixia Zhang <[email-masked]>
Date: Thu, 24 Feb 2022 07:34:26 -0800

> On Feb 24, 2022, at 4:39 AM, Lucas Pardue <[email-masked]> wrote:
> 
> Hey, my conjecture is that if the majority of milestones at a point in time are late, the process is completely flawed.

I would say the process has an issue, that needs to be clearly identified and fixed.

> Since no one seems to be complaining loudly that the milestones are on average 2.5 years late, I just fail to see their relevance. What is their point or purpose?

as the saying goes, "Plans can't keep up with changes"--for plans that set once and never change of course fall further behind.

Personally I believe that's not a suggestion of abandoning the plan entirely. Rather, lets all understand this: 
- a plan is made based on our limited understanding (of many involved issues)
- on top of that, the world changes, unforeseen happens
- all the above simply saying we need to regularly update a plan (ALSO important to document, as far as we can identify, what are the changed conditions that resulted in the plan delay--this would help us continuously improve our plan making for the future)

the essence of a control system is a feedback loop. 
A periodic re-assessment/adjustment is to create a feedback loop for our process

just my 2 cents, Lixia

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 24 Feb 2022 16:54:07 +0100

On 2022-02-24, at 15:53, Warren Kumari <[email-masked]> wrote:
> 
> Different areas and working groups have different ways of operating

Yes!

> -- I personally like undated milestones; or dated ones with the understanding that they are aspirational, not deadlines[1],

They are plans.
“Kein Plan überlebt den ersten Feindkontakt”! [2].

> but I also understand that some WGs really like using them to drive work forward, and that the chairs / ADs point and them and make grumpyface to more documents along.

They are a chairs’ tool, indeed.
But they also can give some indication to other groups that might want to use the planning info for their own planning.

I stand by my suggestion that milestones should expire mid-April, mid-August, and mid-December, unless recently updated or refreshed by the chairs with AD consent.  Chairs should get “expiring soon” mails like authors get for their I-Ds.

Grüße, Carsten

[2]: Actually, Moltke said “Kein Operationsplan reicht mit einiger Sicherheit über das erste Zusammentreffen mit der feindlichen Hauptmacht hinaus”.  Well, he was German, like me.

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 24 Feb 2022 15:55:49 +0000

He Lixia,

On Thu, Feb 24, 2022 at 3:34 PM Lixia Zhang <[email-masked]> wrote:

> time are late, the process is completely flawed.
>
> > Since no one seems to be complaining loudly that the milestones are on
> average 2.5 years late, I just fail to see their relevance. What is their
> point or purpose?
>
> as the saying goes, "Plans can't keep up with changes"--for plans that set
> once and never change of course fall further behind.
>
> Personally I believe that's not a suggestion of abandoning the plan
> entirely. Rather, lets all understand this:
> - all the above simply saying we need to regularly update a plan (ALSO
> important to document, as far as we can identify, what are the changed
> conditions that resulted in the plan delay--this would help us continuously
> improve our plan making for the future)
>

Status tracking and a control loop are good things, we should encourage
that. I just don't see how using the datatracker to do so is effective. The
WGs I participate in are doing well without relying on datatracker
milestone dates e.g. HTTP, HTTPAPI, MASQUE, QUIC (to some extent). The data
analysis indicates that other WGs I don't paticipate in also mostly ignore
milestone dates.

I might be more convinced by dated milestones if the tools we had to manage
them didn't just delete old ones. There's really no way to get, at a
glance, how healthy the status of things are. If participants need to track
status closely themselves, then low fidelity mtlestone are little value.

Personally, I think I have better places to invest my IETF time than
playing milestone date negotiation/brokering every quarter. Instead, I
invest that time work with ADs and authors to understand progression,
blockers, risks, etc.

Cheers,
Lucas

---

From: Robert Sparks <[email-masked]>
Date: Thu, 24 Feb 2022 10:08:02 -0600

My first reactions, and points to consider.

I have strong outlying opinions here, and they color my choice of language:

With a small number of exceptions, milestones are simply ignored. As 
others have pointed out, _some_ groups use them, probably to good 
effect. But as others have pointed out, different groups are managed 
differently, and as your stats show (IMO), for most groups establishing 
these is just a ritual that is occasionally performed to make someone 
feel good, but not to really help the group get its work done. For those 
groups, I argue that the milestones are a harmful distraction and should 
_go away_. The rest of the world gets a better sense of what's happening 
by looking at the dates on the active drafts.

Groups that use milestones well should continue to have good support for 
using them.

Groups that use milestone dates should continue to have good support for 
using dates. Not all groups should be forced to supply dates.

The default should move to dateless, or better to _no_ milestones. 
Milestones should only exist when and AD and chairs agree that they will 
help.

RjS

On 2/23/22 9:07 PM, Lucas Pardue wrote:
> Hello fellow WG chairs,
>
> Some time back before I was a WG chair, the datatracker added a 
> feature to support a list of ordered but undated milestones.
>
> undated list. It's been interesting to see the different perspectives 
> on this, even in the small group. The official literature about IETF 
> it might be interesting to just grab some numbers about WG milestones 
> by querying the Datatracker API [1] [2] and gathering some stats.
>
> These are some preliminary results, so take them with a pinch of salt 
> in case my methodology is wrong. I queried all 123 active WGs listed 
> I'm inclined to think that dated milestones are pointless. This 
> analysis of course also overlooks all the milestones that *should* 
> have been added but have not; something we are guilty of in QUIC (but 
> which we are deferring somewhat based on the decision of the milestone 
> format).
>

---

From: Warren Kumari <[email-masked]>
Date: Thu, 24 Feb 2022 11:31:45 -0500

> Milestones should only exist when and AD and chairs agree that they will

I'm not sure that these are "outlying opinions" at all -- they seem to
reflect how things are currently done.
Yes, the defaults might be different, but ¯\_(ツ)_/¯.

I think that where we may diverge is if (dated or undated) milestones are a
"harmful distraction" - if a tree falls in a forest and no-one notices, has
it distracted anyone?

One thing that I think is worth knowing is that the IESG generally likes
having milestones when a new WG is formed -- it provides a sense of "is the
WG actually accomplishing what it hoped to do?"

In addition, as an AD, it can be hard to shut a working group; even if it
isn't making progress, there are at least some participants who'd like it
to stick around, and will argue against it closing. Being able to point at
milestones and say "Y'all said that you'd do X, Y, Z by 2013 and it's now
2022. You haven't done any of those..." can be very helpful -- it changes
some of the tone of discussion from "You just don't like us / our
technology / <whatever>" to something more metric driven and less
personality driven.

W

---

From: Donald Eastlake <[email-masked]>
Date: Thu, 24 Feb 2022 13:29:16 -0500

I consider dated milestones to be of some utility. Undated milestones seem
sort of like general deliverables goals which are commonly in the WG
Charter. Whenever the group I currently chair has met, I have presented a
slide on milestones status. They can provide some focus.

Thanks,
Donald
===============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 2386 Panoramic Circle, Apopka, FL 32703 USA
 [email-masked]

On Thu, Feb 24, 2022 at 11:32 AM Warren Kumari <[email-masked]> wrote:

> I think that where we may diverge is if (dated or undated) milestones are
> a "harmful distraction" - if a tree falls in a forest and no-one notices,
> has it distracted anyone?
>

---

From: David Schinazi <[email-masked]>
Date: Thu, 24 Feb 2022 13:28:00 -0800

Some of my WG milestones are 4 years late, and nothing bad has happened. I
don't see value in dated milestones since no one makes plans based on those
dates and no one rushes to meet these "deadlines". I'd be supportive of
removing them entirely. If some folks like them, then I'm happy to switch
my WGs to undated instead.

David

---

From: Michael Richardson <[email-masked]>
Date: Thu, 24 Feb 2022 16:47:14 -0500

I find that the need to get AD approval for milestones makes them essentially
useless for WG Chairs.

I would be good with revising all milestones the month after each IETF with
AD.  That would also mean that ADs and WG chairs would be meeting at least
three times/year.

For many (otherwise) functional WGs, ADs tend to give WG chairs lots of rope.
Perhaps we should practice "trust, but verify" here, and if WG chairs are
doing well, then the three-times meeting can be five minutes.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 24 Feb 2022 22:59:07 +0100

On 24. Feb 2022, at 19:29, Donald Eastlake <[email-masked]> wrote:
> 
> Whenever the group I currently chair has met, I have presented a slide on milestones status.

Randomly chosen historic example:

https://datatracker.ietf.org/meeting/99/materials/slides-99-core-consolidated-slides-08#page=9

(Green = good, orange = not so good, I forget the other colors but I said something to them.
I love the orange ones; three of them were wildly optimistic...)

Grüße, Carsten

---

From: Luigi Iannone <[email-masked]>
Date: Fri, 25 Feb 2022 15:52:37 +0100

In LISP we do something similar,

We use a color code to visually indicate how close a WG document is to WGLC.
https://datatracker.ietf.org/meeting/108/materials/slides-108-lisp-chairs-slides-00.pdf <https://datatracker.ietf.org/meeting/108/materials/slides-108-lisp-chairs-slides-00.pdf>

The idea being to make the WG aware where the efforts should be put.

For the milestones… we are in a similar situation like the WG of David Schinazi.. 4 years behind schedule.
Agree with other on the fact that dates are strictly necessary. Milestones can be used to organize the work.

In the LISP specific case I am not sure that a regular update after each IETF would have been that useful.
We were slow in progressing the main documents in-between meeting, after receiving area review or feedback from IESG telechat, often changing completely the work-plan, hence, a date on the milestones would have been still far from the latest reality anyway.

Yes, the solution is to update the milestones as soon as it is needed, but let’s face it (in general) we don’t.

Ciao

L.

> On 24 Feb 2022, at 22:59, Carsten Bormann <[email-masked]> wrote:
>