Subject: RE: Milestones

From: "Presuhn, Randy" <[email-masked]>
Date: Thu, 12 Dec 2002 13:14:14 -0600

Hi -

> From: Wijnen, Bert (Bert) [mailto:[email-masked]]
> Sent: Wednesday, December 11, 2002 16:03
> To: Fred Baker
> Cc: [email-masked]
> Subject: RE: Milestones
...
> AS far as I can tell, all the time I have been an AD, the
> milestones have never been used to kill/close a WG if they
> missed their milestones even by years. So I think out culture
> is not used to it. If we all want it, we can change that,
...

Missing milestones has been used in the distant past to
terminate WGs.  Remember the SNMPv2 and chassis MIB efforts.

 ------------------------------------------------------
 Randy Presuhn          BMC Software, Inc.  1-3141
 [email-masked]  2141 North First Street
 Tel: +1 408 546-1006   San José, California 95131  USA
 ------------------------------------------------------
 My opinions and BMC's are independent variables.
 ------------------------------------------------------

---

From: Randy Bush <[email-masked]>
Date: Thu, 12 Dec 2002 11:56:58 -0800

> terminate WGs.  Remember the SNMPv2 and chassis MIB efforts.

and did that help keep snmp from not making progress in five+
years?  it left a big unoccupied niche which cops, cli, ... are
now trying to fill.  snmp is a big reason i am sceptical about
the heavy-handed management philosphy marshall espouses.

randy

---

From: Marshall Rose <[email-masked]>
Date: Thu, 12 Dec 2002 12:24:53 -0800

> the heavy-handed management philosphy marshall espouses.

randy - if you'll recall the snmp paralysis occurred when two things
happened simultaneously:

    - a strong AD (me) stepped down and was replaced by someone a lot
      nicer (deirdre)

    - the four major contributors (me, case, kzm, waldbusser) had a
      fundamental, irreconcilable disagreement

either one of those is enough to cause serious problems. both of those
is enough to cause a nuclear winter.  of course, the SNMP momentum was
strong enough at that point that work continued to get done, albeit not
at the breakneck pace during the years when i was calling WG chairs on
saturday afternoons to remind them they were overdue on their
milestones...

as to whether my management philosophy is heavy-handed, that is, of
course, a matter of interpretation. everything i espouse is subject to
the usual appeals process, and even when i took extraordinarily
unpopular actions (e.g., saying "no" to standardizing an SNMP API), as
long as i:

    - telegraphed the action beforehand
    - gave folks an opportunity to express their perspectives
    - gave a thoughtful "ruling" in a timely fashion

then folks would grumble and go along. part of it, obviously, is that
folks want to know their input is heard and considered, and another part
of it is that folks don't mind having things decided so they can move
on.

not all styles work for all people in all situations, so there is plenty
of room for perspectives on how an AD should manage each situation that
arises...

/mtr

---

From: Fred Baker <[email-masked]>
Date: Thu, 12 Dec 2002 12:26:54 -0800

At 11:56 AM 12/12/2002 -0800, Randy Bush wrote:
>years?

Let's keep on a single topic. Marshall suggested a specific approach to
managing working groups. I suggested another which is far more flexible but
which none-the-less calls for some form of management by objective. You're
talking about whether a many-working-group-decade-and-a-half effort would
have been different. Listen, whatever the giant-picture issues might have
been during the lifetime of SNMP, the working groups have by-and-large
produced their chartered results on a schedule.

Back to your question: no; what caused that was the complete disinterest of
the NM community in designing a reasonable product (my viewpoint, not news
to anyone). They finally addressed some of the biggest issues in SNMP V3,
and in the dilbert-vision discussion have finally asked the operators what
it was that they wanted in a network management protocol.

---

From: Randy Bush <[email-masked]>
Date: Thu, 12 Dec 2002 20:30:30 -0800

let's try it another way.  my point is that a short term push to
meet milestones can contribute seriously to not making longer term
goals.

aaa was going nowhere fast.  we installed new chairs and gave them
a one year timeline to define requirements and choose a protocol.
we made made sure there were no obstacles, pushed hard, got them
help, cajoled, blah blah blah.  they made the goal.  did a darned
good job of it too.

but then it has taken two more years to get the documents out, and
they still are not out people were really tired from the short-term
push, many dropped out, ...

today, as we need to get the first big doc set out, we get email
from the chair on a document in wg last call

    So far, we have not had a single comment on this draft -- which
    only 4 people indicated they had read at IETF 55. Note that the
    ability of AAA WG to take on new work items is in part
    constrained by our ability to finish old ones.

and folk such as 3gpp are being nice but nearing the end of their
rope waiting for some specific help and new documents, to which the
(excellent and dedicated) aaa chair has said

    My concern is that the resources of AAA WG has dwindled down to
    a small set up of people who are doing all the work - basically
    the WG chairs, [principal editor, another hero] and maybe one
    or two other people who can be called upon for special
    occasions.  This is the sign of a WG that is on a path to
    closure - not something we should be planning on revving up for
    another run at the gold medal.

    With some juggling, this core group *might* be able to complete
    Base and NASREQ soon, then EAP and maybe Credit Control by June
    03. I'm running out of ideas for getting MobileIPv4 and CMS
    completed in a timely way, though these were among the primary
    motivations for creating Diameter in the first place.

imiho, there are a number of causes for this likely failure to
achieve long-term goals

  o big goals (but you should have seen the plan to boil the ocean
    before we scaled it down radically)

  o burnout from the short-term run

  o complexity that could not be kept out of the protocol (a good
    case was made for the need)

  o a bad actor kept disrupting

and i really think the short-term run burnout played a major part.

it's now many months late and in a state that creating any new
deadlines would just be pretend.  but a lot of folk are counting on
the results.  should we close it down?  should we make yet another
set of new dreadlines and then shut it down when it does not make
them?

in the real world things are just not as simple as we might like.

randy

---

From: Jari Arkko <[email-masked]>
Date: Fri, 13 Dec 2002 11:48:17 +0200

> set of new dreadlines and then shut it down when it does not make

For the specific situation: we should just get the base spec
out of the door (which I believe is happening) and then observe
the situation again. I think part of the problem with completing
all the rest of the items on the agenda in this particular case
was that the world didn't stop while the IETF worked on the
protocol. Some things are still quite interesting (the base for
instance), some apps on top of it may not be as interesting any
more, and some news ones have appeared. We need to consider the
goals wrt the current state of the world, not the one that existed
when the WG was created.

Anyways, I wasn't going to write so much about the specific
situation. What can we learn from Randy's story? I think the
effect you are seeing is that people are "tired of the war".
Lots of people contribute on a new and sexy topic. If the job
looks more like a permanent position, not a lot of people want
that. Loss of editors or key people is a serious setback, and we
lost a key editor in the middle of it. You do it too long, it is
simply very probable that some of your key folks are going to
get a new job and do something else. So the lessons for management
are: 

- Don't spend your people on burn out, control the amount of
  push. Also need to worry about wounds from big fights.
- Keep your next deliverables close enough in time (1-2 yr max)
  so that you can actually get folks to do them, not lose anyone,
  and avoid getting tired.
- Produce *some* results early to create a positive
  feeling that the group has actually achieved something
  concrete. Requirements RFCs may not be enough for this.
- Avoid running WGs during economic downturns when everyone's
  feeling depressed about deploying any new stuff and every
  vendor's folks are told that "probably someone else will
  work on the standards".
- If someone vendors or SDOs want you to do something,
  ask something back in return, such as a promise of
  additional (competent) people.

But all the above is management stuff. I'm really more interested
in the technical stuff. And it seems we have been making the
same serious technical mistake in many cases. We are producing
too big base protocol definitions. Diameter, SIP, Mobile IPv6...
all have triple digit page numbers. Why? As I see it there are
a few reasons:

- We require a complete solution before it can be fully
  reviewed and accepted
- Lots of people want their toy thing in and argue loudly (all
  protocols, e.g. MIPv6)
- No reason, just decided to merge a few documents (happened for
  Diameter)
- The Internet is bigger today and our stuff is used for
  serious work. Everything needs to be scalable, secure,
  etc.

I think we need to be more careful about this in the future.
Could we have made Diameter base just format, capabilities
exchange, and hbh security? Does anyone have suggestions on
how to improve our process regarding complete solutions? As
engineers we KNOW that incremental approach is much easier
to work with and there are no big surprises at the end. But
how can that work with e.g. mandatory security requirements,
complying with all functional requirements the WG has decided
upon etc?

Also, how should we manage the "toy thing" issue? I see
it often, and usually I can't decide whether to say NO
and piss off some key person or contributor, or to say YES
and increase the length of the spec. Sometimes that ADs or chairs
can act as the "bad cop" so the relationships in the WG itself
stay good, but we can't do it all the time.

Jari

---

From: Erik Guttman <[email-masked]>
Date: Fri, 13 Dec 2002 13:07:37 +0100 (CET)

> push, many dropped out, ...

The effort was anything but short term.  >1 year to get permission to
hold a BOF.  1 year avoiding the issue of choosing a protocol and
defining requirements.  1 year defining requirements and choosing a
protocol.  Six months of defining problems and solutions, getting
Diameter to address all requirements. Then, with *a very ambitious set
of requirements,* the real work got started. 

Implementors and experts of several stripes started banging on Diameter,
hard, over an extended period of time.  After more than 2 years of this,
(for radius/roamops experts, we are really 6-7 years into the process)
yes, people ARE burned out.

AAA is a bad model to use when assessing our problems.  I submit the
undesirable burnout, slow progress, complex protocol, etc is the
inevitable result of the process which Diameter underwent to justify
itself.  Thankfully this is not the procedure we commonly use to charter
WGs. 

Erik

---

From: Jari Arkko <[email-masked]>
Date: Fri, 13 Dec 2002 15:00:32 +0200

> itself.  Thankfully this is not the procedure we commonly use to charter

Agree with this too.

Jari

---

From: Edward Lewis <[email-masked]>
Date: Fri, 13 Dec 2002 10:31:43 -0500

Wow, a lot to read.

I've not had time to read and grok all of the details of this thread, 
much less the other threads recently and still keep up with other 
work (like chairing my group and keeping my employer happy).  So 
please forgive me if my opinion here is nothing new to the thread.

There is no panacea to effective management.  It takes skill, 
dedication, and work.  No set of tools are going to change that, not 
even milestones.  Tools can make the job of management easier but 
they don't remove the need to work at it.

My first problem with engineers making milestones is that we often 
take it for granted.  If only I had a /24 for every time a project 
manager showed me a project management chart that had me behind 
schedule within a week of the project start.  We need to be much more 
realistic in setting milestones.

One problem in setting realistic milestones is knowing how to predict 
the time something will take.  An unrealistic setting is usually a 
symptom of either an overly anxious chair, an overly optimistic 
chair, or dealing with a topic that is too new to predict. 
Obviously, more experienced chairs and chairs more likely to say "no" 
would take care of the first two causes.  The third is addressed by 
moving half-baked ideas away from the ready to inter-operate ideas. 
(I.e., use the IRTF more.)

Once we have realistic milestones, the asset they provide is a 
defense against someone (or group) sidetracking the process and 
against losing focus.  With a completely open process, it doesn't 
take a lot of energy to provide a disruption to the smooth flow of a 
WG.  A solid charter and a set of clear milestones provide the sense 
of rigidity that is the best backdrop for unrestrained discussion.

Finally, getting people to treat the milestones with any sense of 
respect means that chairs have to make the effort to manage the 
milestones.  It is excusable that priorities shift, funding lapses, 
volunteers take breaks.  It is not excusable to refuse to shift 
milestones in light of this.

Make milestones reasonable.  Use milestones as a resource.  Keep 
milestones realistic.  But please don't rely on them as a crutch.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                          +1-703-227-9854
ARIN Research Engineer

---

From: Randy Bush <[email-masked]>
Date: Fri, 13 Dec 2002 10:28:11 -0800

do not confuse the life of diameter with that of the aaa wg.  in
fact, as far as i could determine, the aaa wg was spun up with the
hidden but major goal to do anything but diameter, and no other
clear goals.  this lack of positive goal set probably led to the
boiling the ocean game i found when i inherited the ops area.

so, we made major changes, found new chair(s), moved the ocean
boiling off to the irtf, and made a short schedule of requirements
and protocol evaluations against those requirements.  to address
those requirements, diameter was seriously modified and thereby
succeeded in winning the beauty contest.

this worked well, was a reasonable schedule, was achieved, ...

but from that point to now, things went to mush.  as jari says, the
job was large, possibly too large.  to be really usable, it turned
out that diameter needed very serious work.  perhaps we should have
just worked on the base and to heck with the other needed pieces.
hard to tell.

but we're trying to close the game now, and it is neither obvious
nor easy.

randy

---

From: Randy Bush <[email-masked]>
Date: Fri, 13 Dec 2002 10:41:56 -0800

> One problem in setting realistic milestones is knowing how to
> predict the time something will take.  An unrealistic setting is
> usually a symptom of either an overly anxious chair, an overly
> optimistic chair, or dealing with a topic that is too new to
> predict.  Obviously, more experienced chairs and chairs more
> likely to say "no" would take care of the first two causes.  The
> third is addressed by moving half-baked ideas away from the ready
> to inter-operate ideas.

one thing we keep ignoring is that, in our day jobs, the manager is
responsible for ALL of the managee's time.  in the ietf, the more
productive folk tend to be active in multiple wgs, and there is no
overall balancing of the multiple schedules.  not that i think
there can be.  but this is the old multi-project matrix management
philosophy that is less and less in fashion in daytime jobs.

randy