Subject: Input solicited: IETF problem pinpointing at the plenary?

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 30 Oct 2002 10:23:43 +0800

Hello, folks!

The present IESG plenary agenda shows an one-line item:

 "The IETF organization - status, challenges and future directions" - 
Harald Alvestrand

What I will be trying to do with this presentation is to review how the 
IETF is working - what is working well, what is working less well, and what 
are problems that we really have to do something about.

Don't take me wrong - I think the work we are doing is important, we 
actually get quite a lot of it done, and what we produce is generally 
useful (sometimes VERY useful) to the Internet community.

But many people have told me that they percieve problems in the current 
process, problems with how we choose what to work on and how we choose to 
work on those things; problems with how we resolve conflicts, and problems 
with how we do the management and quality control of the specifications we 
produce.

One such input is draft-huston-ietf-pact-00; there have been other, less 
formal ones.

When chatting with Marshall Rose about this draft, we came up with the idea 
of trying to embed some of the input from the community within the 
presentation - turning it somewhat from a "presentation by me" into a kind 
of community panel.

If we're going to do that, we need a few (3-5?) volunteers who can take a 
stab at drafting 2-3 slides each identifying what they see as problems and 
possible ideas for solutions, huddle together for a while in Atlanta to get 
the slides to match up, and speak the words at the plenary.

What do you think of the idea? Who wants to work with me on this?

                    Harald

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Tue, 29 Oct 2002 19:13:46 -0800

I'm willing to take a stab....

Kurt

At 06:23 PM 2002-10-29, Harald Tveit Alvestrand wrote:
>"The IETF organization - status, challenges and future directions" - Harald Alvestrand
>
>What I will be trying to do with this presentation is to review how the IETF is working - what is working well, what is working less well, and what are problems that we really have to do something about.
>
>Don't take me wrong - I think the work we are doing is important, we actually get quite a lot of it done, and what we produce is generally useful (sometimes VERY useful) to the Internet community.
>
>But many people have told me that they percieve problems in the current process, problems with how we choose what to work on and how we choose to work on those things; problems with how we resolve conflicts, and problems with how we do the management and quality control of the specifications we produce.
>
>One such input is draft-huston-ietf-pact-00; there have been other, less formal ones.
>
>When chatting with Marshall Rose about this draft, we came up with the idea of trying to embed some of the input from the community within the presentation - turning it somewhat from a "presentation by me" into a kind of community panel.
>
>If we're going to do that, we need a few (3-5?) volunteers who can take a stab at drafting 2-3 slides each identifying what they see as problems and possible ideas for solutions, huddle together for a while in Atlanta to get the slides to match up, and speak the words at the plenary.
>

---

From: Aaron Falk <[email-masked]>
Date: Tue, 29 Oct 2002 19:56:02 -0800

> What do you think of the idea? Who wants to work with me on this?

I'll help out.

---aaron

---

From: Dean Willis <[email-masked]>
Date: 30 Oct 2002 00:34:01 -0600

Ah yes, an opportunity to make noise and beat chest. I'm willing. I was
already considering raising a few probing questions during the plenary
anyhow. The past few months have been very frustrating as I try to
coordinate between opaque IETF processes and external standards bodies
like 3GPP that ask questions like "This draft was approved by the IESG
in June, why don't we have an RFC yet?" and "What do you mean there are
security concerns? Why didn't they come up during IETF Last call? Just
what are these concerns anyhow, and who raised them?"

--
Dean

---

From: Randy Bush <[email-masked]>
Date: Tue, 29 Oct 2002 22:48:20 -0800

> what are these concerns anyhow, and who raised them?"

the question that i would ask was why did the security problem come up
and get solved during the wg internal review or ietf last call?

randy

---

From: Derek Atkins <[email-masked]>
Date: 30 Oct 2002 08:46:47 -0500

> and get solved during the wg internal review or ietf last call?

Quick pithy answer: too many documents and not enough security-guru bandwidth.

Seriously, the IETF does a LOT of work.  It's hard enough just to keep
up with the I-D announcements, let alone read through those documents
at the rate they come through the process.  There just aren't enough
hours in the day to:

        a) have a real job, and
        b) read through every document in last-call with the appropriate
           eye for detail.

Just for comparrison, I was asked by a client to perform a security
review on a 60-page document.  It took me about 40 hours to read the
document and output well-thought-out responses.  While I tried to be
extremely thorough (part of why it took "so long") I have no way of
knowing if I even caught everything.  However at that speed there is
no way to get through all the IETF docs with any reasonable attention
to detail.

Admittedly in the IETF the response does not need to be quite as
well-thought-out; a set of notes would work just as well.  I didn't
measure the analysis v. response split, so estimate it at 50/50.  That
still leaves an approximate "reading rate" of 120 pages of 'spec' per
man-week....

> randy

-derek

-- 
       Derek Atkins
       Computer and Internet Security Consultant
       [email-masked]             www.ihtfp.com

---

From: Frank Kastenholz <[email-masked]>
Date: Wed, 30 Oct 2002 09:24:58 -0500

This is interesting.
I read the ID you cited and thought about this a bit.
It seems that the draft (and presumably the "less formal"
inputs you allude to) all contain "fixes" -- make this
rule change, alter that procedure, and then things will
be just plain wonderful. 

But I find it hard to believe that we can entertain any sort
of rule/process/etc changes without first understanding the
problem. One issue mentioned is the time it takes from
when a WG says it's done with a draft to when the IESG
makes its pronouncement. But why is this? Is it the sheer
number of the drafts? Is it the complexity of the drafts?
Is it that IESG people are still recruited with the completely
lame assertion that being on the IESG is a half-time job, so they
really do spend only half their time on it?  

Some other questions to ask in this vein - how many IDs are
currently on the IESG's plate? How long does the average
draft take to pass through? How long do the longest 10%
of drafts take (and why)? 

Are the current industry problems leading to a reduction in the
number of IETF attendees and a reduction in the number of people
who have to have a working group to solve some problem in a way
that no one else is interested in? 

Are the problems being caused by simply having too many working
groups of marginal utility? This means that the correction action
is to reduce the number of working groups.

Is the problem because there are too many people who really do not
understand the Internet trying to do something and getting all
confused? This means the problem is education, not process...

Is the problem that working group chairs do not know how to run a
tight ship, keeping the group focussed, etc? 

Sorry to approach this issue like an engineer rather than
a power-and-status-hungry-I*-politician.

Frank Kastenholz
Resident Grump

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 30 Oct 2002 06:57:12 -0800 (PST)

Ultimately, you've got to have some metrics that describe the performance
of the process -- and some understanding of what is limiting that
performance. Otherwise, we're just tinkering without much of a scientific
method behind it.

Are we measuring success by throughput -- the number of RFCs published? Or
by delay/latency -- the amount of time it takes to go from a -00 draft to
an RFC? Or by the ratio of offered load (-00 drafts submitted) to
throughput (RFCs)? Or by "quality" as measured by re-work (re-cycling to
Proposed)?

Do we know what the trends are for those metrics? How they break down
between WGs? Do some WGs or areas perform well on the metrics, versus
other WGs? What does the distribution look like? Are there common
characteristics among the poorly performing WGs?

Have we done a root cause analysis of some of the recent fiascos?
Assuming that IESG time is the bottleneck (do we really know this
is true?), do we know how the IESG apportions its time, in terms of
activities? For example, between monitoring mailing lists, reviewing
charters, liason activities, reading drafts, etc.? For how many of these
activities would improved tools be of assistance? How many are inherently
difficult to automate?

Do we know what WGs are most "expensive" in terms of time consumption?
Are they all alike, or are certains types of WGs inherently more stressful
on the system?

And do we have a plan for answering these questions?

---

From: Aaron Falk <[email-masked]>
Date: Wed, 30 Oct 2002 08:07:25 -0800

Frank Kastenholz wrote:
 
> problem. 

I quite agree.  Although what Harald is asking us to present on is a
discussion of where the problems are not what the fixes should be.

I'm curious as to whether mailing lists are still an effective to
work?  I suppose the same question goes for f2f meetings although I
find them both useful.

IETF work is really the work of a bunch of individuals.  I suspect
that the individuals are behaving differently than they did in the
earlier days.  This may not be the 'fault' of some aspect of the IETF
but could be the natural result of growth.  My questions focus on
working group effectiveness.

Are there fewer committed participants in working groups?  If so, why?
(It would be really cool if someone came up with a way to extract some
historicl data from working group mailing list archives.)

  - Could it be the increased number of lurkers?  (too much pressure
    of being 'observed')

  - Could it be the increased number of working groups?  (trying to
    cover all the topics of interest leads some folks to only monitor
    the group where they would otherwise contribute)

  - Could it be that the tone of (some) the meetings has grown too
    caustic or critical that newbies no longer want to jump in?  (We
    still need to attract newbies or attrition will kill us.)

  - Could it be that the movement from mostly academic participation
    to mostly industrial participation has changed the way in which
    members can participate, e.g., focus on breadth rather than depth?

> One issue mentioned is the time it takes from when a WG says it's
> done with a draft to when the IESG makes its pronouncement. But why
> is this? Is it the sheer number of the drafts? Is it the complexity
> of the drafts?  

Also, is it the inter-relationships between different wg docs or
between different wgs?

> Is it that IESG people are still recruited with the completely lame
> assertion that being on the IESG is a half-time job, so they really
> do spend only half their time on it?

IMO, unless it becomes a paid position, IESG needs to be no more than
roughly half time or it won't be possible to recuit from more than a
handful of candidates.

> draft take to pass through? 

Yes, data is a good!

> How long do the longest 10% of drafts take (and why)?

potential rat hole here.

> is to reduce the number of working groups.

one person's marginal utility....  This touches on a change that I
percieve, although I haven't been around long enough for personal
experience to verify: it seems that in the early days it was easy to
agree on what _wasn't_ of marginal value.  So, the IETF could focus on
things that were, in a rough way, agreed to be useful, interesting,
important, etc.  Now, much of the basic technology is in rev 2 or 3,
fixing bugs adding features, etc.  So, our basic needs are met.  We
have food, clothing, shelter.  It is harder to agree on what comes
next.  So, naturally, we pursue almost everything (some folks here are
trying to make a living, after all :).  Our focus is diluted.  

Our process, on the other hand, is oriented to provide the highest
quality output possible.  E.g., we have the IESG review all working
group documents.  This may seem like a effeciency bottleneck, but when
you are talking about food, clothing, and shelter, it's important to
get it done right.  In fact, everyone wants it done right.  However,
now we may not all be in agreement that spending our valuable IESG
resources (or IETF resources) on working groups "of marginal utility"
is the right thing to do.

What to do?  (this is what Frank is warning against... :) Perhaps we
shouldn't worry so much.  After all our basic needs _are_ being met.
The Internet works.  We won.  Also, perhaps we should recognize that
there is not universal agreement that our particular piece of Internet
technology is critical.  Finally, if there are efforts that are agreed
(by the community) to be important and critical, perhaps we should
give them some priority.

--aaron

---

From: Melinda Shore <[email-masked]>
Date: Wed, 30 Oct 2002 11:21:40 -0500

> a power-and-status-hungry-I*-politician.

I basically agree with what you're saying, although it
should be noted that we're all small-p politicians.

One thing I don't like about these recurring discussions is
that there's a tendency to dump it all in the IESG's lap
without clearly defining what "it" is or scoping the problem
that needs to be solved.  I can think of some particular
IESG problems that slow down the work (like the working
group chartering process), but it seems pretty clear to me
that we've got some fundamental problems in how the working
groups are operating and that perhaps addressing those might
be more productive than introducing more levels of hierarchy
or dumping more work on people who are already overloaded.

For one thing, the basic operating principles that once
drove the organization have changed.  In the absence of
altruism, consensus-based decision-making (as the term
"consensus" is used to describe process not product), can't
work.  It's reasonable to assume that people coming into the
IETF these days for the purpose of doing work (i.e. not the
spectators) are putting self-interest and their company's
interests ahead of the common good.  That means that the
conditions in which people are trying to reach agreement are
not the same as they were 15 or even 10 years ago.
Compounding that problem is that decision-making processes
are very difficult to start with and that we, as working
group chairs, are often lacking in those skills (saying "I'm
an engineer, not a politician").

As a result working groups spin their wheels for months at a
time, failing to identify questions that need answers and
failing to answer those questions that have been identified.
We need to take a good look at how well the decision-making
process is working or not working at the working group
level, I think, before trying to offload our problems onto
the IESG.

What's happening in the working groups reflects what's
happening in the IESG, too.  IETF process is *not*
transparent - increasingly less so, in fact.  Is that
causing problems in the day-to-day work of the organization?
Maybe - that's something that needs examination, as well.

As engineers *and* politicians I think that we really need
to sit down and take a look at our own working groups,
identifying where the process failures are happening as we
try to move our documents towards completion.  I don't think
I've ever participated in a working group that didn't have
internal problems - let's start there.

Melinda

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 30 Oct 2002 08:52:42 -0800

>performance. Otherwise, we're just tinkering without much of a scientific

I would be interested in knowing the average number of revisions
an I-D undergoes from the time it is recommended to the AD to the
time it enters the RFC-editor queue.   If the IETF is performing
well, this number should be quite small.  A large number is
indicative of document quality problems.

It's my general opinion that many of the "process problems"
are symptoms of document quality problems.  Basically, poor
quality documents tend to act as DoS attacks... sucking away
process/review bandwidth from quality documents.

I think we not only need to look at ways to improve document
quality (thus reducing the process/review bandwidth required)
but also process mechanisms to ensure process/review time is
fairly allocated to each document in the queue.

Kurt

---

From: Frank Kastenholz <[email-masked]>
Date: Wed, 30 Oct 2002 12:23:11 -0500

>discussion of where the problems are not what the fixes should be.

Did you read the ID Harald referenced? It said that things
are broken and here are the fixes for them. Other comments
are of the same general nature. There is no measurement, no
numbers, no quantifiable information as to what the breakage
is and why it's "broken".  Stating that some bad things
happen is not sufficient. State what they are, how often 
they happen, and then figure out _why_ they happen. 

F

---

From: "James Kempf" <[email-masked]>
Date: Wed, 30 Oct 2002 09:43:03 -0800

Melinda,

You've pretty well summed up what I've found to be the major problem.

I've only got one additional observation. I don't think that individuals acting
as bottlenecks are doing it out of any rational motivation, like they think it
is in their company's interest. I think it is just pure ego and the problems
with scaling a process that was originally designed to work with maybe twenty
people around a table and now has to work with 200 in a room with microphones.
Small group interactions have a different dynamic than larger group ones, one
reason many WGs are moving toward design teams.

            jak

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 30 Oct 2002 10:37:50 -0800

I wonder if anyone else has noticed the growing lack of
"running code" associated with works being progressed on
the Standard Track.  For every standard track submission,
we need to ask "where's the code?" and, when the answer
is "none (yet)", seriously doubt the appropriateness of
progressing the work on the Standard Track.

Kurt

---

From: Marshall Rose <[email-masked]>
Date: Wed, 30 Oct 2002 11:01:07 -0800

> they happen, and then figure out _why_ they happen. 

hi. just a couple of things.
    
first, the I-D is fairly careful to identify four desirable properties
for the IETF and then make suggestions as to how these properties can be
increased within the IETF. in this sense, the I-D is somewhat more
"strategic" in its arguments than your summary above would indicate.
    
second, i recall a couple of guys who published an I-D back in march of
this year which looked at all the qualitative analysis that was
available. although the I-D has since expired, here's a link to the
graphs:
    
    http://xml.resource.org/ietf-analysis/current/
    
the I-D itself was fairly careful to just present all the data available
(after an awful lot of screen scraping) and avoid drawing any
conclusions. (outside of the usual knee-jerk reactions,) the one really
good comment that came back was that the things that are available to
measure aren't that interesting... in other words, it's the qualitative
aspects of the IETF that we should be measuring, but there's no data
available on that.
    
third, as much as i'd enjoy watching a witch-hunt-on-steroids, i don't
think it would serve any useful purpose. that's why the I-D harald
mentioned was fairly careful to keep things positive, i.e., "here are
things we should strive for, here are ways we can do that, and here's
the philosophy behind that."
    
/mtr
    
ps: for the ease of folks who haven't read the I-D in question, the URL
is:
    
    http://www.ietf.org/internet-drafts/draft-huston-ietf-pact-00.txt

enjoy.
    
/mtr

---

From: Kathleen Nichols <[email-masked]>
Date: Wed, 30 Oct 2002 11:13:54 -0800

"Kurt D. Zeilenga" wrote:
> the Standard Track.  

Yes.

---

From: Jun-ichiro itojun Hagino <[email-masked]>
Date: Thu, 31 Oct 2002 04:08:48 +0900

>progressing the work on the Standard Track.

	strongly agree.  i may suggest that we require running code be
	available (publically, or propriatary?) for standard-track submission.

itojun

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 30 Oct 2002 10:08:06 -0800 (PST)

Because in some cases, it is required that the protocol reach Proposed
Standard before software developers will invest in a full implementation,
or hardware makers will start designing ASICs. That's why interoperability
demonstration is required for Draft Standard, not Proposed.

However, I do think that it is reasonable that we restrict standardization
to "things we know how to do." One demonstration of that knowledge could
be the existence of "proof of concept" running code or hardware. If no
such code exists, then that would be an indication that perhaps we don't
really "know how" yet.

---

From: Melinda Shore <[email-masked]>
Date: Wed, 30 Oct 2002 14:15:30 -0500

> "running code" associated with works being progressed on

Of course: "rough consensus and running code" doesn't
describe the current process very well on either count.
Part of this is because of a shift towards starting from
scratch (no protocol), developing requirements and a
framework and then proceeding from that.  (In several
working groups I've been involved with the process has been
subverted somewhat, so that the framework and requirements
are reverse-engineered from an existing protocol, which I
think is a problem of a different kind.)  We also seem to
have a lot of participants who don't write code - technical
marketing-type folks, etc.

I don't really see it as a problem, or at least one that can
be solved without moving away from the requirements/
framework process - it's still the case that a document
can't proceed to Draft Standard without two interoperable
implementations.

Melinda

---

From: Aaron Falk <[email-masked]>
Date: Wed, 30 Oct 2002 11:08:00 -0800

Frank Kastenholz wrote:
> 
> Did you read the ID Harald referenced? 

I have now.

> It said that things are broken and here are the fixes for
> them. Other comments are of the same general nature. There is no
> measurement, no numbers, no quantifiable information as to what the
> breakage is and why it's "broken".   

indeed.  I think it is presumtuous to assume the IESG is the root
cause of any standards bottleneck.  I'm also quite concerned by the
proposals in the draft to 1) devalue cross disciplinary review by the
IESG and 2) increase the role of the calendar in approving standards.

Another suggestion I haven't seen made is to work on improving the
quality of the writing coming out of working groups.  Clarity, and
brevity when possible, will make the rest of the process smoother.
Selection of document editors is important.  I recently switched
editors of two important wg docs because one, while brilliant, wasn't
able to produce timely drafts and the other wasn't a native english
speaker and the resulting drafts were too hard to read.  It was tough
for me to basically take work away from them but I think the resulting
documents benefited greatly.

--aaron

---

From: "Carl" <[email-masked]>
Date: Wed, 30 Oct 2002 11:12:01 -0800

Kurt,

I think you bring up a hen and egg situation here.

Many vendors are not really prepared to invest money in a standards based
product until it is on the standards track.

In some cases you may be lucky to find people or companies who are willing
to spend some research money to make prototyping of the proposed standard
before everything is firmed up (or somebody is trying to make their existing
product become the standard, which usually fails in the IETF), but in
today's market situation you will see fewer and fewer companies investing in
code at an early stage.

I actually used to do this kind of work, but was laid off because the
company wasn't willing to spend the money any more.

Carl-Uno

Carl-Uno Manros
10701 S Eastern Ave #1117
Henderson, NV 89052, USA
Tel +1-702-617-9414
Fax +1-702-617-9417
Mob +1-702-525-0727
Email [email-masked]

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf
> Of Kurt D. Zeilenga
> Sent: Wednesday, October 30, 2002 10:38 AM
> To: Harald Tveit Alvestrand
> Cc: [email-masked]
> Subject: Where's the code (was: Input solicited: IETF problem
> pinpointing at the plenary?)
>
>

---

From: Dean Willis <[email-masked]>
Date: 30 Oct 2002 13:47:28 -0600

Randy asked:
> Dean said:

Ah, if I actually knew the answer to that question. Keep in mind that
I'm actually quite stupid, horribly overloaded, moderately lazy, and
need to have things beaten into my head from time to time. Given that,
I'd like to find process solutions that allow me to answer that sort of
question -- because if it works for me, it'll probably make it a lot
easier to explain externally.

As I recall, the draft in question went through WG review, went through
the IETF Last Call period, then got stopped (a "discuss" if I remember
correctly) in IESG review. After some time, we traced part of the holdup
to an other-area AD who assumed that there was a security issue based on
an unfortunately chosen word in the draft title, perhaps based on the
observation that other things with this same word in them generally had
that same security issue. The matter was resolved quickly after
discussing it with a director from that area. It PROBABLY would have
been resolved quickly had said AD had time to actually read the draft,
but there's SO much stuff to read. The concern was legitimate and
well-intentioned. The resolution of it was straightforward. The PROBLEM
is that there wasn't enough information for the WG chairs or authors to
identify the concern and effect the resolution in a timely manner, so we
burned a couple of months on a document that many others depended on
while solving what amounted to a trivial human communications problem.

One process resolution that might address this sort of situation is for
DISCUSSes to be made public, so that the pool of people out there with
relevant information and the cycles to deal with it can assist in
responding to them. I understand something like this is "in the works",
but I'm not well-informed as to its state vector.

ADs are precious resources. The guidance the IESG provides is a large
part of what differentiates the IETF from "flat" standards bodies like
3GPP. One of our goals as an organization is (or darned well SHOULD be)
to find ways to optimize our use of the IESG's guidance capabilities
without requiring that the ADs individually work themselves to death. At
the risk of spewing out a cliche, the work is NOT going away -- we need
continuous improvement on working SMARTER, not HARDER.

--
Dean

---

From: Andy Bierman <[email-masked]>
Date: Wed, 30 Oct 2002 11:55:19 -0700

>        available (publically, or propriatary?) for standard-track submission.

The official standards process does not support this opinion.
(Note paragraph 3 below.)  

Perhaps you are suggesting the process be revised. Here is the
relevant text from RFC 2026. Please provide suggested revisions.
I think paragraph 4 could be clarified, but the process for
publishing at PS does not need to be changed.

4.1.1  Proposed Standard

   The entry-level maturity for the standards track is "Proposed
   Standard".  A specific action by the IESG is required to move a
   specification onto the standards track at the "Proposed Standard"
   level.

   A Proposed Standard specification is generally stable, has resolved
   known design choices, is believed to be well-understood, has received
   significant community review, and appears to enjoy enough community
   interest to be considered valuable.  However, further experience
   might result in a change or even retraction of the specification
   before it advances.

   Usually, neither implementation nor operational experience is
   required for the designation of a specification as a Proposed
   Standard.  However, such experience is highly desirable, and will
   usually represent a strong argument in favor of a Proposed Standard
   designation.

   The IESG may require implementation and/or operational experience
   prior to granting Proposed Standard status to a specification that
   materially affects the core Internet protocols or that specifies
   behavior that may have significant operational impact on the
   Internet.

   A Proposed Standard should have no known technical omissions with
   respect to the requirements placed upon it.  However, the IESG may
   waive this requirement in order to allow a specification to advance
   to the Proposed Standard state when it is considered to be useful and
   necessary (and timely) even with known technical omissions.

   Implementors should treat Proposed Standards as immature
   specifications.  It is desirable to implement them in order to gain
   experience and to validate, test, and clarify the specification.
   However, since the content of Proposed Standards may be changed if
   problems are found or better solutions are identified, deploying
   implementations of such standards into a disruption-sensitive
   environment is not recommended.

>itojun 

Andy

---

From: Frank Kastenholz <[email-masked]>
Date: Wed, 30 Oct 2002 14:25:04 -0500

At 11:01 AM 10/30/2002 -0800, Marshall Rose wrote:

>increased within the IETF. 

I have two issues here.
1. Do the properties _need_ to be increased, or are they at a 
   high enough level.  Your ID simply asserted that they were
   not high enough and proposed methods to increase them, which
   leads to ...
2. If we take for granted that we need to increase these properties,
   are your proposals the right way? In other words, what is the 
   reason for the properties being at the level they are now? If the
   reason is "A" but your proposal fixes "B" then there will be no
   improvement.

>    http://xml.resource.org/ietf-analysis/current/

I do not recall seeing that ID. Thanks for pointing it out.
Perhaps you could resubmit it, and reference it in your
current draft? Regardless, with that information, a more
reasoned debate could be held as to whether the numbers are
tolerable, bad, too bad, or whatever, and we can also look at
the outliers to see if, for example, it makes sense for the
MPLS working group to have produced 17 RFCs, etc.

>the one really

Well, that implies that X had a "bad time" (for some
value of "bad time") with the process. Perhaps getting
some documentation of the exact problem that X had
and then investigating whether that problem was systemic
or one-off, would be in order (or did a couple of guys
publish an ID on that that I misseed?)

>think it would serve any useful purpose.

I don't recall asking for a witch-hunt (even given today's date).
I just want to know exactly what's broken before we fix it. 

Frank Kastenholz

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 30 Oct 2002 11:24:33 -0800 (PST)

> MPLS working group to have produced 17 RFCs, etc.

There are some conclusions that could be reached, but it struck me that
this document was the beginning of a discussion, not the end of it. I
could well see spending some finite period of time analyzing the data in
more depth, so that we could understand where we are and potentially have
the beginnings of a reporting system that would keep us appraised of that
performance over time. If funding is required to make that happen, I'm
sure it can be arranged.

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 30 Oct 2002 12:32:20 -0800

>demonstration is required for Draft Standard, not Proposed.

I was not suggesting we should require multiple independently
developed interoperable implementations at Proposed.  Nor am
I suggesting any other change to RFC 2026.

When I said "running code" I was referring to the "implementation
and operational experience" encouraged (but not required) to be
present for Proposed Standards in RFC 2026.  What I said was that
I believe that when such experience is absent we should question
the appropriate of the submission. 

>such code exists, then that would be an indication that perhaps we don't

Yes.  Lack of such experience is often a tail-tail sign
of other qualities which should be present in Proposed Standard
submissions are absent.

>In Wed, 30 Oct 2002, Kurt D. Zeilenga wrote:

---

From: Jun-ichiro itojun Hagino <[email-masked]>
Date: Thu, 31 Oct 2002 05:53:53 +0900

>publishing at PS does not need to be changed.

	i guess i am.  how about replacing paragraph 3 and 4 with the following:

	At least one implementation (even if a test implementation) and
	some amount of operational experience is required for the designation
	of a specification as a Proposed Standard.

itojun

---

From: Randy Bush <[email-masked]>
Date: Wed, 30 Oct 2002 12:54:22 -0800

> need to have things beaten into my head from time to time.

you need to volunteer for the iesg using the nomcom mechanism.  posting
to this list does not work. :-)
 
> while solving what amounted to a trivial human communications problem.

we hope that is actually being fixed in the short term with the draft
trucker.

> but I'm not well-informed as to its state vector.

before atlanta

> At the risk of spewing out a cliche, the work is NOT going away
> -- we need continuous improvement on working SMARTER, not HARDER.

send code :-)

randy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 30 Oct 2002 13:27:56 -0700

>        of a specification as a Proposed Standard.

This will certainly reduce the IESG document review workload,
by reducing the chances that vendors will implement IETF technology.
Most vendors don't want to commit resources until the technology
is past the work-in-progress phase and big changes are locked out.
Nor do they want to commit resources to a test implementation
just to prove code can be written that implements a specification.

Although it's important that a specification can be implemented,
the creation of a test implementation does not in any way prove
that the technology is useful to the Internet community.

Andy

---

From: "Kurt D. Zeilenga" <[email-masked]>
Date: Wed, 30 Oct 2002 14:00:12 -0800

>is past the work-in-progress phase and big changes are locked out.

Then they should wait until the document reaches
Draft Standard status.

>just to prove code can be written that implements a specification.

I think operational experience has shown that who ever implements
a specification first is most likely going to find problems with it.
It's highly desirable for these problems discovered and resolved
before the specification becomes a Proposed Standard.

>that the technology is useful to the Internet community.

I don't believe the "running code" is meant to show usefulness.
Lots of running code is useless.   Running code is intended to
show that the specification "works".

---

From: "James Luciani" <[email-masked]>
Date: Wed, 30 Oct 2002 16:53:24 -0500

> that the technology is useful to the Internet community.

I don't get the value of specifying "one" implementation either.  Usefulness
to the internet community is not verified nor is interoperability nor is the
"implementable" criteria truly served unless you plan on doing some sort of
peer inspection.  There are always judgment calls made in any complex system
development.  The outcome of these judgment calls are likely not the same as
the outcome when someone else makes the calls.   To take this to the obvious
limit, who is to judge if someone actually implemented the spec at all and
by what criteria is such a decision made?

---

From: "Tony Hain" <[email-masked]>
Date: Wed, 30 Oct 2002 14:03:42 -0800

Melinda Shore wrote:
> ...
> framework process - ...

I won't argue about the framework part, but watching many discussions
rat-hole over fundemental disagreements in the goal, I would have to say
that writing down and getting consensus on the requirements has to be
the first step. For very focused WG's this could be incorporated in the
charter, but some form of a written statement that says 'the resulting
document must do X' is the only way to run a distributed design team
where membership changes over time.

Tony

---

From: Marshall Rose <[email-masked]>
Date: Wed, 30 Oct 2002 14:41:07 -0800

> an engineer, not a politician").
    
i guess i'm not just this pessimistic. the countervailing force is
called "enlightened self-interest" and, if anything, should be
increasing in strength given the current downturn...
    
/mtr

---

From: Marshall Rose <[email-masked]>
Date: Wed, 30 Oct 2002 16:21:26 -0800

first, thanks for the thoughtful reply. it is much appreciated. this
reply is longer than i'd like, but the issues raised are rather substantive.

>    not high enough and proposed methods to increase them, which

these are reasonable questions that we should address.

i think there is a fairly high-level of disatisfaction with the way
things are going. obviously, not everyone feels this way... but, i think
it fair to say that there was a concensus in the audience at yokohama's
plenary that something has gone wrong. (although i was one of the few
folks in the room to go up to the mic and defend the iesg,) i also
observed that the "complaints" didn't seem to focus on one or two
areas.

so, can we agree that there's a problem? if not, well, then we'll have
to disagree. and that's ok.
    
if we can agree that there's a problem, then we have another problem,
namely trying to get agreement in the community as to what the problem
is. in fact, i think we'll both agree that even on the small subset of
the community that is represented on the wgchairs mailing list, we're
not going to get concensus on what the problem is.

in a situation like that, where you need to effect change, but you can't
get agreement even in relatively small groups, a common technique for
moving forward is to focus on "first principles"...and this is where the
I-D starts, with the "PACT" or "properties".
    
you are absolutely right that the I-D starts off with the assumption
that the PACT needs to be increased. it's an assumption. after yokohama,
i view it as a "self-evident" assumption. and, obviously, not everyone
may view it as self-evident.
    
but, does anyone really care to argue that the "PACT" is not a good
thing? does anyone really care to argue that the level of the IETF's
"PACT" is adequate? (well, i guess it's possible, but doubtful. however,
i invite anyone arguing for less competency (the "C" in "PACT") to stand
up now... ditto for the other three properties.)
    
the reason that the I-D focuses on the IETF's PACT is because these
provide kind of a general framework for judging what's good or bad. i'd
like to thing that those four cover 90% of the IETF's "goodness" (but
there's probably nothing magical about the number four).

>    reason is "A" but your proposal fixes "B" then there will be no

this is a big concern. it's entirely possible that the proposals, whilst
philosophically aligned with the PACT may not solve the exact problems
we have now. so, we agree on this point.
    
i don't know how to address this, other than to look at other possible
solutions that are consistent with the PACT. however, the one thing to
keep in mind is that the proposal you see in the PACT I-D were developed
with an eye towards having a relatively small number of recommendations
that could have a rather large effect.

> MPLS working group to have produced 17 RFCs, etc.

such a debate would be welcome, but if the reception the I-D got seven
months ago is any indication, both of us will grow old waiting for the
debate... (amusingly enough, a fair amount of my time was spent arguing
with some as to whether the abstract was adequate. yes, that's right the
abstract... instead of being engaged on the methodology, the numbers, or
even the colors used in the pngs, the guy wanted to argue about the
friggin' abstract.) given that experience, it's a wonder that geoff and
i even bothered publishing the pact I-D.

i'll resubmit the I-D, unchanged, as -01 (although natalie may decide
i'm doing a no-no and not allow it.).

> publish an ID on that that I misseed?)

i wasn't clear then. the problem is that document progress may not be
the most interesting thing to measure. ideally, you'd like to measure
document quality, and quality is one of those things that is rather hard
to measure...

> I don't recall asking for a witch-hunt (even given today's date).

i really want to avoid trying to do a post-mortem on some of the recent
problems because it will ultimately lead into the behavior of "X"
individuals, and we'll spend more time dancing aorund that than solving
a problem.

> I just want to know exactly what's broken before we fix it. 
    
in an ideal world, this is certainly the next step. the problem is that
i don't think that the comnunity has the ability to agree or even
analyze something to the point of being even modestly "exact".

it is an axiom that any "interesting" topic discussed "IETF-wide" 
generates a sufficiently wide range of commentary so as to block any
serious basis for taking action. accordingly, the best we can hope to
achieve is to get some kind of concensus that
    
    - there's something broken
    - it's fairly serious
    - we need to get past it
    
which takes me full-circle to the way that the PACT I-D was written:
    
    - focus on first principles
    - suggest some ways of getting more of the good stuff
    
in other words: it is probably better, right now, to create a pattern of
positive change than to worry about optimizing the choice of what to
change...

/mtr

---

From: Avri Doria <[email-masked]>
Date: Thu, 31 Oct 2002 10:06:30 +0100

I just read though all the responses so far,  interesting when you batch 
them.

I have a set of suggestions that i don't think were made already. If 
they were, i apologize for missing them and for being repetitive.

While there was a fair amount of discussion on the importance of running 
code, and i don't wish to minimize the importance of ultimately having 
the 2+ interoperable implementations to become a DS, i don't think the 
absence of running code has much of an effect of the PACTful running of 
the IETF.

As i see it, many of the scaling problems stem from the quality of the 
drafts presented to the IESG for review.  It seems to be assumed that 
this is the fault of the WG.  In some cases it may be.  One of the 
things that has always struck me as interesting is that we have a lot of 
process around the training and management of WG chairs, but we do 
little to train or manage the editors.   I suggest that this is an area 
  where improvement would help streamline the process.  I think the 
solution would be twofold:

- training sessions for editors similar to the WG chair sessions
- an advisory function for editors similar to the technical advisory 
function.  I am not suggesting a new job category, but rather a function 
that is taken on by either the technical advisor or some member of the 
area directorate.

Another draft problem has been the fact that documents arrive at the 
IESG's doorstep without having had adequate review.  While part of this 
is the WG responsibility, part is also the IETF at-large's 
responsibility.   Why else do we have a IETF Last Call?  I find it 
baffling, e.g., that the Security folks would allow something to go 
through IETF Last call without a thorough analysis.  I suggest that the 
IETF Last Call is the place where deficiencies, especially those that 
are not in the specific talent base of the WG members, should be 
discovered.  I believe that the ADs could perhaps make use of their 
directorates to ensure this happens. Whenever a draft in is in IETF Last 
Call, each interested AD should make sure that one or more  of her or 
his directorate is assigned to doing a detailed analysis of the document 
and that this review go to the WG chair and editors.  Speaking of IETF 
Last Call comments, what happens to them anyway?   Perhaps the documents 
fro my group that went through IETF Last Call never got any comments, 
but I have never seen any.

Which brings up another question, and perhaps a metric that could be 
collected (yes, a mechanism for collecting it would be needed - perhaps 
an on line form that folks fill in after doing an IETF LC review).  How 
many people, other then ADs, actually review documents in IETF LC.  I 
admit that to date i have only read those i am personally interested in, 
and have rarely done anything that could be called a proper review.  Am 
i alone in this behavior?  What use is the IETF Last Call if no one 
really does a solid review?  I believe there should a through review 
during IETF LC before the IESG considers a document, and that a WG 
should address all LC comments before the IESG deals with a document. 
I.e. bring RFCs half way to being refereed.

Finally, i don't think that the IETF necessarily needs more bureaucracy 
or levels of management.  I do believe it needs to use those it has to 
better advantage.  The technical advisory job is important, yet i 
believe it has a very spotty use.  I also wonder about the fact that so 
many ADs are WG technical advisors.  Do they really have time for the 
hands-on technical advising that the WGs need?  And i think better use 
might be made of the directorates. Instead of just being a bunch of 
sages who advise the ADs on weighty matters, perhaps they can actually 
be volunteered for work, such as reviews.

a.

-- 
Avri Doria
http://www.sm.luth.se/~avri/

---

From: Randy Bush <[email-masked]>
Date: Thu, 31 Oct 2002 05:43:04 -0800

your points are spot on from my limited point of view.

> How many people, other then ADs, actually review documents in
> IETF LC.

as an iesg member, i have observed almost zero ietf last call
comments.  it has mostly become a formality.  while having serious
issues arise that late in the process is not likely fun, it may be
better than their arising during post last call iesg review.

otoh, i am not sufficiently naive to believe that iesg members are
the only people with overburdened schedules.  and the iesg is the
gang actually tasked with full formal review.  so it may be just a
fact of current life that the buck stops there.

> better use might be made of the directorates. Instead of just
> being a bunch of sages who advise the ADs on weighty matters,
> perhaps they can actually be volunteered for work, such as
> reviews.

that is happening.  i am now getting significant review help from
an informal ops directorate.  i think other area directorates are
moving on similar paths.

randy

---

From: Brian Haberman <[email-masked]>
Date: Thu, 31 Oct 2002 09:14:45 -0500

> your points are spot on from my limited point of view.

As from mine.

> better than their arising during post last call iesg review.

It almost seems like people assume that others have done a
review.  I have also observed that that behavior is even more
common based on who is on the draft's author list.

> an informal ops directorate.  i think other area directorates are

I am glad to see that the ADs are trying to utilize the resources
they have.  Does this mean that every Area will end up with a
directorate?

Brian

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 31 Oct 2002 05:25:51 -0800 (PST)

> While part of this
> through IETF Last call without a thorough analysis.

In the case of major security problems, in quite a few cases the issues
were present in very early versions of the draft, and carried all the way
through the WG to IETF last without being detected.

This speaks to having a review of the security approach very early on.

---

From: Randy Bush <[email-masked]>
Date: Thu, 31 Oct 2002 06:23:46 -0800

> they have.  Does this mean that every Area will end up with a

i don't know.  it's the kind of thing where ADs do what they feel
is most useful to them.  many have long had directorates.  the
current ops-dir is the second time i have tried it during my time
on the iesg.

and there is a difference between a directorate and a review team.
i imagine that, in some areas, they might not be identical.

and i note that there is also at least one subject directorate,
dns, which is cross-area.

so it is a tool which is used in various ways.  one of the nice
things about some ietf process infomalities is that we can be
somewhat creative.

randy

---

From: Frank Kastenholz <[email-masked]>
Date: Thu, 31 Oct 2002 09:52:01 -0500

>to disagree. and that's ok.

All I am prepared to admit at this time is that there seems*
to be widespread perception that there is a problem and it
seems that lots of folks do grumble about things. 

(*I say "seems" because I personally have cut back my I* 
involvement the past several years and don't have a lot of
first-hand data to back up the statement)

Granted, it's hard to measure the coefficient-of-grumbling, so
we can leave that alone. However, we certainly can look into
what people are grumbling about and ask some grumbling-related
questions, such as 
- What exactly are people grumbling about?
- Are they grumbling just because they like to gumble? After all,
  the IETF is a group that is notoriously hard to please so we
  should expect and dismiss a certain amount of grumblers simply
  as people who need to get real lives (excellent example over on
  the IETF list right now...)
- Are the things people are grumbling about a few isolated incidents
  or is it a widespread, systemic, problem. Eg, are 500 people 
  grumbling because 1 RFC was late to be published and 500 people
  were interested in it, or are 500 people grumbling because 500 
  RFC's were late to be published, with each grumbling about his own
  RFC.  These are different problems with, prima facie, the same
  symptoms.
So why not get a few people and task them to go out and speak with
the grumblers and record what they are grumbling about. Then report
back. We can look into things and see if there is an issue or not?

>not going to get concensus on what the problem is.

I do not see people in this "small subsete of the community"
disagreeing about whether there is a problem or not and what
that problem is. I see
- a few people saying "there is a problem, here are the fixes"
- a lot of people saying "I've seen/had this problem"
- one person saying "let's get some data together and try
  to understand things before we fix them". 

My view is that without some data, we can never get concensus.

>may view it as self-evident.

Even if it is self-evident, there is also the question of whether
you propose the right measures, and whether the result is worth
the effort. There is also the law of unintended consequences that
should be taken into account.

>up now... ditto for the other three properties.)

I am not questioning whether there is enough PACT or not in the
IETF. However, I do question whether we do understand where and
how we are deficient. Without knowing what the deficiencies are
(and not just "it's not enough") how can we improve things. For
example, if a piece of software is running slow, do you just 
start changing the code, or do you try to understand the algorithms,
the environment, how it's used, do you profile it, and so on,
before changing things?

>solutions that are consistent with the PACT.

Well, we _could_ first figure out how/why/where we are "pact-less"...

>i even bothered publishing the pact I-D.

1. I'd have put that guy in my kill file -- especially if he was
   someone on the IESG or the like.
2. Just plopping an ID in the middle of the world doesn't get
   a whole lot of interest. A few months ago two ID's showed
   up about requirements for a new inter-domain routing architecture.
   One would expect that they would get a lot of people's attention
   since routing is a sexy topic among many who seek fame... Not one
   message (even on the abstracts). Even RFCs get scant reading; there
   are two RFCs (the second a revision of the first) that have a big
   blob of Shakespeare gratuitously plopped into the middle of them.
   As far as I'm aware, the authors received no comments or questions
   on it -- even from the IESG and RFC editor.

However, if things are tied together by, eg, putting out an ID that says
"we think these problems exist, based on this data, and here are solutions"
might get more substantive comments. 

>i'm doing a no-no and not allow it.).

So move some paragraphs around :-)

>the most interesting thing to measure. 

I was using document progress as an example, sorry for the confusion.

>ideally, you'd like to measure

How about
- time it takes to get the document from the WG to the IESG
  on the theory that the longer the document is worked on, the
  more complex it is, and the more complex it is, the more likely
  it is to have problems.
- amount of time it takes the IESG to review the document
  same theory (modulated by IESG-workload, but that's probably
  a constant bias)
- amount of comments the IESG returns to the WG
  which indicates how much work there is left to do, though
  there is a problem with a single "This is rubbish" comment
  vs 10,000 words of detailed editorial nits.
- size of the diffs resulting from the above IESG comments.
- how many documents pass the IESG with now work vs how
  many get returned back.
  if 99% pass, then one might claim that document quality
  is not an issue. But do 99% of them pass? or is it 1%?
  Associated with this is how many times a document is 
  submitted to the IESG before it passes.
- amount of diffs going from proposed to draft standard

And that's without finishing my first cup of coffee this morning...

>individuals, and we'll spend more time dancing aorund that than solving

I disagree completely, and I find it hard to believe that you really
believe this since I recall you calling the ISO protocol effort 
a deer stuck in the headlights of the oncomming 18-wheeler named 
TCP/IP, and had even less polite things to say to Jack Houldsworth 
et al, for whom the ISO effort was a Religious Experience of the 
first order, all in an IETF plenary meeting... If we have to avoid
looking into problems because someone's feelings might get hurt then
we may as well give it up now and go home.

>serious basis for taking action. 

It doesn't have to be IETF wide. It can be 5 people chosen
to look into it and write a report back to the IAB/IESG/
ISOC/IETF/pick-one.  Think of it as a Congressional BRAC
committee...

Frank Kastenholz

---

From: Rob Austein <[email-masked]>
Date: Thu, 31 Oct 2002 10:37:55 -0500

At Thu, 31 Oct 2002 09:52:01 -0500, Frank Kastenholz wrote:
> 
>    Even RFCs get scant reading; there are two RFCs (the second a
>    revision of the first) that have a big blob of Shakespeare
>    gratuitously plopped into the middle of them.

Seemed like a perfectly reasonable commentary on the process by which
that particular RFC was written.

---

From: Marshall Rose <[email-masked]>
Date: Thu, 31 Oct 2002 10:32:14 -0800

frank - again, thanks for the thoughtful note. rather than do a detailed
inline response, i think i need to be more strategic in my reply.

my sense, given your last couple of notes, is that you feel we ought to
be studying the situation before coming up with a plan of action. in
principle, this is a reasoned way to proceed. the problem basically is
that for folks who've been in the trenches for the last few years, we're
past that point.
    
frankly, you and i are so far apart in terms of our perceptions of the
real world, that neither of us is going to be convincing the other.
    
so, for folks who think that there isn't a problem, or that we need to
do a detailed things in order to figure out the problem, i think those
folks should just say the PACT I-D is a bad thing. i'm comfortable with
that.
    
for folks who feel there is a problem that needs to get fixed within our
professional lifetime, then i ask that they study the PACT I-D and
decide whether they like it or not. those who don't like it would do the
community a great service by making concrete suggestions for fixing it
or replacing it. concrete suggestions is how the IETF gets things done.
    
peace,
    
/mtr

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 31 Oct 2002 14:22:44 -0500

I would agree that there are multiple problems.
Some are being addressed.
Some need to be addressed.
Some are probably impossible to address.

I even agree that we have problems with both the quality of work and the 
approach to and timeliness of feedback.
However, I think that most of the proposed remedies in the PACT ID are 
likely to be either ineffective or counter-productive.  I am concerned that 
putting on time limits with automatic approval of documents will result 
in  the short run in many lower quality documents, and in the long run will 
result in the IETF refusing to work on things that it needs to work on.

I actually think that we do need to refuse to work on some things.  But, 
the conditions for refusal should not be whether the work is hard, or even 
if there is too high a probability of a bad outcome.  The conditions should 
be whether there is a reasonable path to a good outcome, and whether it is 
work that we should be doing (that is, the first condition is that it is 
work someone should do, and the second condition is that it is work 
appropriate to the IETF.)

At the moment, we collectively accept almost any working group, and allow 
them to produce almost any result.

While I do not have a comprehensive set of solutions, I think that there 
are a number of steps that need to be taken:
a) Working group chairs need to be more proactive in seeing that their 
groups produce good results.  It should almost never be necessary for an AD 
to even ask whether something is "in charter".
b) The community, including the working group chairs, has to find a mode of 
operation in which we are willing to accept leadership.  All too often, 
when the IAB and / or IESG attempt to bring some sanity by saying "no" to 
something, we respond with "but I'm different, I should be allowed to do 
this."  Sure, every case is special, and sure every protocol has a reason 
for existing.  Do we really need the vast number of tunnelling 
protocols?  No.  But heaven help the AD who tries to prevent the next one.

Yours,
Joel M. Halpern

---

From: Marshall Rose <[email-masked]>
Date: Thu, 31 Oct 2002 12:26:46 -0800

> result in the IETF refusing to work on things that it needs to work on.

all these things in the PACT I-D deal with a balancing of interests.  i
think predictability and timeliness are important properties, and we
need to address them.
    
it's possible, under one scenario that you may see a quality
drop. it's also possible, under another scenario, that this will present
the iesg with incentives to:
    
    - deal with problems earlier, rather than later
    - learn to say "no" more often
    
both of which, i think are good things. further, it's not really clear
to me how many iesg members think that we're producing a lot of
high-quality stuff now.

i think it's reasonable to expect self-interest actors to adjust their
behavior to new sets of rules. call it "unintended consequences". i view
it as a good thing...

> appropriate to the IETF.)

it's not clear whether this is an appropriate philosophy for an
engineering organization. if we're talking research, sure.
    
and, this may explain the PACT I-D's perspective on a lot of these
issues. it's tailored for an engineering organization, where things like
timeliness are first-order concerns...

> them to produce almost any result.

yes, and we both agree it's a bad thing.	

> to even ask whether something is "in charter".
    
it's better to deal with problems earlier rather than later,
obviously. but the quality thing has to start before the wg chair. it
has to start with the charter that gets approved by the iesg. then those
areas that have "a stake in the outcome" need to have a senior person
present to flag problems, suggestion solutions, etc...
    
ideally, we should never have a situation were something goes to last
call without some subset of the iesg already being entirely comfortable
with the work product.

> protocols?  No.  But heaven help the AD who tries to prevent the next one.

my direct experience, albeit from 10 years ago, is that an ad can make
an unpopular decision stick if they're willing to:
    
    - let people present their arguments as to why the answer should be
      "yes" 
    - provide a timely, reasoned explanation as to why the answer is
      still "no"
    
of course, speakingly plainly, if you don't have transparency (aka
accountability), you're never going to get to (b).
    
/mtr