Subject: Re: A non-IPR issue (was: Re: DRAFT Revision to the WG Document Writeup)

From: John C Klensin <[email-masked]>
Date: Mon, 31 Oct 2011 10:58:33 -0400

--On Monday, October 31, 2011 15:12 +0200 Lars Eggert
<[email-masked]> wrote:

>...
>> The problem arises when it (including not just the published
>> and stable RFC 4858 but the evolving web page and tools)
>> starts being treated as normative.  At least IMO, it becomes
>> normative the moment ADs start recording DISCUSS positions
>> based on non-conformance to the outline, the checklist, or
>> the nits checker.
> 
> I disagree. When I was on the IESG, in order to make the best
> use of my time, I would simply return a document to the
> shepherd when I saw nits that I thought should have been
> addressed. Wether those came from idnits or from scanning the
> draft was irrelevant, as was what status RFC4858 has.

If the process involves a discussion between a sponsoring AD and
the shepherd or WG, then that is a management issue and I really
don't care what that AD uses as a basis or authority.  I like it
a lot better if issues of that sort are raised prior to IETF
Last Call rather than after the LC closes, but that, too, is  a
management issue for the AD to work out with his or her WGs.
Once the document goes into IETF review and voting, I think
things change.

> When I
> saw those nits during the review process, I put them into
> comments so they could be addressed, or I raised occasionally
> raised discusses where I felt they really needed to be fixed.
> Where is the problem?

There is a distinction that I see very clearly but that you may
not consider significant.   If you say "DISCUSS because I've
studied this particular document and this is a real and specific
problem with it because it makes it harder to understand", I
don't think there is a problem.  If you say "DISCUSS because the
document violates the xyz rule", then I think the rule has to
represent community consensus or the actual authority of the
IESG to make such rules.

>> If the IESG wants to use it that way (and, by the way, I don't
>> think that is unreasonable in principle), then it needs to be
>> handled the way we normally handle these things: if RFC 4858
>> is going to be cited as justification (as Russ's note does),
>> then it needs to be updated and reclassified as a BCP.
> 
> I disagree. Not everything the IESG does needs to be
> documented in detail in a BCP, and especially not minutiae.

First of all, I didn't suggest a general requirement for BCPs.
While I think it is dangerous in some cases, I'm generally ok
with IESG statements where procedural matters are concerned --
as long as those statements are sufficiently formalized that the
community has the opportunity to object and have the objections
taken seriously.

More important, see the distinction above.  If a reference to
the rule --rather than a specific complaint about a specific
problematic inclusion or omission in a specific document-- is
going to be used as the basis of DISCUSS votes, I don't think we
are talking about minutiae.  Note that the discussion in
http://www.ietf.org/iesg/statement/discuss-criteria.html says
"The specification was not properly vetted against the I-D
Checklist". is, I think, consistent with what I'm suggseting.
While I [still] have problems with such a reference to an
evolving document, I note that statement references the
checklist and not the idnits tool.  More important, when it
illustrates "not properly vetted" it says "Symptoms include
broken ABNF or XML,..."  I suggest those are qualitatively
different and affect the actual usability of the technical
specification (maybe not "impossible" as mentioned in the first
bullet in that list, but serious) and not "you didn't mention a
historical (sic) or procedural relationship where someone thinks
you should have".  That sentence then goes on to say "...missing
Security Considerations, and so on."   The requirement for a
Security Considerations section did get wide community
discussion at the time, discussion that focused on its
importance to a well-specified protocol description.

I won't comment further on the "and so on" or the actual
definition of "properly vetted" beyond noting that both are,
IMO, hand waving that the IESG would never tolerate in a
protocol specification. 
> 
>> IMO, if they are normative requirements that justify a
>> DISCUSS, they are not nits.   On can reasonably comment on
>> nits.  But, if something is sufficiently serious for an IESG
>> member to block progress of a document, even for an IESG
>> meeting cycle, over a "nit" just makes the AD seem silly or
>> unnecessarily obstructionist (behavior about which I'd hope
>> Nomcoms would take notice).
> 
> So maybe the term "nits" is confusing. To me, at least, nits
> are all kinds of non-technical issues. Often, they're almost
> editorial (e.g., missing security considerations), but
> sometimes, they can rise to the level of a discuss. (Example:
> missing normative reference to a spec, wrong document status,
> etc.)

And I don't have any problems with DISCUSSes for missing
normative reference to a spec or wrong document status.  Both of
them are substantive issues.   My problem involves the
combination of references to non-consensus documents, especially
ones that have no clear change control procedures, turning
things that ought to be matters of editorial judgment into
requirements whose violation becomes the basis for insisting on
document changes.

>...
>> Sure.  We agree that is a good idea.  But, when the checklist
>> is realized as an automatic tool, and the tool makes
>> mistakes, it is perfectly reasonable as things are now
>> written for the shepherd to simply ignore those errors.  If
>> the sequence 
>> 
>> 	-- shepherd ignores errors as irrelevant
>> 	-- AD issues DISCUSS because the nits checker finds some
>> 	stuff (i.e., reports its errors as problems)
>> 
>> then we have a procedural problem and a practical one in that
>> the procedure and tools are making things harder and more
>> time-consuming, not easier.
> 
> Normally, the shepherd would document that the tool had false
> positives or negatives, and the AD would not raise an issue
> about it. But even if the shepherd forgot, or the AD didn't
> remember the shepherd's notice, we're at most talking one
> quick email exchange between the two to get things clarified.
> Compared to all the other delays, this one doesn't register on
> the radar.

I've had demonstrations to the contrary within the last month.
As far as what registers on the radar or not: (i) that isn't the
point and (ii) I note that our standards process has gone from
being the envy of many other SDOs for its speed and efficiency
to being really, really, slow.  As far as I can tell, that
slowness is not the result of either a decision to proceed
slowly or of any one particular procedure or bottleneck.  It is,
instead, the cumulative effect of a whole series of little
delays that, individually, "don't register on the radar".

>>   If what the IESG _really_ wants is
>> to have a copy of the nits-checker output attached to the
>> shepherd report, annotated with comments on every outstanding
>> putative issue, then the IESG needs to ask for it, not hope
>> shepherds will guess.
> 
> What the IESG wants - or rather, what I wanted when I was AD -
> was nit-free documents entering IESG review. (The idnits
> report is only a click away anyway on the tools servers.)

If the idnits tool sets the criteria (independent of my other
concerns about that), then you better figure out how to develop
an idnits tool that doesn't confuse section numbers with IP
addresses, email local-parts with periods in them with FQDNs,
references to predecessor documents (especially those that the
I-D obsoletes) to establish relationships with improper
references to documents that have been superceded, complain
about material in section that the RFC Editor is expected to
remove, and so on.  I think that development of such a tool is
likely to be very hard if not almost impossible, especially if
its input is plain text I-Ds.   The alternative is that you
aren't expecting "nit-free" as defined by the tool.   You are
expecting either an assertion that all relevant nits have been
examined and dealt with (an assertion you expect to trust) or
something that functionally resembles the annotated output I
suggested so that you can assure yourself that everything
important has, indeed, been dealt with.

>>  And, if failure to provide that annotated
>> output, or to make the answers clear enough, becomes the basis
>> for DISCUSS votes (rather than, e.g., a discussion between the
>> sponsoring AD and the shepherd and/or WG leadership), then the
>> request needs to be in something normative with community
>> consensus.
> 
> This sounds like you believe that we should have a set of
> community-approved BCPs that document what issues ADs are
> allowed to raise discusses on. I hope I'm misunderstanding?

I note that we already have an IESG statement that is more or
less community-approved that documents what issues ADs are
allowed to raise DISCUSSes on.  That implies to me that the IESG
concluded that such a set of criteria was useful and appropriate
(and, if I recall, did so while you were on the IESG).    My
_only_ objection to that document in this context is that it
makes a normative reference to an unstable document (in the RFC
Editor sense) that is not offered for review to the community
when it changes and for which there is no clear procedure for
objecting and that it treats what should be editorial
suggestions in that document as fully as important as things
that really do affect document usability.  So I'm not sure where
the misunderstanding lies.

> Aren't we trusting NomCom to select technical experts that
> will make reasonable choices in the best interest of the
> community, without handholding? (And when they don't there's
> the appeals process.)

I trust the NomCom process to select people who, in their
judgment, usually represent a compromise between the rather
large spectrum of skills and experiences that would characterize
a hypothetically-perfect AD and the people who are available in
the community. I worry a lot when there are large discrepancies
between that ideal and the available candidate pool, I worry a
lot, but it isn't the NomCom's fault.  As one small but relevant
example, it doesn't necessarily follow that a technical expert
in a particular field is also an expert on editorial matters
generally and the design and construction of technical documents
in particular.  Given a choice between those two types of
expertise, I think most of the community would choose technical
experts with general good judgment.

As far as the appeals process is concerned, I believe (even when
I'm disagreeing with them) that today's IESG consists of folks
who are smart, thoughtful, responsive to the community and who
care deeply about the IETF and the Internet.  I believe that the
overwhelming percentage of the members of prior IESGs fit that
description too.  From that point of view, my purpose in having
conversations like this is precisely to clarify the issues so
that the IESG can review its procedures and criteria as they
then think appropriate.  If no one --on or off the IESG-- ever
says anything, then there is no reason to believe such a review
would occur.  I think this type of discussion is far more useful
and beneficial to all concerned than having to try to get
reforms by appeal, especially appeals that are ultimately based
on the argument that the present procedures and their
interpretation are inadequate to ensure the rights of all
parties to a fair and open Standards Process.

>> Sorry.  You understand it.  I understand it.  A shepherd who
>> has been through the process two or three times previously
>> probably understands it.  You've been on the IESG recently
>> enough to have a lot of knowledge about what the IESG thinks
>> it needs.  Even I've been on the IESG recently enough to be
>...

> Shepherds will learn this when they take their first document
> through the IESG and work with their AD. And in regards to the
> tutorial document, that was the goal behind RFC4858. I think
> it's a decent enough intro, and the details of the process
> must be learned hands-on.

And therein lies a topic on which we may need to agree to
disagree.  When I look at 4858, I see very little useful and
specific information that crosses the boundary of where it
incorporates the Checklist by reference (especially when the
Checklist and proactical application then incorporate the idnits
tool by reference).  I also note that ADs are not completely
consistent in what they expect (perhaps a good thing) and that
learning the process details hands-on is a sufficiently
inefficient process to strongly bias any sensible WG Chair
toward choosing only shepherds who have done the job multiple
times before.  I think the IETF is seriously deficient in
leadership development and that the shepherd role could be an
excellent way for people to learn how the process actually
works.   But, if that has to depend on shepherds learning, one
on one, everything that is expected or assumed that isn't in
4859 from already-overworked ADs, we cannot afford to use the
shepherd role as a leadership development one very often.  That
is why documents and even training courses were invented: to
permit dissemination of a larger fraction of the required
knowledge and skills without requiring extensive one-on-one
interactions with ADs.

>...
>> Years ago, I tried to argue that deployment and use of a
>> well-documented standards-track protocol for years and years
>> constituted implicit community consensus that it was a [full]
>> standard (aee draft-ietf-newtrk-promotion if you are curious,
>> but its contents really aren't important to this discussion).
>> The community, and especially the IESG, wanted nothing to do
>> with the idea: there is no such thing as "implicit community
>> consensus".
> 
> So this is getting off-topic, but you could see the recent
> approval of the two-stage process as a sign that this thinking
> has changed.

One could, but that is really a separate discussion.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Mon, 31 Oct 2011 11:14:06 -0400

best,
   john