Subject: A non-IPR issue (was: Re: DRAFT Revision to the WG Document Writeup)

From: John C Klensin <[email-masked]>
Date: Sat, 29 Oct 2011 23:33:13 -0400

--On Thursday, October 27, 2011 13:40 -0400 Russ Housley
<[email-masked]> wrote:

> The IESG is considering adopting this update the the Document
> Writeup.  Please let us know if this causes any concern.
>...

It causes me one that is more of a process problem.  The problem
doesn't originate with this draft, but the draft highlights it.
Since all of the other comments have been about either IPR or
compound questions, I thought it worth starting another thread.

>...
>   (8) Has the Document Shepherd personally verified that the 
>       document satisfies all ID nits? (See the Internet-Drafts
> Checklist        and http://www.ietf.org/tools/idnits/).
>...

As long as this effort is about WG document writeups to give the
IESG information in more useful form, reviewing it with the WG
Chairs list is entirely reasonable (although I'd be happier if
the review included recent and prospective WG Chairs and
Shepherds, I recognize there is no such list). But I think it
would be extremely unfortunate if we found ourselves making
policy that affects the whole community --and that should
represent community consensus-- through the back door. 

I find the part of paragraph 8 quoted above particularly
problematic.  

First, the nits-checker is an imperfect tool based on a
collection of heuristics.  It works well for some documents;
others completely confuse it.  If what the IESG really wants is
a listing of what the nits-checker finds (if anything),
annotated with what the WG wants to do about it or why the
nits-checker finds issues, that is what you should ask for.
Requiring that the  "document satisfies all ID nits" turns the
nits-checker into a conformance enforcer.  Based on experience,
that isn't what the IESG wants.  And certainly it would be hard
to point to community consensus for that interpretation of the
tool.

Second, no matter how much input they have gotten from either
the IESG or this list, neither the list of things the nits
checker looks for nor the checklist are, as far as I can tell,
community consensus documents.  Most of their provisions come
out of either community consensus document or RFC Editor
requirements, but some do not.    We should be very careful,
IMO, to avoid getting ourselves into a situation in which adding
something to the checklist or nits-checker because they are good
things to check and be aware of turns into a requirement on
documents in the IETF Stream without first getting wide
community exposure and, ideally, consensus.  I can live with
making a highly-visible announcement and then treating silence
as consent in situations like this, but (accidentally or
deliberately) sneaking a new requirement in via the checklist or
nits checker too closely resembles playing "gotcha" with WGs and
document editors.

    john

---

From: Lars Eggert <[email-masked]>
Date: Mon, 31 Oct 2011 09:32:33 +0200

Hi,

> to point to community consensus for that interpretation of the

this text - I think - comes from RFC4858. 

What the IESG saw at least a few years ago when RFC4858 was written/published were drafts entering IESG review with a long list of nits that in theory should have been fixed long before but in practice weren't. This wasted time. The purpose of (8) was to make it clear to the shepherd that it was *their* job to make sure the nits are fixed *before* sending a draft to the IESG.

The canonical check list is the "I-D Checklist" and the idnits tool is - as you correctly point out - a quick heuristic to check many of those nits. But we all know if has false positives and negatives. The shepherds understand this, too, and will typically say something like "idnits complains about X and Y but that is incorrect", in which case the AD will simply note this and move on.

Maybe the wording needs to make it more clear that the "I-D Checklist" is the canonical list of nits that need to be checked against, and the idnits tool is merely a convenient heuristic?

> nits checker too closely resembles playing "gotcha" with WGs and

We've been using the I-D Checklist for years, and I've found it very useful both as an author and when I was on the IESG. (And the idnits tool is great, too.) So in practice, since we've used this for years and years, I'd argue that in fact we have implicit community consensus here. Which nits in particular do you object to?

Lars

---

From: John C Klensin <[email-masked]>
Date: Mon, 31 Oct 2011 08:30:56 -0400

--On Monday, October 31, 2011 09:32 +0200 Lars Eggert
<[email-masked]> wrote:

>> As long as this effort is about WG document writeups to give
>> the IESG information in more useful form, reviewing it with
>> the WG Chairs list is entirely reasonable (although I'd be
>> happier if the review included recent and prospective WG
>...

As Pete pointed out before I got to it (connectivity problems
this weekend), the problem is not with 4858.  It is that 4858 is
an Informational document (not a process BCP) that invokes the
ID-Checklist web page and idnits tool.   It does that without
even citing them so the reader can determine what is supposed to
be normative and what is not.

That would actually be fine with me as long as it is all
informal guidance, based on the opinions of the authors, or even
agreement with the PROTO team.  Indeed, as the Acknowledgements
section of 4858 says, the idea of using such write-ups in the
IETF partially originated with me.  From my point of view, the
idea was to reduce both the load on the IESG that resulted from
tracking processing issues down and on WGs/WG Chairs/Shepherds
by permitting one write-up with a known list of issue to replace
a lot of back and forth about one question at a time.  I was
also concerned about pressures to put material into I-Ds and
RFCs that was important in document review and processing but
would be clutter in permanent documents.

The problem arises when it (including not just the published and
stable RFC 4858 but the evolving web page and tools) starts
being treated as normative.  At least IMO, it becomes normative
the moment ADs start recording DISCUSS positions based on
non-conformance to the outline, the checklist, or the nits
checker.  It makes no difference whether those are
question-DISCUSSes ("I need more information about this or we
need to talk about it") or blocking-DISCUSSes -- the point is
that the checklist and nits checker have suddenly become
sufficiently normative that nothing further is needed to justify
a DISCUSS.

If the IESG wants to use it that way (and, by the way, I don't
think that is unreasonable in principle), then it needs to be
handled the way we normally handle these things: if RFC 4858 is
going to be cited as justification (as Russ's note does), then
it needs to be updated and reclassified as a BCP.  In the
process of doing that, things that it requires --the checklist
and nits-checker-- need to be treated as normative references
and to meet the stability requirements for such references.  A
normative reference to a web page that can be changed without
community review would never be tolerated in a technical
specification; this situation should be no different.

IMO, if they are normative requirements that justify a DISCUSS,
they are not nits.   On can reasonably comment on nits.  But, if
something is sufficiently serious for an IESG member to block
progress of a document, even for an IESG meeting cycle, over a
"nit" just makes the AD seem silly or unnecessarily
obstructionist (behavior about which I'd hope Nomcoms would take
notice).

> What the IESG saw at least a few years ago when RFC4858 was
> written/published were drafts entering IESG review with a long
> list of nits that in theory should have been fixed long before
> but in practice weren't. This wasted time. The purpose of (8)
> was to make it clear to the shepherd that it was *their* job
> to make sure the nits are fixed *before* sending a draft to
> the IESG.

Sure.  We agree that is a good idea.  But, when the checklist is
realized as an automatic tool, and the tool makes mistakes, it
is perfectly reasonable as things are now written for the
shepherd to simply ignore those errors.  If the sequence 

	-- shepherd ignores errors as irrelevant
	-- AD issues DISCUSS because the nits checker finds some
	stuff (i.e., reports its errors as problems)

then we have a procedural problem and a practical one in that
the procedure and tools are making things harder and more
time-consuming, not easier.   If what the IESG _really_ wants is
to have a copy of the nits-checker output attached to the
shepherd report, annotated with comments on every outstanding
putative issue, then the IESG needs to ask for it, not hope
shepherds will guess.  And, if failure to provide that annotated
output, or to make the answers clear enough, becomes the basis
for DISCUSS votes (rather than, e.g., a discussion between the
sponsoring AD and the shepherd and/or WG leadership), then the
request needs to be in something normative with community
consensus.

> The canonical check list is the "I-D Checklist" and the idnits
> tool is - as you correctly point out - a quick heuristic to
> check many of those nits. 

I'm not sure what "canonical" means if we are talking about a
web page that is revised from time to time, without a clear
process for approving the updates that involves community
consensus or, at worst, an appeal-able decision.

> But we all know if has false
> positives and negatives. The shepherds understand this, too,
> and will typically say something like "idnits complains about
> X and Y but that is incorrect", in which case the AD will
> simply note this and move on.

Sorry.  You understand it.  I understand it.  A shepherd who has
been through the process two or three times previously probably
understands it.  You've been on the IESG recently enough to have
a lot of knowledge about what the IESG thinks it needs.  Even
I've been on the IESG recently enough to be able to make good
guesses about how a question should be answered and the answer
backed up.  But we don't require that shepherds be former ADs,
or that they have been through the process a few times
previously, or that they pass a test on report-filling-out.
With reference to another thread, we don't provide a good
tutorial document on what is expected.   And, even if EDU
training were perfect, reached larger audiences, and wasn't
"another meeting", we would have a problem with someone who is
appointed as shepherd for the first time right after an IETF
meeting.   

> Maybe the wording needs to make it more clear that the "I-D
> Checklist" is the canonical list of nits that need to be
> checked against, and the idnits tool is merely a convenient
> heuristic?

That would help, but, as I hope is clear from the above, it is
where I'm having a problem right now.

> We've been using the I-D Checklist for years, and I've found
> it very useful both as an author and when I was on the IESG.
> (And the idnits tool is great, too.) So in practice, since
> we've used this for years and years, I'd argue that in fact we
> have implicit community consensus here.

Years ago, I tried to argue that deployment and use of a
well-documented standards-track protocol for years and years
constituted implicit community consensus that it was a [full]
standard (aee draft-ietf-newtrk-promotion if you are curious,
but its contents really aren't important to this discussion).
The community, and especially the IESG, wanted nothing to do
with the idea: there is no such thing as "implicit community
consensus".  Many people would argue that any decision about
community consensus requires a Last Call and formal decision.
I'm more flexible and, under some circumstances, IESG making
announcements about decisions and assuming community consensus
in the absence of an outcry or appeal. But the I-D Checklist
(which, I note, was last updated in May 2009, long after 4859
was published, does not have a change log, is not listed as an
IESG Statement, and which, as far as I know, has not had an
announcement or call for comments when it has been revised)
isn't part of that process.

> Which nits in particular do you object to?

That isn't really the point.  As I hope is clear from the above,
I'm having a fundamental process problem here that is not about
any particular requirement.  

If you are asking what set me off sufficiently to switch from
muttering into my beard to actively complaining and thinking
about process appeals, the answer would be the set of provisions
(not nits, see above) that require that, if a particular RFC
number appears in the document header (with "updates" and/or
"obsoletes"), it must be mentioned in the Abstract.   I think
Abstracts need to provide the reader with essential information
about what the document is about, and only that essential
information.  I think their style and content should be largely
matters for the RFC Editor as part of maintaining a consistent
style for the series, cross-stream.   Worse, unless the RFC
number is extremely well-known, listing an RFC by number in the
abstract is an implicit citation which violates (or at least
pushes the limits of) the RFC Editor's rule against citations in
abstracts, a rule that, ironically, the nits-checker checks.
I think it would be entirely reasonable for the checklist to say
"if such documents are not listed in the abstract, has the WG
considered whether they are important enough to list?".  But
sometimes they are and sometimes they aren't and creating a firm
requirement on the subject should require IETF consensus even
after getting input, and preferably agreement, from the RFC
Editor.

     john

---

From: Lars Eggert <[email-masked]>
Date: Mon, 31 Oct 2011 15:12:00 +0200

Hi,

> be normative and what is not.

I agree that those references should have been in RFC4858.

> checker.

I disagree. When I was on the IESG, in order to make the best use of my time, I would simply return a document to the shepherd when I saw nits that I thought should have been addressed. Wether those came from idnits or from scanning the draft was irrelevant, as was what status RFC4858 has. When I saw those nits during the review process, I put them into comments so they could be addressed, or I raised occasionally raised discusses where I felt they really needed to be fixed. Where is the problem?

> it needs to be updated and reclassified as a BCP.

I disagree. Not everything the IESG does needs to be documented in detail in a BCP, and especially not minutiae. 

> obstructionist (behavior about which I'd hope Nomcoms would take

So maybe the term "nits" is confusing. To me, at least, nits are all kinds of non-technical issues. Often, they're almost editorial (e.g., missing security considerations), but sometimes, they can rise to the level of a discuss. (Example: missing normative reference to a spec, wrong document status, etc.)

> time-consuming, not easier.

Normally, the shepherd would document that the tool had false positives or negatives, and the AD would not raise an issue about it. But even if the shepherd forgot, or the AD didn't remember the shepherd's notice, we're at most talking one quick email exchange between the two to get things clarified. Compared to all the other delays, this one doesn't register on the radar.

>   If what the IESG _really_ wants is
> shepherds will guess.

What the IESG wants - or rather, what I wanted when I was AD - was nit-free documents entering IESG review. (The idnits report is only a click away anyway on the tools servers.)

>  And, if failure to provide that annotated

This sounds like you believe that we should have a set of community-approved BCPs that document what issues ADs are allowed to raise discusses on. I hope I'm misunderstanding? Aren't we trusting NomCom to select technical experts that will make reasonable choices in the best interest of the community, without handholding? (And when they don't there's the appeals process.)

> appointed as shepherd for the first time right after an IETF

Shepherds will learn this when they take their first document through the IESG and work with their AD. And in regards to the tutorial document, that was the goal behind RFC4858. I think it's a decent enough intro, and the details of the process must be learned hands-on.

> consensus".

So this is getting off-topic, but you could see the recent approval of the two-stage process as a sign that this thinking has changed.

Lars