Subject: Who wanted to eliminate the tokenizer?

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Feb 1999 04:35:57 -0000

It seems that DRUMS is being blamed for a serious mistake in 822bis
that wasn't the fault of the group.

RFC 822 specifies that structured field bodies are split into tokens;
then spaces, tabs, and comments are removed; then the result is parsed.
(See http://pobox.com/~djb/proto/immhf.html for a detailed explanation.)

Pete Resnick released msg-fmt-00 in late 1996. It turned out that he had
eliminated the tokenizer. Three people stated strong objections. Resnick
gave his reasons for eliminating the tokenizer. He didn't respond to
further objections.

Now a DRUMS participant claims that there was ``VERY strong working
group consensus'' on removing the tokenizer:

   The DRUMS working group deliberated on this point quite carefully ...
   the agreement to do the removal was quite strong.

He can't seem to recall when these discussions happened, and can't seem
to find any evidence of them in the DRUMS archive, but he continues to
claim that ``there is strong working group consensus on this point.''

Does anyone remember such discussions? Does anyone remember asking for
the tokenizer to be removed? I thought that Resnick considered this to
be within his own discretion as editor.

Please send responses directly to me; I'll summarize.

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Thu, 11 Feb 1999 13:04:18 +0800

>Now a DRUMS participant claims that there was ``VERY strong working
...
>He can't seem to recall when these discussions happened, and can't seem

Dan,

1.  You have confused two different submissions from two different 
people.  I submitted the "claim" but have never made a comment about the 
details or timing of the cited discussions.  The "can't seem to recall" 
note was from someone else.

2.  You continue to pursue entirely the wrong question.  It is not 
productive to seek a review of history, absent substantial support for the 
underlying technical objection that your are putting forward.

Unless and until you muster serious support for that TECHNICAL objection, 
Dan, beating up the process is nothing more than beating up the 
participants of the group.

d/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PRESIDENTS DAY OPEN HOUSE, 2/13
<http://www.brandenburg.com/misc/presday/presday-invite.gif>

Dave Crocker                                       Tel: +60 (19) 3299 445
<mailto:[email-masked]>             Post Office Box 296, U.P.M.
                                         Serdang, Selangor 43400  MALAYSIA
Brandenburg Consulting
<http://www.brandenburg.com>                       Tel: +1 (408) 246 8253
Fax: +1(408)273 6464             675 Spruce Dr., Sunnyvale, CA 94086  USA

---

From: "D. J. Bernstein" <[email-masked]>
Date: 11 Feb 1999 06:21:49 -0000

Kai Henningsen opposed Resnick's elimination of the tokenizer. So did
Paul Overell. So did I. Several people have subsequently commented that
the 822bis grammar is much more painful than the 822 grammar.

Two people, Dave Crocker and Chris Newman, appear to support Resnick's
elimination of the tokenizer.

If it's true that there were ``deliberations'' leading to ``VERY strong
consensus'' for the removal of the tokenizer, surely there are dozens of
people who can remember participating in those ``deliberations'' and
expressing their support. Somebody? Hello? Is there anybody out there?

Dave Crocker writes:
> You have confused two different submissions from two different people.

No, I have not.

I specifically asked you when these ``deliberations'' took place. I also
asked you where I could find a record of these ``deliberations'' in the
DRUMS archives. You dodged both questions.

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Thu, 11 Feb 1999 15:29:37 +0800

>expressing their support. Somebody? Hello? Is there anybody out there?

indeed.  if you have strong support for your objections, I'm sure those in 
your camp will come forward...  and lest you try the obvious counter, no, 
the working group process does not require that every positive claim for 
action be documented.  this working group has gone for quite a long time 
(in case you hadn't noticed).  the requirement for challenges is to show 
support for objections.  i note that none has been forthcoming.

>DRUMS archives. You dodged both questions.

I didn't dodge them, I ignored them.  The difference is important.  Instead 
you attributed particular meaning to the lack of a response from me, when 
that meaning was entirely off the mark.  It would be helpful for you to 
show a tad more care in interpreting things.

Gosh, Dan, shall we now argue about what I meant, rather than stay focused 
on the main topic?  After all, it's so much more fun to go down these rat 
holes than focus on content issues, isn't it?

d/

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Feb 1999 01:12:22 -0000

Dave Crocker writes:
> action be documented.

You and the chair are claiming ``consensus'' on an action by the editor,
even though nobody else in the group seems to remember ever supporting
that action.

What would stop a dishonest working group chair and editor from making
false claims of consensus, with exactly the same level of evidence?

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Fri, 12 Feb 1999 10:46:31 +0800

>even though nobody else in the group seems to remember ever supporting

Dan, you persist in this false "nobody else in the group seems to remember" 
claim.  Only one person has claimed to lack sufficient memory of the 
PRECISE timing of the supporting action and NOBODY has claimed no memory of 
support.

Nobody but you, of course.  Surely you have noticed the lack of others 
supporting your latest challenge?

In any event, you are constantly attributing very detailed meaning to an 
utter absence of data.

Doesn't that bother you just a tad?

>false claims of consensus, with exactly the same level of evidence?

An entirely reasonable question.  It gets into the question of protecting 
against Type I and Type II errors, namely false rejection versus false 
acceptance.

IETF working groups may work in a "default yes" or "default no" mode or 
hybrid jode, as it sees fit, with the chair declaring the need for explicit 
support for, or against, a decision to be shown.  In different 
circumstances, one mode or the other can be vastly more efficient depending 
upon the topic and the working group dynamics.

The protection against having the chair make an incorrect declaration is 
that the working group, itself, will challenge the chair's 
declaration.   In other words, there will be a working group rough 
consensus that the chair has made an error.  This happens periodically.

There is also the formal challenge process which invokes an IESG review of 
working group process.

d/

---

From: "D. J. Bernstein" <[email-masked]>
Date: 12 Feb 1999 06:18:34 -0000

Dave Crocker claims that ``the DRUMS working group deliberated on this
point quite carefully'' and ended up with ``VERY strong consensus'' to
eliminate the tokenizer.

C'mon, folks, who remembers participating in those ``deliberations''?

Look at this from the point of view of, for example, an FTC attorney
reviewing the DRUMS archives. At this point he'll be fairly confident
that Crocker's story is not true. Is that what you want him to think?

Dave Crocker writes:
> claim.

``Nobody else seems to remember'' is a perfectly accurate description of
the discussion so far. Nobody else has spoken up to confirm your story.

Of course, it has only been a day since I started this thread; some
DRUMS readers may be behind on email. But if DRUMS had actually
``deliberated'' this issue then people would be jumping up to say so.

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Fri, 12 Feb 1999 16:20:34 +0800

>that Crocker's story is not true. Is that what you want him to think?

Fine, Dan.  Go get one of those attorneys lined up to support you. It will 
be a relief to have you waste their time rather than ours.

IETF procedures and the implementation of those procedures have been 
developed over the years with a substantial attention to questions of legal 
challenge, as well as basic concerns for fairness.  While I've no doubt 
that one or another IETF piece of work will yet be legally challenged and 
that one such challenge will succeed, I am equally confident, Dan, that the 
DRUMS working group's efforts won't suffer any such outcome.

You like to talk about dodging points and you like to interpret absent 
data, yet you fail to respond to the fact that you are getting no support 
from anyone else in the working group.

NOT ONE OTHER PERSON.

Take that to an attorney, Dan, and see what sort of advise you get from them.

>``deliberated'' this issue then people would be jumping up to say so.

Oh well then.  As your strong and broad base of support develops, of course 
the handling of your silliness will need revision...

d/

---

From: Brad Knowles <[email-masked]>
Date: Fri, 12 Feb 1999 10:17:17 +0100

>There is also the formal challenge process which invokes an IESG review of 

    Of course, that actually requires that there is someone over at the
IESG who hasn't yet set up procmail so as to /dev/null everything coming
from Dan, so maybe that step might be a little hard for him to take.

-- 
  These are my opinions -- not to be taken as official Skynet policy
 ____________________________________________________________________
|o| Brad Knowles, <[email-masked]>            Belgacom Skynet NV/SA |o|
|o| Systems Architect, News/mail/FTP Admin   Rue Col. Bourg, 124   |o|
|o| Phone/Fax: +32-2-706.11.11/12.49         B-1140 Brussels       |o|
|o| http://www.skynet.be                     Belgium               |o|
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
                The Sky is no longer the limit

---

From: Brad Knowles <[email-masked]>
Date: Fri, 12 Feb 1999 10:15:38 +0100

>that Crocker's story is not true. Is that what you want him to think?

    Is there a single person on this planet who is actually so stupid as
to think that the DRUMS archives are the sum total of all the discussions
that this working group has had on this topic?!?

    C'mon Dan, I'm engaged to a lawyer.  I have at least some inkling of
an idea what their thought processes are like.  I don't think it's
physically possible to pass the bar exam if your thought processes are so
very one-dimensional.  And anyone who brings up issues of "lawyerly
thought" without actually being able to make a semi-decent pass at the
topic is even dumber and more foolish than the lawyer who defends himself
in court.

    IANAL.  And I don't pretend to be one.  But when someone else does
pretend to be one, they had at least better be able to do so without
making a total public ass of themselves if they want to avoid getting a
little flamage from me on the topic.

    I strongly encourage you to step back and think about the time when
you were entering high school or college and you first came across the
phrase "it is better to be quiet and thought a fool than to open your
mouth and leave no doubt."

    Unfortunately, I don't think you're physically capable of sending out
ten e-mail messages without at least making some really stupid asinine
remark (and proving to everyone involved just how much of an ignoramus
you really are) in at least nine and a half of them.

    Fortunately for me, I can /dev/null everything you have to say. 
Unfortunately for the chair, the editor, and certain other members of
this working group and the IESG, they cannot do the same.

    Therefore, I refuse to take the easy way out, in hopes that I'll see
something incredibly stupid from you (like this) and be able to at least
assist in the defence against your infinite capacity for wrongheadedness
and dogged determination to destroy the process (and everyone involved)
if you're unable to get your way.

    Jeez.  There are times when you're so incredibly stupid that I
actually start to wonder about the quality of your crypto.  I mean if
you're this dense and incapable of understanding the reality of the
situation with regards to things like Internet mail, just how incredibly
bad is your crypto?  Or are you actually secretly working for the NSA and
actively trying to subvert any efforts that would make any real progress?

    Sorry folks, it's hard to see stuff like this without getting into ad
hominem responses.  I'll stop now.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 12 Feb 1999 19:20:59 +0800

Brad,

>from Dan, so maybe that step might be a little hard for him to take.

Alas as I noted in one of my broadsides back to the complainant, today, the 
folks who do IETF management feel compelled to suffer fools to a yeoman 
level.  If we had far more strict and detailed procedures -- in other words 
if we were so encrusted with formality that we had to attend more to the 
procedural details than to the real content -- the folks doing the 
management would have a much easier time of it.

I think our occasional dogfights in this community can be quite 
frustrating, but the real fights are based on real issues.

The sort of silliness that the current thread involves is purely a waste of 
their time.  That's why I decided to step in try to provide some direct 
response to the silly content of the complainant.  I don't hold any formal 
position -- I'm considering the ABNF document irrelevant to the current 
issues -- but do have some background in both the topic and the working 
group and want to make sure that the current record contains clear 
commentary to counter the silliness.  I'm hoping that this can streamline 
(limit) the effort IETF management has to devote to this wastage.

We've have working groups screw up periodically, so the challenge process 
is not just a matter of academic (and legal) purity and credibility.  It 
really does provide corrective benefit.

But the DRUMS working group has been a stellar model of good management and 
good working group participation, in my opinion.  Other than the 
broadly-held and continued frustration of slow turnaround by the editors -- 
and I note that for reportorial completeness, rather than as a personal 
complaint -- I believe that the working group deserves massive praise, 
rather than having to put up with the sort of crap now being generated by a 
lone, disgruntled source of disturbance.

d/

---

From: Keith Moore <[email-masked]>
Date: Fri, 12 Feb 1999 10:50:43 -0500

> false claims of consensus, with exactly the same level of evidence?

Objections from several working group participants would be highly 
likely to attract the attention of the AD, the IESG, or the IAB,
each of which can take corrective action.  

Keith

---

From: Keith Moore <[email-masked]>
Date: Fri, 12 Feb 1999 10:54:09 -0500

> C'mon, folks, who remembers participating in those ``deliberations''?

I recall expressing concerns about the change in the grammar 
to explicitly handle white space and comments, etc. and
the impact of that change on the readability of the grammar and 
the quality of implementations.  I also recall that several 
people expressed a preference that the explicit handling of
these remain in the document.  It was my judgement that the
working group consensus did not support my position.

Keith

---

From: "D. J. Bernstein" <[email-masked]>
Date: 13 Feb 1999 02:32:32 -0000

Keith Moore writes:
> I also recall that several people expressed a preference that the
> explicit handling of these remain in the document.

Earlier you claimed that ``a large number of DRUMS participants---
probably a rough consensus---argued for the changes to the grammar
notation.''

Now you say ``I recall that several people expressed a preference.''
``Several people,'' out of 50 or 60 people at a DRUMS meeting, is
supposed to be ``a large number of DRUMS participants''?

Is this supposed to confirm Dave Crocker's story? ``The DRUMS working
group deliberated on this point quite carefully---considering both
retention and removal''? ``VERY strong working group consensus''?

Does anybody other than Newman and Crocker remember speaking up in favor
of Resnick's elimination of the tokenizer?

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Sat, 13 Feb 1999 21:15:09 +0800

>of Resnick's elimination of the tokenizer?

For those not familiar with IETF procedures, Dan remains confused about 
what is appropriate at this stage of our work's life-cycle.

Anyone may, at any time, claim that working group process was followed 
inappropriately.

Anyone may, at any time, claim that a decision includes a technical errors 
which render the work inoperable.

Otherwise, questions such as "does anyone remember" or "i don't like" or 
any of the variety of other lesser challenges are out of order.

We've got procedures for pursuing valid challenges.  Dan's query is neither 
on-point nor pursued in a constructive and appropriate manner.

As such, let me encourage folks to ignore his query.  The only effect of 
doing otherwise will be to introduce further delay in the advancement of 
the working group's documents.

That work has taken long enough.

d/

---

From: Robert Elz <[email-masked]>
Date: Sun, 14 Feb 1999 17:29:19 +1100

Date:        13 Feb 1999 02:32:32 -0000
    From:        "D. J. Bernstein" <[email-masked]>
    Message-ID:  <[email-masked]>

  | Does anybody other than Newman and Crocker remember speaking up in favor
  | of Resnick's elimination of the tokenizer?

I recall having the discussions at some drums meeting or other.   It is
so long ago now (by guess would be in 96 or early 97) that I personally
have no idea who said what.  I do generally recall the consensus being
that the way the doc was being done was the way we ought to go.

The issue revolved around how we permit comments, white space, and
folding insertion - and the desire to be much more limited than "between
any two tokens" as to where all that ought to be permitted.   The old
tokenising behaviour with "noise" elimination at the lexical level made
that much more difficult, and basically made it impossible to prohibit
writing (for example) a time as

	18(hours): (min) 14 (utes)
		:
	   (and) 23 (seconds)

There has been (during several discussions) fairly strong consensus that
we want to restrict where comments, white space, and folding, are permitted.

Moving all this into the grammar allows all that, and allows it rationally.
It certainly makes the grammar more complex, both in now specifying more,
and because we have to have both the "must accept to conform with 822
implementations" and "only generate according to this" all in there.

If this were ever an issue that many people were particularly concerned
about there would have been much discussion about it on the list during
the past couple of years.  That there hasn't been is a very good indication
that over all the working group has no difficulty with the way the grammar
has been written.   That few people can recall a relatively non-controversial
discussion several years ago proves very little at all.

kre

---

From: "D. J. Bernstein" <[email-masked]>
Date: 14 Feb 1999 11:15:30 -0000

It has been three days now. If the working group ``deliberated on this
point quite carefully,'' and ended up with ``VERY strong consensus,''
then where are all the people who reached that ``consensus''?

Elz claims that the RFC 822 tokenizer ``basically made it impossible''
to prohibit generation of various newlines, spaces, tabs, and comments.
That's absurd. It isn't just possible; it's simple, clean, and very easy
to understand. See http://pobox.com/~djb/proto/immhf.html.

I'm not saying that there's consensus against Resnick's presentation of
the grammar. But there clearly isn't consensus _for_ that presentation.
Why are certain people falsely blaming the working group for actions
taken singlehandedly by the document editor?

---Dan

---

From: Dave Crocker <[email-masked]>
Date: Sun, 14 Feb 1999 23:43:00 +0800

>then where are all the people who reached that ``consensus''?

Again, Dan, you misunderstand the process.  There is no positive 
requirement that every time someone asks to review a matter that it be 
reviewed or that every time someone says 'prove that there was support' 
there needs to be a re-hashing of prior work or the support for it.

There is only one significant thing from this last sequence and that is 
that you continue to have not even one other person sharing your concerns.

Hence, you lack any constituency for pursuing the matter further.

d/

---

From: [email-masked] (Kai Henningsen)
Date: 14 Feb 1999 18:01:00 +0200

> the 822bis grammar is much more painful than the 822 grammar.

Just for the record, I didn't change my mind. I still think the current  
approach is seriously inferior to the former approach.

And I believe it is well-known that I am not a fan of Dan. In fact, I  
think his repeated attacks of "dishonesty" are themselves highly  
dishonest.

However, I also do not remember any consensus here. And I'm definitely  
still unhappy about that. For whatever reasons, though, I gave up on  
trying to change it.

It would be easier to get this stuff right if Dan were acting more like a  
civilized person, of course.

MfG Kai

---

From: "D. J. Bernstein" <[email-masked]>
Date: 15 Feb 1999 05:42:14 -0000

Dave Crocker writes:
> Again, Dan, you misunderstand the process.

I understand the process just fine. The document editor makes whatever
changes he wants to make.

In cases where the rest of us agree with the editor, there's no problem.

In cases where the rest of us disagree with the editor, the burden is on
us to (1) continue raising objections until the DRUMS management agrees
to allow the issue at an in-person meeting, (2) prove consensus at the
meeting, and (3) object again when the editor ignores the consensus.

In cases where some people agree with the editor and some don't, the
editor wins by default. His changes are then labelled ``consensus.''
Anyone who disagrees is challenged to prove consensus to the contrary.

Here's one of many examples.

RFC 821 says that an SMTP client ``should not'' close a connection
without waiting for a reply to QUIT. John Klensin changed ``should not''
to ``MUST NOT intentionally.''

As you read the rest of this story, please keep in mind that DRUMS
rejected Klensin's change at the August 1998 DRUMS meeting. According to
the minutes:

   Consensus is to change "MUST wait" to "SHOULD wait" (and some would
   prefer "MAY wait" or nothing at all).

Back in July 1996, I suggested a careful transition away from QUIT. Four
hours later, after a few people had expressed interest, Keith Moore said

   It's not worth the trouble to write it up, or even to argue about it.
   I believe we probably have rough consensus that QUIT should stay. Are
   there any objections other than from Dan?

When I reminded Moore that he had misrepresented DRUMS consensus before,
he said

   I'm reasonably confident that this will be the consensus. ... I don't
   want to see us waste time discussing yet another issue for which
   there isn't any significant support for changing the status quo.

After more people expressed interest, Chris Newman tried to cut the
discussion short by calling for ``concensus'' on his own minority views.
He subsequently apologized.

In August 1997 I announced that I was no longer going to wait for a QUIT
response in qmail. (Exim now behaves the same way; I haven't checked
other MTAs.) Two people responded with personal attacks.

In June 1998 I again objected to Klensin's ``MUST NOT.'' Nick Shelness
said

   Dan has been beating this dead horse for quite a while and seems to
   still be in a minority of one.

I responded: ``This is not the first time that the QUIT supporters, who
I suspect are a tiny minority among implementors, have claimed that
their position is the DRUMS consensus, and attempted to discourage
further discussion. Their behavior will make it easy for implementors to
discredit DRUMS.''

At this point Newman tried once again to close the discussion:

   I certainly don't find the QUIT issue compelling enough to delay the
   publication of a revised SMTP spec and extend the DRUMS working group
   for a few more months. Unless others do feel it is that compelling I
   will consider the issue closed.

Need I point out that the issue was raised in 1996? In any legitimate
standards organization it would have been settled at that time; there
would have been no need to point it out again two years later.

After more people spoke up in favor of my position, Newman tried _again_
to close the discussion:

   If there are fewer than two objections to any of the proposed
   resolutions here, I will assume rough concensus has been achieved.

QUIT was item #22; the proposed resolution was to leave Klensin's ``MUST
NOT'' in place. Newman didn't care that there had _already_ been more
than two objections; as usual, an extra burden was placed on people who
disagreed with the document editor.

After this attempt failed, Newman finally put ``reduce to SHOULD NOT''
on the list of issues to be discussed at the August 1998 meeting. The
result, as quoted above, was consensus against Klensin's change.

Klensin still didn't fix his document. (From his editing notes it would
seem that he somehow managed to misinterpret the meeting minutes.) I
caught this in January 1999, during the smtpupd Last Call.

Before Last Call, Newman had stated a policy formulated at the December
1998 IETF meeting: he would assume consensus against any objection that
wasn't stated by four people. But what does this mean when the document
was changed by the editor without authorization from the working group?

I posed a question for Newman: ``In cases like this, where DRUMS has
_already_ agreed that the document should be changed, are you still
demanding that four people speak up?'' For reasons that still haven't
been explained, Newman didn't respond.

After the Last Call period, Newman asked someone to prepare a summary of
the discussion, excluding objections that weren't raised by four people.
He hasn't stated that there will be a new Last Call period. I complained
to the IESG a week ago; Newman _still_ hasn't answered my question. Is
Klensin going to fix the document?

It probably didn't take Klensin more than a minute to make his change.
I've spent hours trying to get him to fix it---even though I'm in the
majority! The document editor's ideas aren't competing fairly with the
ideas of everyone else in the group.

---Dan

---

From: Chris Newman <[email-masked]>
Date: Tue, 23 Feb 1999 13:07:59 -0800 (PST)

> been explained, Newman didn't respond.

The answer to your question is no.  The reason I didn't respond is that
I've been too busy to wade through the hundreds of messages on the DRUMS
list.

		- Chris