Subject: Q re schedule and (premature?) implementation of drums-msg-fmt

From: "Sam Roberts" <[email-masked]>
Date: Wed, 2 Feb 2000 18:09:59 -0500

Hi. 

A few questions:

* Is drums on schedule? I though it had been abandoned since the schedule at
  http://www.ietf.org/html.charters/drums-charter.html slipped so far, and had
  subscribed to list expecting no traffic at all. Is it expected to produce an update
  to RFC822 by the end of 2000, say?

* I'm implementing a C++ toolkit for composing/decomposing rfc822 and
  mime messages. The recursive descent parser and naming conventions
  I'm using are based on the names and syntax from rfc822. The drums syntax
  appears clearer, especially w.r.t. where LWSP and comments can appear.
  Should I use it? I'd like to be forward looking, but am slightly wary of basing
  my library on a not-yet RFC. I'd like not to get a "huh?" when I say I've implemented
  the syntax from drums-msg-fmt but that its backwards compatible with rfc822.
  Another complication is that the MIME syntax references the rfc822 syntax, so
  I would be in the odd position of using a different parser for rfc822 messages than
  mime messages.

* Would it be helpful or just noise for me to implement the drums syntax, and
  have I arrived too late for meaningful/useful comments/questions?

* I'm considering whether to implement rfc2047 (mime header extensions).
  Is it actually used? Can anybody point me towards MUAs that compose/parse
  headers of that format that I could test against for interoperability?

* Are these questions inappropriate for this list?

Thanks,
Sam

--
Sam Roberts, sam at cogent dot ca, www.cogent.ca

---

From: Pete Resnick <[email-masked]>
Date: Wed, 2 Feb 2000 18:47:23 -0600

On 2/2/00 at 6:09 PM -0500, Sam Roberts wrote:

>* Is drums on schedule?

Not even close.

>Is it expected to produce an update to RFC822 by the end of 2000, say?

Jeez, I hope so. I hope much sooner.

>   have I arrived too late for meaningful/useful comments/questions?

It would absolutely be helpful. If you find anything glaring very 
soon, it can always get included. In any event, drums-msg-fmt will 
likely be released as a Proposed Standard. Implementation experience 
from that spec will be exceedingly important if we want it to advance.

>   I would be in the odd position of using a different parser for 

Obviously I'd like to see it done. If it turns out not to be backward 
compatible to 822 in any significant way, I'd sure like to know 
before it advances at all. You'll have to ask other implementors for 
their opinions; certainly we've put some of it into Eudora.

BTW: The opinion expressed in Dan Bernstein's message is rooted in 
his overall objection to the document in that it doesn't describe how 
to implement a parser. He is not, as I read it, saying not to 
implement those things required by the new document, but rather not 
to use it as a guide to implementation. (It doesn't, for instance, 
talk about whether or not to use a two-level parser.)

>   headers of that format that I could test against for interoperability?

It's definitely used. I would recommend *not* using Eudora as an 
example; it generates decent headers, but anything more than 
ISO-8859-1 will cause it grief.

>* Are these questions inappropriate for this list?

Not at all.

pr
-- 
Pete Resnick <mailto:[email-masked]>
Eudora Engineering - QUALCOMM Incorporated
Ph: (217)337-6377 or (858)651-4478, Fax: (858)651-1102

---

From: Ofer Inbar <[email-masked]>
Date: Sun, 6 Feb 2000 18:16:26 -0500

In 3.6.4,
> The "References:" field will contain the contents of the parent's
> "References:" field (if any) followed by the contents of the parent's
> "Message-ID:" field (if any). If the parent message does not contain a
> "References:" field but does have an "In-Reply-To:" field containing a
> single message identifier, then the "References:" field will contain
> the contents of the parent's "In-Reply-To:" field followed by the
> contents of the parent's "Message-ID:" field (if any). If the parent
> has none of the "References:", "In-Reply-To:", or "Message-ID:"
> fields, then the new message will have no "References:" field.
>
> Note: Some implementations parse the "References:" field to display
> the "thread of the discussion". These implementations assume that each
> new message is a reply to a single parent and hence that they can walk
> backwards through the "References:" field to find the parent of each
> message listed there. Therefore, trying to form a "References:" field
> for a reply that has multiple parents is discouraged.              

This text is not entirely clear on what should be done if the parent 
message does not contain a References: field but does contain an     
In-Reply-To: field with multiple identifiers.  The text of the second
paragraph here says, "a References: field for a reply that has
multiple parents" - but what the first paragraph doesn't cover is how
to form a References: field for a reply *to* a reply that has multiple
parents.  If the parent has multiple identifies in its In-Reply-To:,
and you are replying to that parent, your new message is a reply with
exactly one parent.  What do you put in the References: header? 

I can see some other vaguenesses and ambiguities in this pair of
paragraphs, but they get too confusing to think about until this one
central problem is fixed.  What did we really mean to say here?      

  --  Cos (Ofer Inbar)            --  [email-masked]  [email-masked]
  --  Exodus Professional Services            --  [email-masked]
 "Christ was miquoted.  He meant to say 'The GEEK shall inherit the earth'"
       -- Mathew A. Hennessy <[email-masked]>

---

From: Ofer Inbar <[email-masked]>
Date: Sun, 6 Feb 2000 20:00:12 -0500

> 3.3. Date and Time Specification
> 
> Date and time occur in several header fields. This section specifies
> the syntax for a full date and time specification. Though folding
> white space is permitted throughout the date-time specification, it is
> recommended that only a single space be used where FWS is required and
> no space be used where FWS is optional in the date-time specification;
> some older implementations may not interpret other occurrences of
> folding white space correctly.

My fuzzy memory tells me we agreed this should be a SHOULD
requirement.  The text here seems to be saying the same thing as a
SHOULD - if you don't do things the way we recommend, you will hurt
interoperability.  Is there any reason not to use SHOULD / SHOULD NOT
language to make this clearer?

I propose:
  "Though folding white space is permitted throughout the date-time
  specification, where FWS is required it SHOULD be only a single
  space, and where FWS is optional it SHOULD NOT be used; some older
  implementations may not correctly interpret a date-time that does
  not follow this rule."

However, I do see a problem with this rule:
date-time       =       [ day-of-week "," ] date FWS time [CFWS]
day-of-week     =       ([FWS] day-name) / obs-day-of-week
date            =       day month year
day             =       ([FWS] 1*2DIGIT) / obs-day

So, no FWS is *required* between the comma after day-of-week and the
day.  Also, no FWS i *required* at the beginning of a date-time.
Surely the intention wasn't to recommend Dates that look like:
  Date: Sun,06 Feb 2000 12:56:39 -0500
  Date:Sun, 06 Feb 2000 12:56:39 -0500

Is this a simple error?  I just searched my archive of 82,013 messages,
many of which date back as far as 1990, and found exactly one Date:
header line that had a day-of-week,day with no whitespace after the
comma.  And that one happened to be in a spam message.  I also found
only one Date: header line with no space after the colon.  I asked a
friend whose archive contains 44,667 messages, and she found none of
either of these forms.  Her archive only goes back about two years.

It seems to me that date-time should really be:

date-time       =       FWS [ day-of-week "," FWS ] date FWS time [CFWS]

Then day and day-of-week caan just be:
day             =       1*2DIGIT / obs-day
day-of-week     =       day-name / obs-day-of-week

Another way to change the syntax would be to leave date-time as is and
require FWS in day and day-of-week, though I think that's less readable:

date-time       =       [ day-of-week "," ] date FWS time [CFWS]
day-of-week     =       (FWS day-name) / obs-day-of-week
day             =       (FWS 1*2DIGIT) / obs-day

If we don't want to change the syntax, the other alternative would be
to explicitly state these two exceptions to the date-time whitespace
rule.  We do recommend a single space after the colon, and we do
recommend a single space after the comma, even though both are
optional in the syntax.  I think changing the syntax is clearer and
more elegant, but it does have the effect of relegating these two ugly
forms of Date: to the obsolete syntax altogether.  Given the tiny
number of messages I found with these forms, I don't think that's a
problem.

Whatever we do, we shouldn't let the document out in its current form,
because on careful reading it actually recommends the bad forms.

  --  Cos (Ofer Inbar)            --  [email-masked]  [email-masked]
  --  Exodus Professional Services            --  [email-masked]
  It's been said that if a sysadmin does his job perfectly, he's the
  fellow that people wonder what he does and why the company needs him,
  until he goes on vacation.     -- comp.unix.admin FAQ

---

From: Ofer Inbar <[email-masked]>
Date: Sun, 6 Feb 2000 20:21:40 -0500

In 3.2.3,
> Also, strings of characters that are treated as comments may be
> included in structured field bodies as characters enclosed in
> parenthesis.

Shouldn't that be "parentheses"?

3.2.3 includes
> However, where CFWS occurs in this standard, it MUST NOT be inserted
> in such a way that any line of a folded header field is made up
> entirely of WSP characters and nothing else.

Shouldn't something to this effect also be included in 2.2.3?

I propose adding, right after the Note: in 2.2.3,
  "A header line MUST NOT be folded in such a way that any line of the
  folded header field is made up entirely of white space and nothing else."

> 3.5 Overall message syntax
> 
> A message consists of header fields, optionally followed by a message 
> body. Lines in a message MUST be a maximum of 998 characters excluding 
> the CRLF, but it is RECOMMENDED that lines be limited to 78 characters 
> excluding the CRLF. (See the note in section 2.1 for explanation.) In

The wording is a little confusing, though if one thinks about it the
meaning is clear.  Still, "Lines in a message MUST be a maximum of 998
characters" ... that's poor.  The wording in 2.1 is a lot better.

How about this:
  "Lines in a message MUST NOT exceed 998 characters in length each,
  excluding the CRLF, and SHOULD NOT exceed 78 characters in length
  each, excluding the CRLF.  This applies to the header fields as well
  as the body of the message."

In 3.6.4,
in-reply-to     =       "In-Reply-To:" 1*msg-id CRLF

This should be,
in-reply-to     =       "In-Reply-To:" 1*msg-id CRLF / obs-in-reply-to

Section 4 does define an obs-in-reply-to, and that form appears to
still be in wide use.  For example, I am using mutt 0.95.6, a program
that is less than a year old, and it generates in-reply-to headers
that look like this:
  In-Reply-To: <[email-masked]>; from [email-masked] on Fri, Jan 28, 2000 at 06:41:05AM -0500

In 3.6.6,
> Note: Reintroducing a message into the transport system and using
> resent fields is a different operation from "forwarding". Forwarding
> has two meanings: First, there is a sense of forwarding in that a mail
> reading program can be told by the user to forward a copy of the
> message to another person is to make a message the body of a new
> message. A forwarded message in this sense does not appear to have

The sentence beginning with "First," is clumsy, at least.  How about,
  "First, forwarding may mean the case where a person uses a mail
  reading program to forward a copy of a message to another person,
  by making that copy be the body of a new message."

> Appendix A. Examples messages

Should be either "Examples" or "Example messages".

  --  Cos (Ofer Inbar)           --  [email-masked]  [email-masked]
  --  Exodus Professional Services           --  [email-masked]
  "OSI is a beautiful dream, and TCP/IP is living it!"
   -- Einar Stefferud <[email-masked]>, IETF mailing list, 12 May 1992

---

From: Pete Resnick <[email-masked]>
Date: Sun, 6 Feb 2000 22:33:50 -0600

>exactly one parent.  What do you put in the References: header?

I believe if you look back at the discussion we had on this topic, 
you will see that the answer to this question is "punt". There is 
*no* implementation precedence for how to deal with multi-parent 
messages, let alone doing it without screwing up news completely. 
Therefore, we decided to not comment on it, leaving it for someone 
else to write an Informational or Experimental document later.

pr

---

From: Pete Resnick <[email-masked]>
Date: Sun, 6 Feb 2000 22:45:19 -0600

On 2/6/00 at 8:00 PM -0500, Ofer Inbar wrote:

>>Date and time occur in several header fields. This section 
>>specifies the syntax for a full date and time specification. Though 
>>folding white space is permitted throughout the date-time 
>>specification, it is recommended that only a single space be used 
>>where FWS is required and no space be used where FWS is optional in 
>>the date-time specification; some older implementations may not 
>>interpret other occurrences of folding white space correctly.
>
>My fuzzy memory tells me we agreed this should be a SHOULD requirement.

I believe you are correct. That "recommended" simply needs to be 
changed to "RECOMMENDED" and it'll be fine.

>[...]
>[...]
>If we don't want to change the syntax, the other alternative would 
>be to explicitly state these two exceptions to the date-time 
>whitespace rule.

That's the easiest. I can change the language to:

Though folding white space is permitted throughout the date-time 
specification and is required in certain places, it is RECOMMENDED 
that a single space be used each place FWS (whether optional or 
required) appears.

>I think changing the syntax is clearer and more elegant, but it does 
>have the effect of relegating these two ugly forms of Date: to the 
>obsolete syntax altogether. Given the tiny number of messages I 
>found with these forms, I don't think that's a problem.

I'm a little worried to make that change at this stage unless I get 
buy-in from the chair and the ADs.

pr

---

From: Pete Resnick <[email-masked]>
Date: Sun, 6 Feb 2000 22:54:51 -0600

>Shouldn't that be "parentheses"?

Darn! I thought I squashed all of those. Fixed.

>3.2.3 includes
>>However, where CFWS occurs in this standard, it MUST NOT be 
>>inserted in such a way that any line of a folded header field is 
>>made up entirely of WSP characters and nothing else.
>

No, I disagree. 2.2.3 is general description, not syntax rules.

>>A message consists of header fields, optionally followed by a 
>>message body. Lines in a message MUST be a maximum of 998 
>>characters excluding the CRLF, but it is RECOMMENDED that lines be 
>>limited to 78 characters excluding the CRLF. (See the note in 
>>section 2.1 for explanation.) In
>
>meaning is clear.

(*Shrug*) Unless there's big objection, I'm leaving it alone.

>in-reply-to     =       "In-Reply-To:" 1*msg-id CRLF / obs-in-reply-to

No. You will find obs-in-reply-to in obs-fields in section 4.5, where 
it belongs.

>In 3.6.6,
>[...]
>The sentence beginning with "First," is clumsy, at least.

(*Shrug, again*) I'll leave these grammar things as something for the 
IESG to edit; people like to find *something* to correct. :-)

>Should be either "Examples" or "Example messages".

Yup. Fixed.

pr

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 7 Feb 2000 01:37:07 -0500

> else to write an Informational or Experimental document later.

Given that this is the case, I think you should be explicit about
it in the document.
-- 
		<a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a>

Courage is resistance of fear, mastery of fear, not absence of fear.

---

From: Pete Resnick <[email-masked]>
Date: Mon, 7 Feb 2000 00:37:32 -0600

On 2/7/00 at 1:37 AM -0500, Eric S. Raymond wrote:

>Given that this is the case, I think you should be explicit about it 
>in the document.

Why isn't this explicit enough?

Note: Some implementations parse the "References:" field to display 
the "thread of the discussion". These implementations assume that 
each new message is a reply to a single parent and hence that they 
can walk backwards through the "References:" field to find the parent 
of each message listed there. Therefore, trying to form a 
"References:" field for a reply that has multiple parents is 
discouraged.

pr

---

From: "Eric S. Raymond" <[email-masked]>
Date: Mon, 7 Feb 2000 01:49:58 -0500

> Why isn't this explicit enough?

I didn't see anything in the language that *says* the behavior is
meant to be unspecified; and if there were such language I suspect
the question that originated this thread would not have arisen.
-- 
		<a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a>

Live free or die; death is not the worst of evils.
	-- General George Stark.

---

From: Ofer Inbar <[email-masked]>
Date: Mon, 7 Feb 2000 03:09:20 -0500

Ofer Inbar <[email-masked]> wrote:
> parents.  If the parent has multiple identifiers in its In-Reply-To:,

> Given that this is the case, I think you should be explicit about

Yes, agreed, that is part of what I meant.

> for a reply that has multiple parents is discouraged.              

And here's an example of a simple thread:

Message 1 -+
Message 2 -+
           +-- Message 3 (reply to 1 and 2)
                 |
                 +-- Message 4 (reply to 3)

Now, I recall very clearly from our past discussions on this topic,
that we decided to punt References: for a reply that has multiple
parents, i.e. Message 3.  That is indeed what the second paragraph
says, and that's OK as far as it goes.  But saying something is
discouraged implies that it is allowed.  And, as Eric is pointing out,
if it is allowed, but isn't being defined, that needs to be explicit.

If you *do* decide to put a References: header in such a message,
we're not telling you how to form that header, except for the syntax.
If we don't say *that*, we leave the implementor puzzling over the
first paragraph trying to figure out whether we did or did not specify
what that (albeit discouraged) References header is supposed to
contain.  Just saying that it's discouraged isn't enough.

But that's only half the problem.  The other half is, as far as I can
tell, we've said *nothing* at all Message 4's References: header.
  "Therefore, trying to form a References: field for a reply that has
   multiple parents is discouraged."

Message 3 is "a reply that has multiple parents".  So, this paragraph
is saying that we discourage putting a References: header in Message
3.  However, whether or not it has a References: header, Message 3
will have an In-Reply-To: header with multiple identifiers.  Message
4, therefore, is a reply to a message whose In-Reply-To: has multiple
identifiers.  This means that the first paragraph does not explain how
to form a References: header for Message 4.  But Message 4 itself is a
reply with exactly one parent, so the "discouraged" paragraph isn't
saying anything about it either.

So, what's up with Message 4's References header?  We haven't
explained how it should be formed, nor have we said that it is
discouraged.  Whether we want to punt on this case or not, we have to
say *something* about it.

If you'd like, I'd be happy to write a complete replacement for these
two paragraphs.  Before I do that, though, I want to be clear on what
exactly we want to say about this last case.

  --  Cos (Ofer Inbar)       --  [email-masked]  [email-masked]
  --  WBRS Waltham 100.1 FM  --  [email-masked]  http://www.wbrs.org/
  "24 hour parking this side even days of month begins at 8:30am.
   No parking other times.  U.P.D."   -- sign in Utica, NY

---

From: Ofer Inbar <[email-masked]>
Date: Mon, 7 Feb 2000 03:23:36 -0500

> that a single space be used each place FWS (whether optional or 

Oh.  Huh.  You're right, I failed to notice that the rules work this
way, to allow it to be stated this simply.  I'd clear up the wording
just a little bit:

Though folding white space is permitted throughout the date-time
specification and is required in certain places, it is RECOMMENDED
that exactly one space be used anywhere FWS appears in a date-time
(whether it is required or optional).  Some older implementations may
not correctly interpret a date-time if this rule is not followed.

> buy-in from the chair and the ADs.

I still think that it was an unintentional error to make FWS optional
between the comma and the day, and based on my mail archive search
nobody does this anyway, so if we can make that change in the syntax
it's probably a good idea.  But, if not, I'm happy with the language
proposed earlier in this message, and it at least makes that FWS a
SHOULD.

  --  Cos (Ofer Inbar)           --  [email-masked]  [email-masked]
  --  Exodus Professional Services           --  [email-masked]
   "We all misuse the net for personal gain, one way or another."
       -- Larry Wall <[email-masked]>

---

From: Ofer Inbar <[email-masked]>
Date: Mon, 7 Feb 2000 03:56:57 -0500

> No, I disagree. 2.2.3 is general description, not syntax rules.

Someone writing code for a mail program is going to wonder, "what were
those rules for header folding again?" and is going to pull up this
document and search for the word folding, and hit on 2.2.3.  They will
see folding fully explained in 2.2.3, and will write code to follow
that rule.  They won't even think about the fact that code written to
follow the description of folding in 2.2.3 might generate a line that
is entirely whitespace, even if at some other point they may have read
the entire document and thus "know" that this is not allowed.  Their
program will, in some weird cases, generate such lines.  And given
that 2.2.3 really does describe header line folding almost completely,
except for the blank line rule... I can't say I'd really blame that
implementor for missing it.

If you want to force people to go to 3.2.3 when they look up the
folding rules, then 2.2.3 needs to be shorter and more vague.  Right
now, it *looks* complete, and that appearance is potentially
misleading.  I think the simplest fix is to actually make it complete,
by adding that one sentence, lifted right from 3.2.3.

> No. You will find obs-in-reply-to in obs-fields in section 4.5, where 

But in that very same section, we have:
id-left         =       dot-atom-text / no-fold-quote / obs-id-left
id-right        =       dot-atom-text / no-fold-literal / obs-id-right

There are various other places in section 3 where the obs-<foo>
nonterminal is listed as an alternative.  Then again, there are many
other places in section three where there is a corresponding obs-<foo>
in section 4, but it is not listed as an alternative.  I only happened
to notice this instance because the two opposite cases appeared in the
same block of production rules and made it obvious.

Emerson's Hobgoblin would be most displeased :)

Really, they should either be everywhere they apply, or nowhere except
in section 4.  If any of the productions in 3 have obs- alternatives
listed, it gives the appearance that the ones that don't have them
listed, don't have them at all.

> IESG to edit; people like to find *something* to correct. :-)

In that case, I suggest you leave "parenthesis" as is in 3.2.3,
and leave the title of Appendix A as "Examples messages". :-)
Certainly this sentence in 3.6.6 is worse than either of those.

  --  Cos (Ofer Inbar)       --  [email-masked]  [email-masked]
  --  WBRS Waltham 100.1 FM  --  [email-masked]  http://www.wbrs.org/
       "Aren't you ever *not* on-line?"   -- Caitlin Xantha Hazen

---

From: Pete Resnick <[email-masked]>
Date: Mon, 7 Feb 2000 03:20:51 -0600

On 2/7/00 at 3:56 AM -0500, Ofer Inbar wrote:

>that rule.

Unless they read section 1.3 at some point in their lives which tells 
them that it would be a big no-no to do this.

I can explain and re-explain every point 10 times in this document. I 
really don't want to. This is one of those places that seems like 
overkill to me.

>Emerson's Hobgoblin would be most displeased :)

Not only would the hobgoblin be pleased, but he would be running 
around with Occam's razor (to mix a few metaphors)!

The in-reply-to production does not appear anywhere in the obs- 
syntax. Therefore, it does not require an obs- alternative defined 
with it; it would be completely redundant to do so. Conversely, 
domain, which appears both in addr-spec in the section 3 rules, and 
in obs-domain-list in the section 4 rules, needs an obs- alternative. 
I tried (and I believe succeeded) in keeping the obs- rules to the 
minimum count necessary for the document.

pr

---

From: Pete Resnick <[email-masked]>
Date: Mon, 7 Feb 2000 03:44:31 -0600

On 2/7/00 at 3:09 AM -0500, Ofer Inbar wrote:

>But saying something is discouraged implies that it is allowed. 
>And, as Eric is pointing out, if it is allowed, but isn't being 
>defined, that needs to be explicit.

I am perfectly willing to add the phrase "and how to do so is not 
defined in this document" to the end of the last sentence if that 
anwers your objections.

>tell, we've said *nothing* at all Message 4's References: header.

Yes we have. Assuming Message 3 has no References field, Message 4's 
References field will contain Message 3's message-id. See the first 
sentence. Perhaps I need to change the third sentence to read:

If the parent has no "References:" field, no "Message-ID:" field, and 
no "In-Reply-To:" field with a single message identifier, then the 
new message will have no "References:" field.

>If you'd like, I'd be happy to write a complete replacement for 
>these two paragraphs.

We are way past the stage for rewriting large chunks here. I still 
believe that minor tweaks can make this crystal clear for the odd 
case of someone who wants to read something weird into this stuff and 
that there is no need for any major change.

pr

---

From: Tony Hansen <[email-masked]>
Date: Mon, 07 Feb 2000 08:23:35 -0500

Or add a note saying something like "For the complete rules, see section
XXX below."

        Tony Hansen
        [email-masked]