Subject: Re: Mandatory From field, anonymity, and hacks
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Date: Mon, 31 Jan 2005 21:35:36 GMT
Newsgroups: local.mime
List-Id: <ietf-822.imc.org>

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Sat January 29 2005 04:23, Hector Santos wrote:

>> Most, if not all SMTP receivers detect valid headers and add what is
>> necessary.

>No, MTAs are not supposed to alter the message content except
>to add trace fields (Received, Return-Path).  MSAs are permitted
>to make specific modifications.

Have you ever tried sending a message with no Message-ID through sendmail?

>> Are you proposing further relaxation as follows?
>> 
>> 822:    Valid RFC header = Date && (cc || bcc || To)
>> 2822:   Valid RFC header = Date && (Sender || Reply-To)

>Yes, ....
> 
>> Technically, a message needs no headers at all.

>No, a message must have a message header, and that message
>header must have at least one header field.  See RFC 822
>section 4.1, RFC 2822 sections 3.5 & 3.6, and RFC 2046
>section 5.2.1.

I see nothing in those places that requires "at least one header field"
except for the provision that certain header fields are not optional, and
hence must be present. But since you seem prepared to reduce the number of
required header fields, then you are in no position to object when Hector
suggests the possibility of reducing them to zero.

And RFC 2046 section 5.2.1 says nothing about headers of messages.

>If that's an SMTP question, yes, because SMTP transfers
>messages, and the message format (822/2822) requires a
>header with certain fields.

False premise, leading to false conclusion.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4411
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Mon, 31 Jan 2005 21:27:04 GMT
Lines: 140
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Fri January 28 2005 11:07, Charles Lindsey wrote:
>> 
>> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
>> 
>> >* E.g. ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lilly-from-optional-00.txt
>> 
>> I have several problems with this.
>> 
>> 1. Is it really necessary to modify RFC 822 and well as RFC 2822?

>Yes.

Why?

>> 2. 
>>    This memo (if approved) updates the Internet Message Format
>>    specification [N1.STD11], [N2.RFC2822] as used by various
>>    applications (electronic mail [I3.STD10], [I1.RFC2821], Usenet news
>>    [I4.RFC1036], Internet fax [I5.RFC2305], VPIM [I6.RFC3801], EDI
>>    [I7.RFC1767], [I8.RFC1865], etc.).  It applies across the board to
>>    applications using the Internet Message Format.  However it does not
>>    discuss similarly named fields in unrelated formats and protocols
>>    such as [I9.HTTP] or [I10.SIP].
>> 
>> I think it is most unwise to attempt to impose this change on
>> protocols/applications other them electronic mail without first consulting
>> the working groups of other bodies responsible for those protocols.

>It imposes no changes on protocols/applications other than those
>that use the Internet Message Format.

As clearly stated in the "Scope" section on RFC 2822, the "Internet
Message Format" applies first and foremost to "electronic mail". Thus, if
you update RFC[2]822, your change is binding on email. But that does not
make it binding on the other protcols you mention, except insofar as they
define particular forms of electronic mail.

Other protocols are defined by their own standards which can, and often do
(and have good reason to so do) incorporate the Internet Message Format by
reference to RFC[2]822, and do so to whatever extent is written into them.

In particular, RFC 1036 (although technically not a standard) incorporates
the Internet Message Format, subject to its own provisions and
interpetations (which, I will grant you, are often less than clearly
expressed). In particular, it is clearly stated that the From header is
Mandatory in News, and that situation will not change even if your draft
is accepted.

By way of a further analogy, it is currently the case that the Subject
header is Mandatory in News according to RFC 1036, even though it is
optional in RFC [2]822, and acceptance of your draft would simply put the
>From header into the same status as the present Subject header.

Likewise, both those headers are Mandatory in the current USEFOR draft,
and there are no plans to make it otherwise. You are of course welcome to
suggest such a change in the USEFOR draft, and you are aware of the proper
channels for doing so.

>That is certainly possible via a number of existing mechanisms
>(From field with an "effective and useful for replies" [RFC 2821
>section 3.8.4] mailbox, a standard Comments or Organization (for
>Usenet specifically) field w/o a  mailbox, or indication in the
>body of the message), or an extension field which is not an
>address field could be defined.

Very droll. I presume those are not intended to be serious suggestions.

I assume you are well aware that News agents regularly provide for
articles to be presented sorted by the poster as indicated by the From
header, and that hiding articles from named posters is the commonest
application for Killfiles. Come to think of it, sorting by From header is
commonly provided in MUAs too.

>> 	Allow the <mailbox> to be omitted when <display-name> is present.
>> 	Allow some form of dummy <mailbox>, such as '<>'.

>No, in the From field that would be incompatible with long-standing
>field syntax specifications, as specifically discussed in the draft
>and as discussed here (specifically including "<>" and syntactically
>legal but semantically empty alternatives) this past Summer.

Yes, that would be a change to the existing syntax. Clearly, existing
agents (and modified ones too) would be unable to send replies to such
>From fields (that is, after all, the purpose of the anonymity). But can you
point to some realistic scenario (or, better, some actual MUA) in which
something might actually break if such a modified header were to arrive at
an agent? After all, all that a user agent expects to do in that situation
is to display and/or print the header correctly.

As a matter of curiosity I tried relaying such messages (telnet to port 25
of my own machine). Sendmail was happy to accept them at face value. It
recognized the display-names in both cases. They appeared intact in my
POP3 box and the MUA I tried to read them with had no problems in
displaying them, including sorting them by From header. Of course, one
successful experiment does not prove very much.

On the other hand, when I tried the same experiment with no From field at
all, it confused sendmail utterly (note that my use of sendmail was for
relaying, and not as a submission agent). Upon finding no from header, it
promptly inserted one "From: Charles Lindsey <[email-masked]>"
which, fortuitously, was correct. But for that to happen to an anonymous
message received from outside would be intolerable.

And one failed epxperiment does, unfortunately, prove a lot.

(My MUA, BTW, was not particularly bothered by the missing From field, and
was happy to sort it before all others.)

>> 	Encourage the use of clearly unresolvable domains, such as those
>> 	ending in '.invalid'.

>Unresolvable domain names present problems for DNS, as discussed here
>in July, where that (".invalid") proposal was called a "showstopper".
>In light of the "SiteFinder" debacle (see the reference in the draft),
>that's a serious matter, since it's not always possible -- on the
>Internet -- to distinguish a non-existent domain; it certainly might
>be impossible off the Internet, such as on the non-Internet side of
>a gateway.

I have email from John Klensin, whom I take to be well versed in these
matters, to the effect that the relevant standards require negative
caching to be properly implemented.

And he also indicated that, if he were ever to revise RFC 3696, he would
consider making a special case for reserved TLDs such as those mentioned
in RFC 2606.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: "ietf-822" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Sat, 29 Jan 2005 23:55:38 -0500
Cc: Hector Santos <[email-masked]>
Sender: [email-masked]

On Sat January 29 2005 04:23, Hector Santos wrote:

> The first question.
> 
> Which software is responsible to make sure the format is correct?  The MUA?
> The SMTP Receiver?

If that's a general question, the answer is a combination of
MUA and MSA (RFC 2476).
 
> Most, if not all SMTP receivers detect valid headers and add what is
> necessary.

No, MTAs are not supposed to alter the message content except
to add trace fields (Received, Return-Path).  MSAs are permitted
to make specific modifications.
 
> Based on RFC 822 (A.3.1.  Minimum required), a valid RFC 822 header is one
> that has:
> 
>     Valid RFC header = Date: && From: && (cc: || bcc: || To:)
> 
> This was relaxed in RFC 2822 , where a valid RFC 2822 header is one that
> has:
> 
>     Valid RFC header = Date: && (From: || Sender: || Reply-To:)

RFC 2822 does not require a Sender field unless there are multiple
authors, and does not require Reply-To under any conditions.

> Are you proposing further relaxation as follows?
> 
> 822:    Valid RFC header = Date && (cc || bcc || To)
> 2822:   Valid RFC header = Date && (Sender || Reply-To)

See above and RFC 2822 section 3.6.

Yes, your error in interpretation of those RFC aside, the intention
is to make the From field optional. [MIME media type message/rfc822
has yet another set of rules, but a From field is not mandatory
there; however if a From field is absent, there must be at least
one of Subject or Date fields.]
 
> From a point of view of Completeness:
> 
> Technically, a message needs no headers at all.

No, a message must have a message header, and that message
header must have at least one header field.  See RFC 822
section 4.1, RFC 2822 sections 3.5 & 3.6, and RFC 2046
section 5.2.1.

> So the next and final question is:
> 
> Is a RFC header required in the DATA stage at all?

If that's an SMTP question, yes, because SMTP transfers
messages, and the message format (822/2822) requires a
header with certain fields.

From: "Hector Santos" <[email-masked]>
To: "ietf-822" <[email-masked]>
Cc: "\"Pete Resnick\"" <[email-masked]>, "\"Bruce Lilly\"" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Sat, 29 Jan 2005 04:23:23 -0500
Organization: Santronics Software, Inc
Sender: [email-masked]

----- Original Message -----
From: "Bruce Lilly" <[email-masked]>
To: "ietf-822" <[email-masked]>
Cc: "Pete Resnick" <[email-masked]>
Sent: Thursday, January 27, 2005 1:57 PM
Subject: Re: Mandatory From field, anonymity, and hacks

> As a result of our discussion starting July 15, 2004, I
> have prepared an Internet Draft; draft-lilly-from-optional-00.txt
> should be available from the usual places [*].  Public comments
> may be posted to the ietf-822 list; private comments to the
> author are also welcome.
>
> * E.g.
ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lilly-from-optional-
00.txt
>

Bruce, I don't wish to be combative here so please bear with me here.

We never depended on the 822.From, but obviously this is going to change a
few things. So I wish approach this by asking what as the expected "design"
behavior for SMTP, GATEWAY software that do more than just move mail, but
also help prepare the expectations of the edge software.

The first question.

Which software is responsible to make sure the format is correct?  The MUA?
The SMTP Receiver?

Most, if not all SMTP receivers detect valid headers and add what is
necessary.

Based on RFC 822 (A.3.1.  Minimum required), a valid RFC 822 header is one
that has:

    Valid RFC header = Date: && From: && (cc: || bcc: || To:)

This was relaxed in RFC 2822 , where a valid RFC 2822 header is one that
has:

    Valid RFC header = Date: && (From: || Sender: || Reply-To:)

Are you proposing further relaxation as follows?

822:    Valid RFC header = Date && (cc || bcc || To)
2822:   Valid RFC header = Date && (Sender || Reply-To)

I guess a comment can be made about valid versus completeness.

>From a point of view of Completeness:

Technically, a message needs no headers at all.

In a 822 model,  a missing Date: is the time of the transport and the To:
can be the 2821.RCPT TO, and any CC: can be the 2nd and additional 2821.RCPT
TO: in the transaction.

In the 2822 model, the missing Date is the time of the transport, and the
Originator can be the 2821.MailFrom.

So the next and final question is:

Is a RFC header required in the DATA stage at all?

This is important because it is always import in the AVS arena.

If a system allows for no RFC headers, than the new message will have based
on your proposal:

Return-Path:
Received:
Date:  <-- Transport Time
Sender:  <-- 2821.MailFrom
To: <-- 2821.RCPTO(1)
cc: <-- 2821.RCPTO(2..n)

>From a point of view of Validity:

A system can view the lack of such information for message
validity/integrity with possible rejection logic behind it.  I personally
don't know of any server who does this, but it is possible.

I guess basically what would really help in this proposal (as well as
others) is what is the expected design for the software when such
information is not available and/or it needs to make an adjustment.   There
is talk of "Should, Must" etc, but what does that mean today?

I understand fully and agree 100% that mail integrity must be maintain, but
where is the borderline here?  What makes the message header valid and if
one or more "required" parts are missing?  What does RFC 2822 and your
proposal say should be done?

Start with the obvious. No headers provided.

I bring this up because it is more than just the receiver that we are asking
the vendors of software to do.  We are asking the creator of mail to do
more, to be more consistent, be it for mail authentication etc,  we need to
look at how one system is expected to behave versus how we can use this
information for verification just as well.

Thanks

---
Hector Santos
WINSERVER "Wildcat! Interactive Net Server"
http://www.santronics.com

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Sat, 29 Jan 2005 07:56:35 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 17
Sender: [email-masked]

Bruce Lilly wrote:

> the only way to determine if a hypothetical domain name is
> valid is to query DNS.
..^^^^^
It's the very idea of "TLD" .invalid to be invalid even without
querying DNS.  Of course you're free to do what you like in
your LAN.  But if you need a TLD and a guarantee, that it will
never clash with one of ICANN's TLDs, then .test, .example, and
.localhost should be good enough for this purpose.  Better stay
away from "TLD" .example unless you use it in an example.

                         Bye, Frank
-- 
<http://purl.net/net/msgid/[email-masked]>
<http://purl.net/net/msgid/[email-masked]>

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Fri, 28 Jan 2005 17:44:42 -0500
Cc: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

On Fri January 28 2005 11:07, Charles Lindsey wrote:
> 
> In <[email-masked]> Bruce Lilly <[email-masked]> writes:
> 
> >As a result of our discussion starting July 15, 2004, I
> >have prepared an Internet Draft; draft-lilly-from-optional-00.txt
> >should be available from the usual places [*].  Public comments
> >may be posted to the ietf-822 list; private comments to the
> >author are also welcome.
> 
> >* E.g. ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lilly-from-optional-00.txt
> 
> I have several problems with this.
> 
> 1. Is it really necessary to modify RFC 822 and well as RFC 2822?

Yes.

> 2. 
>    This memo (if approved) updates the Internet Message Format
>    specification [N1.STD11], [N2.RFC2822] as used by various
>    applications (electronic mail [I3.STD10], [I1.RFC2821], Usenet news
>    [I4.RFC1036], Internet fax [I5.RFC2305], VPIM [I6.RFC3801], EDI
>    [I7.RFC1767], [I8.RFC1865], etc.).  It applies across the board to
>    applications using the Internet Message Format.  However it does not
>    discuss similarly named fields in unrelated formats and protocols
>    such as [I9.HTTP] or [I10.SIP].
> 
> I think it is most unwise to attempt to impose this change on
> protocols/applications other them electronic mail without first consulting
> the working groups of other bodies responsible for those protocols.

It imposes no changes on protocols/applications other than those
that use the Internet Message Format. The quoted text above clearly
and specifically says so.  As for WGs and "other bodies", there
is certainly opportunity for comment now, as well as at the
time of a Last Call prior to submission for publication.

> In the case of Netnews, in particular, I would regard this proposal as
> totally unacceptable, since it is clearly desirable and widely expected
> that it will be possible to identify the (claimed) poster of any Usenet
> article, even if only by the pseudonym that poster chooses to be known by.

That is certainly possible via a number of existing mechanisms
(From field with an "effective and useful for replies" [RFC 2821
section 3.8.4] mailbox, a standard Comments or Organization (for
Usenet specifically) field w/o a  mailbox, or indication in the
body of the message), or an extension field which is not an
address field could be defined.  Making the From field optional
for the purpose of permitting anonymity and to appropriately
handle the case of an author with no Internet mailbox any of
those mechanisms which do not require an effective and useful
Internet mailbox.

> I would sugges that, for Usenet, at least the display name should be
> provided when no email address is available. That would leave the
> following possibilities:
> 
> 	Allow the <mailbox> to be omitted when <display-name> is present.
> 	Allow some form of dummy <mailbox>, such as '<>'.

No, in the From field that would be incompatible with long-standing
field syntax specifications, as specifically discussed in the draft
and as discussed here (specifically including "<>" and syntactically
legal but semantically empty alternatives) this past Summer.

> 	Encourage the use of clearly unresolvable domains, such as those
> 	ending in '.invalid'.

Unresolvable domain names present problems for DNS, as discussed here
in July, where that (".invalid") proposal was called a "showstopper".
In light of the "SiteFinder" debacle (see the reference in the draft),
that's a serious matter, since it's not always possible -- on the
Internet -- to distinguish a non-existent domain; it certainly might
be impossible off the Internet, such as on the non-Internet side of
a gateway.

> It is, in
> any case, not clear that some existing user agents will not barf at the
> proposed total absence of the From header.

Well, they certainly won't be able to compose responses to the
author, but (as discussed in the draft) that is not something
that one could do for the case of an anonymous author or an
author with no Internet mailbox, which are the cases covered
by the draft.

> 4. 
>    Some documents have suggested use of the reserved ".invalid" TLD
>    (top-level domain name) [I18.BCP32] to provide some degree of
>    anonymity.  With relaxation of the requirement for a From field in
>    the Internet Message Format, such hacks and their negative impact on
>    the root name service are unnecessary, at least within the scope of
>    Internet Messages.
> 
> That reference to "hacks" and "negative impact" is hardly fair.

It is based on the discussions here in July.  At the moment,
the two RFCs that do so are SIP documents, which is specifically
outside of the scope of the draft; however making the From field
optional would obviate any tendency to repeat such a mistake in
the messaging area.

> I have 
> been assured by people who understand the DNS system better than I do
> that it is a common and recommended practice for DNS failures to be
> cached,

If it were a common practice, there wouldn't be an RFC (2308)
lamenting the lack of such optional caching of negative responses
in practice.  Moreover, if that practice were in fact widespread
on the Internet, it would not help in the case of inappropriate
positive responses (as in the SiteFinder debacle), nor would it
help systems not directly connected to the Internet (as in
offline message composition, and non-Internet and/or non-mail
systems which make use of a gateway for sending to Internet
mailboxes). 

> Moreover, '.invalid' can, and should be,
> built into agents so that they do not waste time even trying.

Absolutely not; RFC 3696 specifically discusses such nonsense --
the only way to determine if a hypothetical domain name is
valid is to query DNS.  Building special hardwired cases into
code has caused, and can be expected to continue to cause,
problems.

To: [email-masked]
Xref: clerew local.mime:4406
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Date: Fri, 28 Jan 2005 16:07:37 GMT
Lines: 80
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>As a result of our discussion starting July 15, 2004, I
>have prepared an Internet Draft; draft-lilly-from-optional-00.txt
>should be available from the usual places [*].  Public comments
>may be posted to the ietf-822 list; private comments to the
>author are also welcome.

>* E.g. ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lilly-from-optional-00.txt

I have several problems with this.

1. Is it really necessary to modify RFC 822 and well as RFC 2822?

2. 
   This memo (if approved) updates the Internet Message Format
   specification [N1.STD11], [N2.RFC2822] as used by various
   applications (electronic mail [I3.STD10], [I1.RFC2821], Usenet news
   [I4.RFC1036], Internet fax [I5.RFC2305], VPIM [I6.RFC3801], EDI
   [I7.RFC1767], [I8.RFC1865], etc.).  It applies across the board to
   applications using the Internet Message Format.  However it does not
   discuss similarly named fields in unrelated formats and protocols
   such as [I9.HTTP] or [I10.SIP].

I think it is most unwise to attempt to impose this change on
protocols/applications other them electronic mail without first consulting
the working groups of other bodies responsible for those protocols.

In the case of Netnews, in particular, I would regard this proposal as
totally unacceptable, since it is clearly desirable and widely expected
that it will be possible to identify the (claimed) poster of any Usenet
article, even if only by the pseudonym that poster chooses to be known by.

If you think otherwise, then I invite you to raise this matter on the
USEFOR list.

I would sugges that, for Usenet, at least the display name should be
provided when no email address is available. That would leave the
following possibilities:

	Allow the <mailbox> to be omitted when <display-name> is present.
	Allow some form of dummy <mailbox>, such as '<>'.
	Encourage the use of clearly unresolvable domains, such as those
	ending in '.invalid'.

Since the first two possibilities might provide some compatibility
problems for existing user agents (though I rather doubt this), perhaps
the use of .invalid could be suggested as an interim measure. It is, in
any case, not clear that some existing user agents will not barf at the
proposed total absence of the From header.

3. I would have thought that the absence of a From header would have been
unacceptable to most users in Email as well as in Netnews, but that is a
matter which is well within the purview of this List to discuss.

4. 
   Some documents have suggested use of the reserved ".invalid" TLD
   (top-level domain name) [I18.BCP32] to provide some degree of
   anonymity.  With relaxation of the requirement for a From field in
   the Internet Message Format, such hacks and their negative impact on
   the root name service are unnecessary, at least within the scope of
   Internet Messages.

That reference to "hacks" and "negative impact" is hardly fair. I have
been assured by people who understand the DNS system better than I do
that it is a common and recommended practice for DNS failures to be
cached, and that the inpact of using the TLD .invalid on the root servers
should therefore be minimal. Moreover, '.invalid' can, and should be,
built into agents so that they do not waste time even trying.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Mandatory From field, anonymity, and hacks
Cc: Pete Resnick <[email-masked]>
Date: Thu, 27 Jan 2005 13:57:33 -0500
Sender: [email-masked]

As a result of our discussion starting July 15, 2004, I
have prepared an Internet Draft; draft-lilly-from-optional-00.txt
should be available from the usual places [*].  Public comments
may be posted to the ietf-822 list; private comments to the
author are also welcome.

* E.g. ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-lilly-from-optional-00.txt

To: [email-masked]
Xref: clerew local.mime:4404
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #3 with multiple Reference headers
Date: Wed, 26 Jan 2005 10:32:45 GMT
Lines: 26
Sender: [email-masked]

In <tFXxIsHlJN/+6084uu/[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>Charles Lindsey writes:

>> But all the reports so far (not a representtive sample, to be sure) 
>> indicate that no harm arises from multiple References headers (i.e. 
>> they progagate, and are interpreted as well as could be hoped for).

>(The software on your own machine is an exception, in that it didn't 
>propagate the field.)

No, that is not correct. It failed to let it out as a newly generated
message. It is, and was, perfectly happy to receive such messages arriving
from outside, as it did when my 2nd and 3rd attempts came back from the
List.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 25 Jan 2005 14:53:32 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Experiment #3 with multiple Reference headers
Sender: [email-masked]

Charles Lindsey writes:
> Yes, if having two References headers turns out not to be feasible, 
> then having one References header plus some number of secondary 
> References headers with some different name (I don't think I would 
> want to use See-Also for the purpose) would be another way out.

Sounds like RFC 2822 to me. IIRC 2822 says something like this:

1. Generate a References field using the usual algorithm.
2. If the new message is a reply to multiple messages, copy their 
message-ids to the In-Reply-To field.

> But all the reports so far (not a representtive sample, to be sure) 
> indicate that no harm arises from multiple References headers (i.e. 
> they progagate, and are interpreted as well as could be hoped for).

(The software on your own machine is an exception, in that it didn't 
propagate the field.)

Arnt

To: [email-masked]
Xref: clerew local.mime:4402
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: General considerations for new message field specifications
Date: Tue, 25 Jan 2005 12:36:01 GMT
Lines: 54
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <blilly@xxxxxxxxx> writes:

>Do you believe that the recommendations in draft section
>4.2.1 do not adequately address whitespace, line folding,
>and comments (which are the principal differences in BNF
>between 822 and 2822)?

I think it might be expanded a little, perhaps by giving a practical
example (for which the case of new parameters for Content-Type headers
would be well suited).

>As for addressing both EBNF and ABNF, I believe that's
>not particularly difficult to do, as in RFC 3282 for
>example.

I don't think the notational differences between EBNF and ABNF cause any
particular problems. They are similar enought that one can easily be
translated into the other.

>> >> 3. Make it clear when the value part of any parameters that are defined
>> >> need to be quoted.
>> 
>> >That's covered by RFCs 2045 and 2231 for the fields where it
>> >applies.
>> 
>> Yes, but you are writing an informational document for people who may be
>> inventing new headers. They need to be warned of things they might easily
>> overlook.

>It hardly seems worthwhile expanding the draft to cover a
>little-used (ca. 3% of fields) feature.

Who are you to say whether that 3% will or will not increase in the
future? The fact remains that it is rather easy to miss stuff out when
introducing new parameters. Perhaps when Ned gets around to writing
RFC2045bis he will be able to cover parameters in a more general way so
that it will be easier to incorpate them into other headers in future.

>...  I certainly don't intend to turn the
>draft into a hundred page behemoth that will put off potential
>readers.

I am glad to hear it.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4401
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #3 with multiple Reference headers
Date: Tue, 25 Jan 2005 12:25:26 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Frank Ellermann <nobody@xxxxxxxxxxxxxxxxx> writes:

>If I'd really want to mix the ideas of See-Also (= literal
>sense of references) and References, then I'd probably identify
>one thread as _the_ thread, copy its References adding a msg-id
>as explained in 2822, and then I'd add the Message-IDs of any
>further "see also" references.  Maybe with a limit of at most
>14 additional references, because that still allows 3+3 msg-ids
>for _the_ thread.  OTOH that sounds like an idea for a time
>travel to 1995, too little and too late today. ;-)  Bye, Frank

Yes, if having two References headers turns out not to be feasible, then
having one References header plus some number of secondary References
headers with some different name (I don't think I would want to use
See-Also for the purpose) would be another way out.

But all the reports so far (not a representtive sample, to be sure)
indicate that no harm arises from multiple References headers (i.e. they
progagate, and are interpreted as well as could be hoped for).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: General considerations for new message field specifications
Date: Fri, 21 Jan 2005 10:17:11 -0500
Cc: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

On Mon January 17 2005 11:33, Charles Lindsey wrote:

> Yes, but if a document makes normative reference to earlier standards,
> some written in RFC 822 style and some in RFC 2822 style, then life get
> complicated. I think you could usefully give more guidance on how to cope
> with this, or at least warn them that the problem has to be addressed
> somehow.

Do you believe that the recommendations in draft section
4.2.1 do not adequately address whitespace, line folding,
and comments (which are the principal differences in BNF
between 822 and 2822)?

As for addressing both EBNF and ABNF, I believe that's
not particularly difficult to do, as in RFC 3282 for
example.  The only significant reason to use RFC 822
EBNF for new general extension fields is for drafts that
are expected to progress along the Standards Track faster
than 2822, and in light of RFC 3967, that is arguably not
as great a factor as it is sometimes taken to be.
 
> >There are only three fields out of about 140 that use those
> >so-called "parameters". They are the MIME fields Content-Type
> >and Content-Distribution, and Keith's Auto-Submitted field
> >(which is probably partially my fault, as otherwise it would
> >have introduce a third syntax for parameters (the second being
> >that used by the Disposition-Notification-Options field).
> 
> >> 3. Make it clear when the value part of any parameters that are defined
> >> need to be quoted.
> 
> >That's covered by RFCs 2045 and 2231 for the fields where it
> >applies.
> 
> Yes, but you are writing an informational document for people who may be
> inventing new headers. They need to be warned of things they might easily
> overlook.

It hardly seems worthwhile expanding the draft to cover a
little-used (ca. 3% of fields) feature. It is, however
addressed by the general text of section 4.2.1, which recommends
careful review of ABNF.  I certainly don't intend to turn the
draft into a hundred page behemoth that will put off potential
readers.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: Experiment #3 with multiple Reference headers
Date: Wed, 19 Jan 2005 13:39:43 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 21
Sender: [email-masked]

Charles Lindsey wrote:

> 1. Did it arrive with all of them intact?

Probably, 2nd and 3rd in the form of Original-References on
GMaNe, <http://article.gmane.org/gmane.ietf.rfc822:11263:raw>

> the first (can be recognized by presence of
> <[email-masked]>)?

Yes, that's the last msg-id in the 1st References: header.

If I'd really want to mix the ideas of See-Also (= literal
sense of references) and References, then I'd probably identify
one thread as _the_ thread, copy its References adding a msg-id
as explained in 2822, and then I'd add the Message-IDs of any
further "see also" references.  Maybe with a limit of at most
14 additional references, because that still allows 3+3 msg-ids
for _the_ thread.  OTOH that sounds like an idea for a time
travel to 1995, too little and too late today. ;-)  Bye, Frank

Date: Wed, 19 Jan 2005 09:27:50 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Laird Breyer <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Wed, 19 Jan 2005, Laird Breyer wrote:

> Either way, I believe that Bruce's point about decades of consistent
> prior use in RFCs trumps all personal preferences. 

In natural language, of course, "decades of consistent prior use" stands 
for nothing. New meanings wipe away old ones almost overnight. The word 
"gay" is a good example. But that's irrelevant. RFCs are not, in that 
sense, "natural language". More like legal documents, I guess.

Anyway, when I raised this topic, I wasn't trying to push a personal
preference. I'm quite happy to use the formal RFC terminology. However,
many people don't, and I'm not sure that "header field" is widely
understood by folk who are not versed in the content of the RFCs.

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Wed, 19 Jan 2005 09:15:40 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: ietf-822 <[email-masked]>
Subject: Re: Resent- fields
Sender: [email-masked]

On Mon, 17 Jan 2005, Bruce Lilly wrote:

> No, "header" in the mail standards has a very clear meaning:

I entirely agree. However, I observe that there is very widespread 
misuse of this terminology, leading to possible confusion.

> Your proposal, which is to replace precise,
> well-defined terms with poorly-defined colloquialisms, would
> have a negative effect on interoperability and on general
> understanding of messages and messaging protocols.

It was a point for discussion, not a formal proposal, and as such it 
succeeded. :-) If, however, I had been making a formal proposal, it 
would have been to replace precise, well-defined but frequently 
misunderstood terms with new, equally precise, equally well-defined, but 
less likely to be misused or misunderstood terms. However, I didn't 
actually expect anything to happen other than there to be a discussion, 
which at least raises the point in people's minds.

Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Wed, 19 Jan 2005 10:31:45 +1000
From: Laird Breyer <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked]
Subject: Re: terminology
Reply-To: [email-masked]
Mail-Followup-To: ietf-822 <[email-masked]>, [email-masked]
Sender: [email-masked]

On Jan 18 2005, Bruce Lilly wrote:
> On Mon January 17 2005 18:58, Laird Breyer wrote:
> 
> > If people start using header lines in this sense, I think there's the
> > inbuilt potential for confusion in some types of discussion.
> 
> It has been used precisely that way for quite a long time; in
> this case Charles is correct.

Being used that way previously does not necessitate future usage, if
the potential for confusion is felt to be big enough. There's always
the option of deprecating the usage of "header lines", as opposed to
actually changing its meaning.

That said, I doubt that "header line" in the sense of a single text
line is a sufficiently common topic of discussion to warrant its
excessive regulation.

>  
> > To me, it's more natural to associate "header line" with the fully unfolded
> > piece of the header which begins with a field name and colon, since such
> > a "header line" is invariant under mail transport (as much as possible).
> 
> No, that's a "header field".

That's fine by me.

-- 
Laird Breyer.

Date: Wed, 19 Jan 2005 10:17:32 +1000
From: Laird Breyer <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Reply-To: [email-masked]
Mail-Followup-To: Charles Lindsey <[email-masked]>, [email-masked]
Sender: [email-masked]

On Jan 18 2005, Charles Lindsey wrote:
> 
> Yes, but what term do you use when you want to say "This *mumble* is
> longer than 998 characters", or "No *mumble* is allowed to consist of
> whitespace only"? I will grant you that perhaps "header line" is not the
> right term, because it is frequently used as a synonym for "header field"
> (aka, in common parlance, "header").

Right, if it was me, I'd probably say "this message line" or "this
text line", both of which are purposely scoped so widely that they
don't evoke a single field. But I'd have to grep my archives for the
string "line" to be sure ;-)

Either way, I believe that Bruce's point about decades of consistent
prior use in RFCs trumps all personal preferences. 

-- 
Laird Breyer.

Date: Tue, 18 Jan 2005 12:11:05 -0500
From: Keith Moore <[email-masked]>
To: ietf-822 <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: terminology
Sender: [email-masked]

I think we should just stick with "header" consisting of one or more
"header fields" where each "field" can span one or more "lines".  
There's been some effort to maintain that consistency of terminology in
the past (to the extent that drafts often got feedback of the form "you're
not using the words in the standard way", and were often changed to 
reflect the above terminology).  

And yes, a few RFCs have fallen through the cracks and failed to adhere
to the standard terminology.  But we'd have the same problem if we moved
to different terminology.  I really can't see the justification for 
doing so as long as the existing terminology suffices to describe these
protocol elements.  Choosing new terms won't do anything to prevent 
misuse of those new terms.  And choosing new meanings for old terms 
will only create more confusion.

And we should also just accept that "header fields" are going to be 
abbreviated to "headers" once in awhile, and we're going to have to
sort out the differences by context.   (At least in English and in
several other languages, such disambiguation is quite normal and we
don't even think about it as long as we can get the right meaning
most of the time.)

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>, [email-masked]
Subject: terminology
Date: Tue, 18 Jan 2005 11:35:57 -0500
Cc: [email-masked]
Sender: [email-masked]

On Mon January 17 2005 18:58, Laird Breyer wrote:
> 
> On Jan 18 2005, Charles Lindsey wrote:
> 
> > In <[email-masked]> Keith Moore <[email-masked]> writes:
> > 
> > >OTOH, "header line" seems ambiguous to me - I think you must mean 
> > >something _besides_ header field because you're not using either 
> > >"header field" or the common abbreviation "header".  :)
> > 
> > I think we need the term "header line" when speaking of the individual
> > lines of a header field that has been folded.
> > 
> 
> A "header line" as you propose is not a semantic unit, since folding
> white space can break the information at all sorts of places, and as
> the mail winds its way to its destination, folded lines can be
> rearranged several times in semantically equivalent ways.

Correct.

> If people start using header lines in this sense, I think there's the
> inbuilt potential for confusion in some types of discussion.

It has been used precisely that way for quite a long time; in
this case Charles is correct.
 
> To me, it's more natural to associate "header line" with the fully unfolded
> piece of the header which begins with a field name and colon, since such
> a "header line" is invariant under mail transport (as much as possible).

No, that's a "header field".

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: non-standard prefixing considered harmful, unnecessary fiddling with message content considered harmful
Date: Mon, 17 Jan 2005 21:22:26 -0500
Cc: "william(at)elan.net" <[email-masked]>
Sender: [email-masked]

On Sun January 16 2005 18:45, william(at)elan.net wrote:

> At the same time for majority of other headers (except Resent and Received)
> adding additional header line with same name is not allowed.

That is not quite correct; the maximum number of instances of
a particular header field in a message or MIME-part header
varies according to the field. RFC 2822 contains a detailed
table for the basic fields; several RFCs introducing extension
fields likewise indicate the maximum number of instances per
header (some however, fail to do so, and that is an ambiguity
in the specification).

> The common  
> practice has been to simply replace original header line with new one.

The accepted practice is that an SMTP server "receives mail
from an SMTP client and transmits it, without modification to
the message data other than adding trace information" (RFC 2821
section 2.3.8).  N.B. "without modification".  Also STD 3 (Host
Requirements) section 5.2.6: "An Internet host that is
forwarding a message but is not a gateway to a different mail
environment (i.e., it falls under (1) or (2)) SHOULD NOT alter
any existing header fields, although the host will add an
appropriate Received: line as required in Section 5.2.8.".

Altering message header fields during transport invariably
leads to problems; the classic example is the originator-
supplied Reply-To field, which is sometimes inappropriately
altered by mailing list expanders, causing a variety of
problems.  Such a reprehensible practice is the equivalent
of a postal service opening letters and altering the
content. 

> But 
> its really good for debugging of email routing problems to have seen what
> the value has been

It's never supposed to be altered (with specific exceptions
for MSAs under specific conditions, and for gateways)!  And
specific standards for specific types of gateways (e.g. RFC 2156)
may make provision for retaining content in a manner which
preserves it for debugging.

If the problem that you're trying to solve is that some non-
gateway systems are irresponsibly altering message content,
than an appropriate solution is to convince the (ir)responsible
parties not to make alterations.  Please don't screw things
up for everybody by encouraging such unnecessary modification
of message content!

> [...] Original- header lines are more like email trace header fields (and 
> similarly multiple of same named ones can be in the message) and have same 
> purpose to help track changes to email message, they should not be viewed 
> as replacement for real headers or used directly by MUA.

Gack, NO! There are already well-defined
Original-Encoded-Information-Types, Original-Envelope-ID,
Original-Message-ID, and Original-Recipient fields. Mindlessly
prefixing "Original-" to fields will result in the inability
to distinguish an Original-Message-ID field from a Message-ID
field that has been so altered.

> Now as has been mentioned, I plan to document these Original- headers in the
> Internet draft so if you like to provide some feedback

Please don't.  It's the worst idea I've heard in a long time.

> Accidentally I can't find any good way to register with IANA a group of headers
> like Original- (as common prefix) as RFC3864 specifies only how to do it for
> one particular header field. Please suggest what to do about it.

You can't find it because field names don't work that way. There
simply are no generic prefixes.  Each field is separately
defined (see, for example, RFC 2822's definitions for Resent-To,
Resent-From, Resent-Cc, Resent-Date, Resent-Message-ID, Resent-Sender,
Resent-Bcc, and the deprecated Resent-Reply-To fields (N.B. there are
no other Resent- fields!)).

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Resent- fields
Date: Mon, 17 Jan 2005 20:32:32 -0500
Cc: Philip Hazel <[email-masked]>
Sender: [email-masked]

On Sun January 16 2005 16:57, Philip Hazel wrote:

> it is impossible to distinguish
> between different sets of "resent-" header lines. For example, if you
> receive
> 
> resent-from: X
> resent-to: Y
> resent-date: Z
> resent-cc: Q
> resent-from: A
> resent-to: B
> 
> to which set do "date:" and "cc:" belong?

*If* the header fields have erroneously (i.e. in violation of
RFC 2822 section 3.6) been jumbled, one cannot tell. Under
normal conditions it is simple: the initial recipient will
have a Return-Path field, followed by one or more Received
fields, followed by other fields.  When resending, the
Resent- fields are prepended in a block (RFC 2822 section 3.6,
3.6.6, 3.6.7).  Likewise if that message is received and
subsequently resent.  I.e. each block of Resent- fields will
be separated from the others by one or more trace (Received
and Return-Path) fields.  Moreover, each block of Resent-
fields will include its own Resent-Date field (it is mandatory)
so to answer your question, one would simply group the fields
bottom-up (your example is malformed as it does not contain
as many Resent-Date fields as Resent-To and Resent-From
fields).

> Incidentally, one of these days we should perhaps bring the 
> specification's terminology in line with what everybody actually uses. 
> Pretty well everybody, when they write "header" mean a single header 
> line (as you seem to above).

No, "header" in the mail standards has a very clear meaning:
is is a group of fields separated from body content by an
empty line.  The same term has the same meaning in HTTP and
SIP protocols, though each of those protocols' transactions
begins with a "start line" which has no equivalent in the
mail protocols.  In a message, one has the message header
and may have zero or more MIME-part headers (some of the
MIME-part headers might contain no fields).  The term
"header" was first defined that way in RFC 561 more than
three decades ago and has been used that way consistently
in its successors through RFC 2822.  It is directly analogous
to the use of the term "header" in IP, TCP, etc. packets;
people who are not clueless regarding packet communications
do not refer to individual IP header (bit) fields as "headers".
We should expect no less of technical communication regarding
messaging protocols and formats.

> The RFC actually calls this a "header  
> field":

The term "field" was formalized in RFC 724, if not earlier
(RFC 680 is not online), more than a quarter century ago,
and has also been used consistently (as has "field body")
by its successors.

> I have been trying to 
> remember to use "header line" consistently, but the colloquial use of
> "header" is so prevalent that I think it's a losing battle

It is important to distinguish colloquial use from precise
terminology used in professional technical communication.
The primary purpose of technical standards is to ensure
interoperability, and that is best achieved by using
precise, well-defined terms. That is why we have RFC 2119
for example, which provides precise definitions of several
commonly used terms in IETF documents.

> I propose that next time there's a big revision, we should go with the 
> current actual usage, and find a new name for the collection of
> "headers".

The term for the collection of (message and MIME-part) headers
is in fact "headers".  The term for the collection of (message
and MIME-part) headers in a collection of messages is "headers".
The definitions have been in use consistently for so long that
an attempt to redefine them would be disastrous.  It would be
equivalent to saying something like "when I say 'white', I don't
really mean white, I mean black, and when I say 'black', I don't
really mean black, I mean 'red', and when I say 'red' I don't
really mean red, I mean white.  Such circumlocutions would only
cause confusion.  Instead we should continue to use the well-
defined terms "field body", "field", "header", and "headers" as
they have been used for decades, encourage the correct use of
those terms in technical communications, and actively discourage
incorrect use in technical communications.  Some education of
semi-technical people in the correct definitions and usage would
reduce confusion and may have a peripheral positive effect on
interoperability.  Your proposal, which is to replace precise,
well-defined terms with poorly-defined colloquialisms, would
have a negative effect on interoperability and on general
understanding of messages and messaging protocols.

Date: Tue, 18 Jan 2005 13:33:47 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Charles Lindsey <[email-masked]>
cc: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Tue, 18 Jan 2005, Charles Lindsey wrote:

> Yes, but what term do you use when you want to say "This *mumble* is
> longer than 998 characters",

"physical line" ?

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

To: [email-masked]
Newsgroups: local.mime
Path: chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: General considerations for new message field specifications
Date: Mon, 17 Jan 2005 16:33:22 GMT
Lines: 68
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Fri January 14 2005 11:32, Charles Lindsey wrote:

>> The most prominent of these would be to mention Graham Klyne's RFC 3864,
>> which introduces the IANA header registry

>I have never heard of "Content-Distribution". Do you have a reference?

Sorry. Content-Disposition.

>> The problems all such proposal have in common is the necessity to:
>> 
>> 1. Make it clear where [CFWS] can be inserted (currently, it is hard work
>> to deduce this from RFC 2045 cum RFC 822).

>Yes, that is an issue with RFC 822, which affects several RFCs
>that use 822 syntax, and is addressed to some extent by RFC 2822
>(and will hopefully be improved in 2822's successor). It is addressed
>by the draft.

Yes, but if a document makes normative reference to earlier standards,
some written in RFC 822 style and some in RFC 2822 style, then life get
complicated. I think you could usefully give more guidance on how to cope
with this, or at least warn them that the problem has to be addressed
somehow.

>> 2. Make it clear that more parameters will appear in future extensions,
>> which MUST NOT be generated yet, but MUST be accepted and ignored now.
>> Except that paramaters with x-tokens MAY be generated now.

>There are only three fields out of about 140 that use those
>so-called "parameters". They are the MIME fields Content-Type
>and Content-Distribution, and Keith's Auto-Submitted field
>(which is probably partially my fault, as otherwise it would
>have introduce a third syntax for parameters (the second being
>that used by the Disposition-Notification-Options field).

>> 3. Make it clear when the value part of any parameters that are defined
>> need to be quoted.

>That's covered by RFCs 2045 and 2231 for the fields where it
>applies.

Yes, but you are writing an informational document for people who may be
inventing new headers. They need to be warned of things they might easily
overlook.

>> RFC 3834 makes a reasonable job of all these things, except that it avoids
>> #3 by not actually defining any parameters :-) . But the OpenPGP draft is
>> a bot weak, hence the need for you to cover thses issues.

>Specific concerns w.r.t. a specific draft are probably best addressed
>to the author(s) of that draft at this time. 

Yes, I have done, and had a nice acknowledgement too.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Newsgroups: local.mime
Path: chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Mon, 17 Jan 2005 17:09:40 GMT
Lines: 32
Sender: [email-masked]

In <[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>Charles Lindsey writes:
>> Well, as you know, the first attempt failed because of excessing 
>> checking before it left my machine.

>It didn't fail - it just showed that a transit agent will "clean" the 
>message in ways that would modify the meaning of the message, should 
>multiple references fields be allowed.

>Which transit agent was it, if I may ask?

It wasn't a transit agent. I read this list in a local newsgroup, and so
it was the injection agent of Henry's CNews that (quite justifiably)
pruned it.

But that is no problem because, if this (or any similar) proposal were
ever to come to fruition, agents than generate or inject such messages
would need to be upgraded. What we require is that existing transit or
reading agents should not do anything harmful or worse than happens at
present.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4386
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Tue, 18 Jan 2005 11:08:23 GMT
Lines: 49
Sender: [email-masked]

In <[email-masked]> "william(at)elan.net" <[email-masked]> writes:

>On Mon, 17 Jan 2005, Charles Lindsey wrote:

>> I think that registry is intended for fields that agents might be expected
>> to recognize and act upon. I doubt any MUA is going to have a
>> built-in-recognition of any and all fields beginning with "Original-".
>> Surely they are intended for human consumption, in which case using
>> "X-Original-xxxxxx" would he just as good, and would break no rules or
>> regisries?

>Aren't trace fields like "Received:" intended for human consumption too? :)

ALL headers need to be readable by humans upon occasions. But some are not
really suitable for reading by other than humans.

>And in this case, while its primarily trace data for humans for debugging 
>it could potentially be used by MTAs, for example to detect complex mail 
>loop. Besides that some systems might choose to display "Original-Sender"
>or "Original-From" in MUA as well (on part with Sender, From, Resent-From,
>Resent-Sender when showing the entities involved in email transmission).

Well any header that is intended to be used by MTAs needs to be
standardized in its own right. What I am doubting is whether any MTA is
ever going to be programmed to take some generic action on Original-Foo:
headers for all possible values of "Foo".

>And its not exactly that it would "break" the registry rule, its just
>that currently there is no way to do it and I think that means we'd
>have to expand IANA header registry to allow to define certain set of
>headers with common prefix or to specifically reserve "prefix" for
>use with one type of headers (which are afterward specifically defined).
>That generally means either separate draft for IANA or expansion of 
>RFC3864 (but I really would prefer header prefix registry by on same page 
>at IANA as specific header names registry).

Well if you want to embark upon an enhancement to RFC3834, then I would
suggest some preliminary discussion with Graham Klyne.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4387
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Tue, 18 Jan 2005 11:35:31 GMT
Lines: 29
Sender: [email-masked]

In <20050117235827.GA27714@ender> Laird Breyer <[email-masked]> writes:

>On Jan 18 2005, Charles Lindsey wrote:

>> I think we need the term "header line" when speaking of the individual
>> lines of a header field that has been folded.
>> 

>A "header line" as you propose is not a semantic unit, since folding
>white space can break the information at all sorts of places, and as
>the mail winds its way to its destination, folded lines can be
>rearranged several times in semantically equivalent ways.

Yes, but what term do you use when you want to say "This *mumble* is
longer than 998 characters", or "No *mumble* is allowed to consist of
whitespace only"? I will grant you that perhaps "header line" is not the
right term, because it is frequently used as a synonym for "header field"
(aka, in common parlance, "header").

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 18 Jan 2005 09:22:04 +0000
To: [email-masked]
From: Paul Overell <[email-masked]>
Subject: Re: Experiment #3 with multiple Reference headers
Sender: [email-masked]

In message <[email-masked]>, Charles Lindsey 
<[email-masked]> writes
>
>This message responds to several others, and contains a References header for
>each of them. So it cannot be expected that existing MUAs will thread them
>particularly well. So the things to look out for are:
>

MUA Turnpike Six v6.05

>1. Did it arrive with all of them intact?

Yes

>2. Did is arrive with at least one of them intact? If so, was it the first
>(can be recognized by presence of <[email-masked]>)?

Yes

>3. Did it do anything really nasty with the threading?

No, threaded using first References: line

>4. Did it even manage to make some use of the extra headers?

No, ignored them.

Regards
-- 
Paul Overell         Internet Platform Development Manager, Thus plc

Date: Tue, 18 Jan 2005 09:18:45 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Charles Lindsey <[email-masked]>, Laird Breyer <[email-masked]>, "william(at)elan.net" <[email-masked]>
cc: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Mon, 17 Jan 2005, Charles Lindsey wrote:

> I think we need the term "header line" when speaking of the individual
> lines of a header field that has been folded.

With respect, I think that would be a really bad idea, because 
non-experts will never interpret it that way. And MUAs often re-fold 
header lines anyway.

On Tue, 18 Jan 2005, Laird Breyer wrote:

> A "header line" as you propose is not a semantic unit, since folding
> white space can break the information at all sorts of places, and as
> the mail winds its way to its destination, folded lines can be
> rearranged several times in semantically equivalent ways.

Absolutely!

> To me, it's more natural to associate "header line" with the fully unfolded
> piece of the header which begins with a field name and colon, since such
> a "header line" is invariant under mail transport (as much as possible).

That's how I've been using "header line".

On Mon, 17 Jan 2005, william(at)elan.net wrote:

> And I do believe believe "header" and "headers" are what majority who talk 
> about email (and don't write RFCs) use nowdays.

Quite. That's why I started this discussion...

On Tue, 18 Jan 2005, Laird Breyer wrote:

> "header", "message header" or "mail header" refers to the full set of
> data before the empty line which marks the beginning of the
> body. (Actual layout and spaces play no role).

I think that usage confuses those who talk about email and don't
read/write RFCs.

> I do like to put the ':' after a field name to make implicitly clear
> that it's a header line being discussed (e.g. let's discuss the
> Received: line), even though the RFC (2)288 doesn't consider ':' part
> of the field name.

Yes; I do that too.

Regards,
Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Mon, 17 Jan 2005 19:57:22 -0800 (PST)
From: "william(at)elan.net" <[email-masked]>
To: Charles Lindsey <[email-masked]>
cc: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Mon, 17 Jan 2005, Charles Lindsey wrote:

> In <[email-masked]> "william(at)elan.net" <[email-masked]> writes:
> 
> >Now as has been mentioned, I plan to document these Original- headers in the
> >Internet draft so if you like to provide some feedback about it, please go
> >ahead (and note that I already know of number of different programs adding 
> >these headers without any documentation and draft will try to keep it
> >primarily as documentation of current practice and explain what these are).
> 
> >Accidentally I can't find any good way to register with IANA a group of headers
> >like Original- (as common prefix) as RFC3864 specifies only how to do it for
> >one particular header field. Please suggest what to do about it.
> 
> I think that registry is intended for fields that agents might be expected
> to recognize and act upon. I doubt any MUA is going to have a
> built-in-recognition of any and all fields beginning with "Original-".
> Surely they are intended for human consumption, in which case using
> "X-Original-xxxxxx" would he just as good, and would break no rules or
> regisries?

Aren't trace fields like "Received:" intended for human consumption too? :)

And in this case, while its primarily trace data for humans for debugging 
it could potentially be used by MTAs, for example to detect complex mail 
loop. Besides that some systems might choose to display "Original-Sender"
or "Original-From" in MUA as well (on part with Sender, From, Resent-From,
Resent-Sender when showing the entities involved in email transmission).

And while use I have seen involves both "X-Original" and "Original-"
(with same header names used afterward) depending on which system was 
doing it, lately "Original-" is becoming more common and I believe if 
it is to be documented in the RFC, it would have to be without "X-".

And its not exactly that it would "break" the registry rule, its just
that currently there is no way to do it and I think that means we'd
have to expand IANA header registry to allow to define certain set of
headers with common prefix or to specifically reserve "prefix" for
use with one type of headers (which are afterward specifically defined).
That generally means either separate draft for IANA or expansion of 
RFC3864 (but I really would prefer header prefix registry by on same page 
at IANA as specific header names registry).

This is useful for more then just "Original-", it can well be argued that
"Resent-" should also part of this header prefix registry as well as "MIME-"
and "Content-" (specific headers for those are defined in documents but
prefix use is reserved) and likely "Envelope-" would fall into this 
category as well for reserving for specific use. Another of my upcoming 
drafts (see www.metasignatures.org) is using "Saved-" headers in similar 
way as "Original-" for duplication of existing header data (although I'm 
seriously considering if/how it could be combined with use of "Original-").

-- 
William Leibzon
Elan Networks
[email-masked]

Date: Mon, 17 Jan 2005 17:39:17 GMT
From: Charles Lindsey <[email-masked]>
Newsgroups: local.mime
Subject: Experiment #3 with multiple Reference headers
Apparently-To: [email-masked]
Sender: [email-masked]

This message responds to several others, and contains a References header for
each of them. So it cannot be expected that existing MUAs will thread them
particularly well. So the things to look out for are:

1. Did it arrive with all of them intact?
2. Did is arrive with at least one of them intact? If so, was it the first
(can be recognized by presence of <[email-masked]>)?
3. Did it do anything really nasty with the threading?
4. Did it even manage to make some use of the extra headers?

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Your second References: are shown as Original-References:
>For gateways that's probably the best solution, and it's
>compatible with William's draft about Original-* headers.

Fair enough!

In <[email-masked]> Keith Moore <[email-masked]> writes:

>barf.  the last thing we need is for agents to start renaming headers.

No, that is perfectly acceptable in this case. All I expect for the moment is
for existing agents to see and utilize at least one of those headers,
preferably the first one. That should get the message threaded with at least
one of its precursors, and one cannot expect more of an existing user agent in
_any_ scheme designed to address this problem.

In <[email-masked]> Paul Overell <[email-masked]> writes:

>Arrived with two (identical) References: headers.

>The MUA, Turnpike, will ignore the second and subsequent Reference: 
>headers.   So no problem caused and threaded OK.

OK, that is fine.

The copy that came back to my own machine also had both headers intact.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4381
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Mon, 17 Jan 2005 17:35:03 GMT
Lines: 29
Sender: [email-masked]

In <[email-masked]> "william(at)elan.net" <[email-masked]> writes:

>Now as has been mentioned, I plan to document these Original- headers in the
>Internet draft so if you like to provide some feedback about it, please go
>ahead (and note that I already know of number of different programs adding 
>these headers without any documentation and draft will try to keep it
>primarily as documentation of current practice and explain what these are).

>Accidentally I can't find any good way to register with IANA a group of headers
>like Original- (as common prefix) as RFC3864 specifies only how to do it for
>one particular header field. Please suggest what to do about it.

I think that registry is intended for fields that agents might be expected
to recognize and act upon. I doubt any MUA is going to have a
built-in-recognition of any and all fields beginning with "Original-".
Surely they are intended for human consumption, in which case using
"X-Original-xxxxxx" would he just as good, and would break no rules or
regisries?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 18 Jan 2005 12:56:16 +1000
From: Laird Breyer <[email-masked]>
To: "william(at)elan.net" <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Reply-To: [email-masked]
Mail-Followup-To: "william(at)elan.net" <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

On Jan 17 2005, william(at)elan.net wrote:
> 
> It would be nice if you give direction on what to use in documents. 
> 

To expand my point of view, FWIW:

As a general principle, for (mail transport) discussions, commonly
used words such as "mail", "body", "header" should be invariant under
(mail transport) transformations. This is important, otherwise we have
to constantly think how the object being discussed changes as the mail
is moved. (This is a principle used in other areas of science).

For example, suppose the "header line" is defined as Charles proposed.
Say I want to discuss the third "header line" in the following text
fragment:

Received: from sokol.elan.net (localhost.localdomain [127.0.0.1])
        by sokol.elan.net (8.12.11/8.12.5) with ESMTP id j0I0l460005503;
        Mon, 17 Jan 2005 16:47:04 -0800

I probably wouldn't discuss the third line because it's number three,
but I might say "let's discuss the header line containing the
date-time element".  Now everybody has to waste time thinking about what
happens if the Received: line is reformatted, etc. This example is
easy enough, but other examples would not be.

Of course, sometimes it is important to discuss non-invariant
quantities.  In those cases, it's reasonable to use more words to
describe sufficiently precisely the objects under scrutiny. For
example, the third "header line" in the example above could be called
the "third fold of the Received: field".

So the principle says that commonly used words should be reserved for
invariant objects. 

Since others are explaining their preferred terminology, here's what I
like:

"header", "message header" or "mail header" refers to the full set of
data before the empty line which marks the beginning of the
body. (Actual layout and spaces play no role).

"header line" is a single unfolded "keyword: data" pair.

"X: header", "X: line", where X is a keyword is also a single unfolded 
pair of the form "X: data".

"X: value" (e.g. From: value) is the data portion of the line.

I don't think I use anything else, it's too rare for me to discuss smaller
objects directly. 

I do like to put the ':' after a field name to make implicitly clear
that it's a header line being discussed (e.g. let's discuss the
Received: line), even though the RFC (2)288 doesn't consider ':' part
of the field name.

-- 
Laird Breyer.

Date: Mon, 17 Jan 2005 16:47:04 -0800 (PST)
From: "william(at)elan.net" <[email-masked]>
To: Laird Breyer <[email-masked]>
cc: [email-masked], <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Tue, 18 Jan 2005, Laird Breyer wrote:

> A "header line" as you propose is not a semantic unit, since folding
> white space can break the information at all sorts of places, and as
> the mail winds its way to its destination, folded lines can be
> rearranged several times in semantically equivalent ways.
> 
> If people start using header lines in this sense, I think there's the
> inbuilt potential for confusion in some types of discussion.
> 
> To me, it's more natural to associate "header line" with the fully unfolded
> piece of the header which begins with a field name and colon, since such
> a "header line" is invariant under mail transport (as much as possible).

It would be nice if you give direction on what to use in documents. 

Personally I've used simply "header" most of the time and sometimes more 
formal "header line" where as when referencing all headers I use "headers"
or "headers data". I typically do not differentiate as to if header has one 
or more then one line in the message or refer to any specific line because 
any header can have more one line but it does not change that it is still 
one header that could be represented by one single line if its all folded
together (where as two different headers could not).

And I do believe believe "header" and "headers" are what majority who talk 
about email (and don't write RFCs) use nowdays.

-- 
William Leibzon
Elan Networks
[email-masked]

Date: Tue, 18 Jan 2005 09:58:28 +1000
From: Laird Breyer <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Reply-To: [email-masked]
Mail-Followup-To: [email-masked], [email-masked]
Sender: [email-masked]

On Jan 18 2005, Charles Lindsey wrote:

> In <[email-masked]> Keith Moore <[email-masked]> writes:
> 
> >OTOH, "header line" seems ambiguous to me - I think you must mean 
> >something _besides_ header field because you're not using either 
> >"header field" or the common abbreviation "header".  :)
> 
> I think we need the term "header line" when speaking of the individual
> lines of a header field that has been folded.
> 

A "header line" as you propose is not a semantic unit, since folding
white space can break the information at all sorts of places, and as
the mail winds its way to its destination, folded lines can be
rearranged several times in semantically equivalent ways.

If people start using header lines in this sense, I think there's the
inbuilt potential for confusion in some types of discussion.

To me, it's more natural to associate "header line" with the fully unfolded
piece of the header which begins with a field name and colon, since such
a "header line" is invariant under mail transport (as much as possible).

-- 
Laird Breyer.

To: [email-masked]
Xref: clerew local.mime:4380
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Mon, 17 Jan 2005 17:28:06 GMT
Lines: 19
Sender: [email-masked]

In <[email-masked]> Keith Moore <[email-masked]> writes:

>OTOH, "header line" seems ambiguous to me - I think you must mean 
>something _besides_ header field because you're not using either 
>"header field" or the common abbreviation "header".  :)

I think we need the term "header line" when speaking of the individual
lines of a header field that has been folded.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: Philip Hazel <[email-masked]>, Frank Ellermann <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Mon, 17 Jan 2005 11:55:57 -0500
To: "william(at)elan.net" <[email-masked]>
Sender: [email-masked]

>> Lists that set Sender are broken anyway, because (as you point out)
>> this obscures who originally sent the message.  Of course, there are
>> exceptions - such as digests and anonymous lists - but they're rare.
>
> I agree with you, but this is not what current practice is and  
> practicly
> all mail lists do add Sender header (including this ietf-822 list).

...and it needs to be stopped, because now the pseudo-authentication  
anti-spam systems want to depend on that behavior, which will only make  
mail even less reliable than it is now.

> And I think this may even have been documented, I seem to remember  
> reference
> in some RFC that Sender header should be added by mail lists.

maybe.  I don't remember having seen it.  if you can find the reference  
please let us know.

>> In general, if software thinks it needs to rename a field that's
>> already there, there's usually something fundamentally wrong with the
>> _intent_ of that software.   The primary purpose of submission agents,
>> MTAs, lists, and recipient MUAs is to _preserve_ the original message,
>> not to alter it.
>
> Well, the primary purpose of any MTS is to delivery email or rather in
> current world (with spam all around) it is to delivery email that  
> recipient
> would like to see. Header data are part of toolset used by MUAs and  
> MTAs
> as means of identifying properties of that messages and how it its been
> (or being) delivered. Addition of information to identify that message
> passed through email list and identification of mail list email address
> seems quite appropriate and such data is usefull for the user.

we have List-* fields for that purpose.

> What is not ok is loss of data, such as original Sender or in fact data
> about previous mail lists if message passed through more then one and
> then List-* headers are all overridden.

well, arguably List-* fields tell the recipient about the list from  
which he received the message.

and rfc2369 says:
>    Mail list processors should not allow any user-originated list  
> header
>    fields to pass through to their lists, lest they confuse the user  
> and
>    have the potential to create security problems.

which sort of implies that List-* fields from earlier lists are going  
to be deleted, because the list processor has no way of knowing the  
difference between "user-originated" fields and fields added by earlier  
lists.

> But in the mean time, you're welcome to comment on the following:
>   
> http://www.elan.net/~william/emailsecurity/draft-leibzon- 
> emailredirection-traceheaders-00.txt

thanks.  it will probably be a week or three, but I'll look at it.

Keith

Date: Mon, 17 Jan 2005 05:56:31 -0800 (PST)
From: "william(at)elan.net" <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Philip Hazel <[email-masked]>, Frank Ellermann <[email-masked]>, <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Sun, 16 Jan 2005, Keith Moore wrote:

> Lists that set Sender are broken anyway, because (as you point out) 
> this obscures who originally sent the message.  Of course, there are 
> exceptions - such as digests and anonymous lists - but they're rare.

I agree with you, but this is not what current practice is and practicly 
all mail lists do add Sender header (including this ietf-822 list). And I 
think this may even have been documented, I seem to remember reference 
in some RFC that Sender header should be added by mail lists. 

> In general, if software thinks it needs to rename a field that's 
> already there, there's usually something fundamentally wrong with the 
> _intent_ of that software.   The primary purpose of submission agents, 
> MTAs, lists, and recipient MUAs is to _preserve_ the original message, 
> not to alter it.

Well, the primary purpose of any MTS is to delivery email or rather in
current world (with spam all around) it is to delivery email that recipient
would like to see. Header data are part of toolset used by MUAs and MTAs 
as means of identifying properties of that messages and how it its been 
(or being) delivered. Addition of information to identify that message 
passed through email list and identification of mail list email address
seems quite appropriate and such data is usefull for the user.

What is not ok is loss of data, such as original Sender or in fact data 
about previous mail lists if message passed through more then one and 
then List-* headers are all overridden.
 
> > Now as has been mentioned, I plan to document these Original- headers 
> > in the
> > Internet draft so if you like to provide some feedback about it, 
> > please go
> > ahead
> 
> Naturally, I'd like to see the draft before I comment on it.

I'm working on half a dozen documents and among then I'm afraid this draft 
is not high priority (and its not going to be anything super new anyway), 
but I'll try to finish within next 3 months...

But in the mean time, you're welcome to comment on the following:
 http://www.elan.net/~william/emailsecurity/draft-leibzon-emailredirection-traceheaders-00.txt

Note - it was published as internet draft, but IETF version has several 
formatting errors with new lines, actually even above on my page still has 
few of these and this first version has number of other text problems too, 
so I'd like to see comments on the concept in general for now.

-- 
William Leibzon
Elan Networks
[email-masked]

Cc: Philip Hazel <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Mon, 17 Jan 2005 08:10:54 -0500
To: Tony Finch <[email-masked]>
Sender: [email-masked]

>>> OTOH, "header line" seems ambiguous to me - I think you must mean 
>>> something
>>> _besides_ header field because you're not using either "header 
>>> field" or the
>>> common abbreviation "header".  :)
>>
>> With respect, you are an "unaverage reader" :-)  I use "header line" 
>> in
>> documentation about Exim (for example), because I strongly suspect 
>> that,
>> to the "average" reader, the word "field" is strange and esoteric when
>> used in this kind of sense.
>
> There's precedent in RFC 2821 which has a section 3.8.2 Received Lines 
> in
> Gatewaying, and refers to lines rather than fields throughout section 
> 4.4
> Trace Information. Also, RFC 2822 says that header fields are lines,
> albeit with rules for folding them into a multi-line form. See also
> section 3.6.3. Destination address fields for some slack editing - line
> appears five times where field would be more correct.

I guess you could say I'm "old school" :)

I think of header fields as being able to be wrapped onto one or more 
lines, each terminated by CRLF.

Date: Mon, 17 Jan 2005 11:43:04 +0000
From: Tony Finch <[email-masked]>
To: Philip Hazel <[email-masked]>
cc: Keith Moore <[email-masked]>, [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Mon, 17 Jan 2005, Philip Hazel wrote:
> On Sun, 16 Jan 2005, Keith Moore wrote:
>
> > OTOH, "header line" seems ambiguous to me - I think you must mean something
> > _besides_ header field because you're not using either "header field" or the
> > common abbreviation "header".  :)
>
> With respect, you are an "unaverage reader" :-)  I use "header line" in
> documentation about Exim (for example), because I strongly suspect that,
> to the "average" reader, the word "field" is strange and esoteric when
> used in this kind of sense.

There's precedent in RFC 2821 which has a section 3.8.2 Received Lines in
Gatewaying, and refers to lines rather than fields throughout section 4.4
Trace Information. Also, RFC 2822 says that header fields are lines,
albeit with rules for folding them into a multi-line form. See also
section 3.6.3. Destination address fields for some slack editing - line
appears five times where field would be more correct.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
COLWYN BAY TO THE MULL OF GALLOWAY INCLUDING THE ISLE OF MAN: SOUTHWEST 5 OR 6
INCREASING 7 TO SEVERE GALE 9. SHOWERS, TURNING WINTRY LATER. MODERATE OR
GOOD. MODERATE BUILDING ROUGH.

Date: Mon, 17 Jan 2005 10:21:26 +0000
To: Philip Hazel <[email-masked]>
Cc: Keith Moore <[email-masked]>, Frank Ellermann <[email-masked]>, [email-masked]
From: Richard Clayton <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

In article <[email-masked]>,
Philip Hazel <[email-masked]> writes

>My feeling is that almost everybody has a similar kind of idea of what a
>"line" is...

I suspect the original authors may have worried about how obvious a
concept a "line" might be when many of them were likely to be folded :(

>Anyway, if we can agree to call each individual line/field a "header", 
>and never to use "header" to mean the entire collection, then even more
>people may be less confused, 

:)

>which will be a Good Thing.

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

Date: Mon, 17 Jan 2005 09:17:16 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Sun, 16 Jan 2005, Keith Moore wrote:

> OTOH, "header line" seems ambiguous to me - I think you must mean something
> _besides_ header field because you're not using either "header field" or the
> common abbreviation "header".  :)

With respect, you are an "unaverage reader" :-)  I use "header line" in 
documentation about Exim (for example), because I strongly suspect that,
to the "average" reader, the word "field" is strange and esoteric when
used in this kind of sense. If I say "the From: header line", I think 
that more people - in particular including those for whom English is not
their first language - will understand what I mean than if I were to say
"the From: header field".

Indeed, if I hadn't read the RFC myself (and, let's face it, a heck of a 
lot of people who are involved in running email systems don't actually 
read the RFCs), I would have guessed that "the From: header field" meant
the data that followed the word "From:" rather than the whole thing,
with "From:" naming the field. I guess this is because, to me, a "field"
is a smallish chunk of data. I don't know if I'm atypical in this
respect. Maybe I am.

My feeling is that almost everybody has a similar kind of idea of what a
"line" is...

Of course, I may be entirely wrong here. It's all very subjective.

Anyway, if we can agree to call each individual line/field a "header", 
and never to use "header" to mean the entire collection, then even more
people may be less confused, which will be a Good Thing.

Philip

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Date: Mon, 17 Jan 2005 09:17:18 +0000
To: [email-masked]
From: Paul Overell <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

In message <[email-masked]>, Charles Lindsey 
<[email-masked]> writes
>
>Well, as you know, the first attempt failed because of excessing checking
>before it left my machine. This one will be piped directly into sendmail,
>and I will examine the outgoing queue to make sure it is intact.
>
>This message contains two Reference headers. Please report whether it is
>delivered intact and whether it causes any other problems, such as
>peculiar threading.

Arrived with two (identical) References: headers.

The MUA, Turnpike, will ignore the second and subsequent Reference: 
headers.   So no problem caused and threaded OK.

>
>For this first experiment, both the Reference headers are identical. If
>this causes no obvious problems, then I shall try further examples
>representing various DAGs.
>

Regards
-- 
Paul Overell         Internet Platform Development Manager, Thus plc

Cc: Philip Hazel <[email-masked]>, Frank Ellermann <[email-masked]>, <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Sun, 16 Jan 2005 19:37:59 -0500
To: "william(at)elan.net" <[email-masked]>
Sender: [email-masked]

>>> I'm afraid we've had that for some time. For example, if I recall
>>> correctly, Pine renames "resent-" header lines if you resend the
>>> message
>>> again.
>>
>> which is a good example of why renaming header fields doesn't work 
>> very
>> well.  we don't need to rename received fields, why do we need to
>> rename other fields?
>
> RFC2821 is quite clear that software should not be doing it but should 
> instead
> add additional set of Resent-* header lines [aka headers] to the top 
> of the
> message. The same document also explains that Resent-* headers have 
> meaning
> as a group and must not be separated afterward. So situation that some 
> other
> Resent-* header from previous addition gets mixed with newly added 
> Resent-
> header is practically impossible (not unless there were no other trace 
> headers
> like received in between).
>
> At the same time for majority of other headers (except Resent and 
> Received)
> adding additional header line with same name is not allowed. The common
> practice has been to simply replace original header line with new one. 
> But
> its really good for debugging of email routing problems to have seen 
> what
> the value has been (for example I really would like to see original 
> Sender
> header line value preserved somewhere for email messages that go 
> through
> email list).

Lists that set Sender are broken anyway, because (as you point out) 
this obscures who originally sent the message.  Of course, there are 
exceptions - such as digests and anonymous lists - but they're rare.

In general, if software thinks it needs to rename a field that's 
already there, there's usually something fundamentally wrong with the 
_intent_ of that software.   The primary purpose of submission agents, 
MTAs, lists, and recipient MUAs is to _preserve_ the original message, 
not to alter it.

Again, there are exceptions that justify some lack of transparency, 
such as virus filters.   But those should be acting on entire messages 
or entire body parts - they shouldn't be messing with fields of 
messages and body parts that they deliver intact.

> Now as has been mentioned, I plan to document these Original- headers 
> in the
> Internet draft so if you like to provide some feedback about it, 
> please go
> ahead

Naturally, I'd like to see the draft before I comment on it.

Keith

Date: Mon, 17 Jan 2005 00:02:47 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

> > > I'm afraid we've had that for some time. For example, if I recall
> > > correctly, Pine renames "resent-" header lines if you resend the
> > > message
> > > again.
> >
> > which is a good example of why renaming header fields doesn't work very
> > well.  we don't need to rename received fields, why do we need to
> > rename other fields?
>
> RFC2821 is quite clear that software should not be doing it but should instead
> add additional set of Resent-* header lines [aka headers] to the top of the
> message.

The reason Pine renames Received- fields is in order to remain compatible
with RFC 822, which does not support multiple sequential re-sendings of
the same message.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SOUTH UTSIRE: SOUTHWEST 5 TO 7, INCREASING GALE 8 FOR A TIME. RAIN OR SHOWERS.
MODERATE OR GOOD.

Date: Sun, 16 Jan 2005 15:45:53 -0800 (PST)
From: "william(at)elan.net" <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Philip Hazel <[email-masked]>, Frank Ellermann <[email-masked]>, <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Sun, 16 Jan 2005, Keith Moore wrote:

> > I'm afraid we've had that for some time. For example, if I recall
> > correctly, Pine renames "resent-" header lines if you resend the 
> > message
> > again.
> 
> which is a good example of why renaming header fields doesn't work very 
> well.  we don't need to rename received fields, why do we need to 
> rename other fields?

RFC2821 is quite clear that software should not be doing it but should instead
add additional set of Resent-* header lines [aka headers] to the top of the
message. The same document also explains that Resent-* headers have meaning
as a group and must not be separated afterward. So situation that some other
Resent-* header from previous addition gets mixed with newly added Resent- 
header is practically impossible (not unless there were no other trace headers
like received in between).

At the same time for majority of other headers (except Resent and Received)
adding additional header line with same name is not allowed. The common 
practice has been to simply replace original header line with new one. But
its really good for debugging of email routing problems to have seen what
the value has been (for example I really would like to see original Sender
header line value preserved somewhere for email messages that go through 
email list). So some software instead of removing the header line that it 
is about to be replaced, duplicates it into corresponding Original- (some 
rename and leave in same place inside email headers structure, this is 
less good) and then adds new header line with same name. So in practice 
Original- header lines are more like email trace header fields (and 
similarly multiple of same named ones can be in the message) and have same 
purpose to help track changes to email message, they should not be viewed 
as replacement for real headers or used directly by MUA.

Now as has been mentioned, I plan to document these Original- headers in the
Internet draft so if you like to provide some feedback about it, please go
ahead (and note that I already know of number of different programs adding 
these headers without any documentation and draft will try to keep it
primarily as documentation of current practice and explain what these are).

Accidentally I can't find any good way to register with IANA a group of headers
like Original- (as common prefix) as RFC3864 specifies only how to do it for
one particular header field. Please suggest what to do about it.

> (actually, resent doesn't work very well.  but it's rarely a problem as 
> long as it is only used for its intended purpose.  unfortunately people 
> keep trying to redefine the meaning of resent-* and other fields, most 
> recently to accommodate various shortsighted anti-spam schemes)

I certainly agree with you that Microsoft proposed SenderID ho they want to
use Resent- headers is incorrect and against existing standards (RFC2821)
and in general their proposal is bad and will hardly do anything to stop
spam flow of spam or phishing.
 
---
William Leibzon, Elan Networks:
 mailto: [email-masked]
Anti-Spam and Email Security Research Worksite:
 http://www.elan.net/~william/emailsecurity/

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers
Date: Sun, 16 Jan 2005 22:45:39 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 13
Cc: [email-masked]
Sender: [email-masked]

Keith Moore wrote:

>> For gateways that's probably the best solution, and it's
>> compatible with William's draft about Original-* headers.

> barf.  the last thing we need is for agents to start renaming
> headers.

Better than silently removing it, and better than rejecting the
article at the mail2news gateway.  What would you do as gateway
in this case, keep it as is ?
                              Bye, Frank

Cc: Frank Ellermann <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Sun, 16 Jan 2005 17:17:42 -0500
To: Philip Hazel <[email-masked]>
Sender: [email-masked]

>> barf.  the last thing we need is for agents to start renaming headers.
>
> I'm afraid we've had that for some time. For example, if I recall
> correctly, Pine renames "resent-" header lines if you resend the 
> message
> again.

which is a good example of why renaming header fields doesn't work very 
well.  we don't need to rename received fields, why do we need to 
rename other fields?

(actually, resent doesn't work very well.  but it's rarely a problem as 
long as it is only used for its intended purpose.  unfortunately people 
keep trying to redefine the meaning of resent-* and other fields, most 
recently to accommodate various shortsighted anti-spam schemes)

> Incidentally, one of these days we should perhaps bring the
> specification's terminology in line with what everybody actually uses.
> Pretty well everybody, when they write "header" mean a single header
> line (as you seem to above). The RFC actually calls this a "header
> field":

yup.  I do try to use "header field" like the RFC says, but sometimes I 
slip and abbreviate this to "header".  fortunately, it's usually not a 
problem to distinguish between "header" as used to mean "header field", 
and "header" as used to mean "entire header" from context.

OTOH, "header line" seems ambiguous to me - I think you must mean 
something _besides_ header field because you're not using either 
"header field" or the common abbreviation "header".  :)

Date: Sun, 16 Jan 2005 21:57:12 +0000 (GMT)
From: Philip Hazel <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Sender: [email-masked]

On Sun, 16 Jan 2005, Keith Moore wrote:

> barf.  the last thing we need is for agents to start renaming headers.

I'm afraid we've had that for some time. For example, if I recall
correctly, Pine renames "resent-" header lines if you resend the message
again. It does this, I suspect, because it is impossible to distinguish
between different sets of "resent-" header lines. For example, if you
receive

resent-from: X
resent-to: Y
resent-date: Z
resent-cc: Q
resent-from: A
resent-to: B

to which set do "date:" and "cc:" belong?

Incidentally, one of these days we should perhaps bring the 
specification's terminology in line with what everybody actually uses. 
Pretty well everybody, when they write "header" mean a single header 
line (as you seem to above). The RFC actually calls this a "header 
field":

   A message consists of header fields (collectively called "the header
   of the message") followed, optionally, by a body.  The header is a  
   sequence of lines of characters with special syntax as defined in 
   this standard. 

Note that "the header" is the whole thing. I have been trying to
remember to use "header line" consistently, but the colloquial use of
"header" is so prevalent that I think it's a losing battle (and I tend 
to forget sometimes). Also, phrases such as "the From: header line" are 
probably felt to be a bit cumbersome, compared with "the From: header".

I propose that next time there's a big revision, we should go with the 
current actual usage, and find a new name for the collection of
"headers".

-- 
Philip Hazel            University of Cambridge Computing Service,
[email-masked]      Cambridge, England. Phone: +44 1223 334714.

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Sun, 16 Jan 2005 16:19:12 -0500
To: Frank Ellermann <[email-masked]>
Sender: [email-masked]

> You can see what arrived on GMaNe with its "raw" display:
> <http://article.gmane.org/gmane.ietf.rfc822:11236:raw>
>
> Your second References: are shown as Original-References:
> For gateways that's probably the best solution, and it's
> compatible with William's draft about Original-* headers.

barf.  the last thing we need is for agents to start renaming headers.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: General considerations for new message field specifications
Cc: "Charles Lindsey" <[email-masked]>
Date: Sun, 16 Jan 2005 15:54:37 -0500
Sender: [email-masked]

On Fri January 14 2005 11:32, Charles Lindsey wrote:

> The most prominent of these would be to mention Graham Klyne's RFC 3864,
> which introduces the IANA header registry

OK, I can add that as an informative reference.

Mind you, IANA seems to be more than a bit confused, as
their web page for the registry refers to "header messages"
and "message header messages", neither of which makes any
sense, but that's another matter...

> The next point concerns the use of RFC 2822-style syntax, which you
> rightly seek to encourage. A problem arises when you need to take or
> modify something out of an earlier standard which was produced back in the
> pre-2822 days. A good working example would be the MIME-style <parameter>.
> There are now at least two other standards which have taken over the basic
> idea first introduced in RFC 2045 (strictly, in its predecessor). One is
> the Content-Distribution header and the other is the auto-submitted field
> (RFC 3834). I have also seen a recent draft for an OpenPGP header.

I have never heard of "Content-Distribution". Do you have a reference?

> The problems all such proposal have in common is the necessity to:
> 
> 1. Make it clear where [CFWS] can be inserted (currently, it is hard work
> to deduce this from RFC 2045 cum RFC 822).

Yes, that is an issue with RFC 822, which affects several RFCs
that use 822 syntax, and is addressed to some extent by RFC 2822
(and will hopefully be improved in 2822's successor). It is addressed
by the draft.

> 2. Make it clear that more parameters will appear in future extensions,
> which MUST NOT be generated yet, but MUST be accepted and ignored now.
> Except that paramaters with x-tokens MAY be generated now.

There are only three fields out of about 140 that use those
so-called "parameters". They are the MIME fields Content-Type
and Content-Distribution, and Keith's Auto-Submitted field
(which is probably partially my fault, as otherwise it would
have introduce a third syntax for parameters (the second being
that used by the Disposition-Notification-Options field).

> 3. Make it clear when the value part of any parameters that are defined
> need to be quoted.

That's covered by RFCs 2045 and 2231 for the fields where it
applies.

> 4. Make it clear thqt RFC 2231 applies.

I hesitate to list every RFC that might apply to some extension field,
as that would be a very long list which would quickly become outdated.
 
> RFC 3834 makes a reasonable job of all these things, except that it avoids
> #3 by not actually defining any parameters :-) . But the OpenPGP draft is
> a bot weak, hence the need for you to cover thses issues.

Specific concerns w.r.t. a specific draft are probably best addressed
to the author(s) of that draft at this time. 

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Date: Sun, 16 Jan 2005 18:55:28 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 13
Sender: [email-masked]

Charles Lindsey wrote:

> Please report

You can see what arrived on GMaNe with its "raw" display:
<http://article.gmane.org/gmane.ietf.rfc822:11236:raw>

Your second References: are shown as Original-References:
For gateways that's probably the best solution, and it's
compatible with William's draft about Original-* headers.

                Bye, Frank

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Sat, 15 Jan 2005 15:52:12 -0500
Cc: Keith Moore <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Fri January 14 2005 20:21, Keith Moore wrote:
> If done in conjunction with an online test suite, and a way to report 
> errors, it might be more useful.   That way, if vendors didn't test 
> their products, their users would.  Ideally we'd get IETF to host the 
> test suite.  something like mimetest.ietf.org .  though there should be 
> a backup plan in case this is too politically sensitive.

The MIME message generator is OK for testing parsers to some
extent (limited types, there had been some errors which I
think have been corrected), as are examples in various RFCs.
For testing generator output, there is a service available
via one of the IETF pages, but last time I checked it didn't
recognize the standard "Comments" field.

Testing parsers is difficult because of the tremendous variety
of (valid and invalid) message content that can be generated.
Testing generators is comparatively simple, as validating
parsers exist.

Cc: Arnt Gulbrandsen <[email-masked]>
From: Keith Moore <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 20:21:02 -0500
To: ietf-822 <[email-masked]>
Sender: [email-masked]

> If you want to write a "Testing Considered Useful" draft,
> go ahead; but I doubt that any implementers who have
> ignored the examples contained within the specification
> RFCs will wake up and take notice.

If done in conjunction with an online test suite, and a way to report 
errors, it might be more useful.   That way, if vendors didn't test 
their products, their users would.  Ideally we'd get IETF to host the 
test suite.  something like mimetest.ietf.org .  though there should be 
a backup plan in case this is too politically sensitive.

Keith

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 17:59:04 -0500
Cc: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Fri January 14 2005 08:40, Arnt Gulbrandsen wrote:

> 4. Some people test only with a corpus of real-world mail. Me, I have a 
> corpus of 1.6 million messages where groups don't occur even once. In 
> such a setting, it's easy for group-related bugs to slip through.

Maybe you should add the dozen or so examples contained
within RFC 2822...
 
> The primary barrier is IMO in the large number of deployed mail 
> parsers/readers that haven't been tested with groups, and cannot be 
> assumed to work.

If we start throwing away pieces because some implementer
botched handling, we'd get rid of all of MIME, Date fields,
Received fields, all address fields, In-Reply-To, and
References... I doubt that much would be left.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 17:51:01 -0500
Cc: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Fri January 14 2005 07:37, Arnt Gulbrandsen wrote:

> > surely it's harder to read and adhere to the combination of both RFCs 
> > X and Y? particularly when they give conflicting advice?
> 
> Specious comparison. People aren't really adhering to RFC X in the first place.

Flawed generalization from a specific case; just because "Gmail"
doesn't adhere to the specifications doesn't mean that nobody
pays attention to the relevant RFCs.  Some of us take particular
care to adhere to the specifications.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 17:43:24 -0500
Cc: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

On Fri January 14 2005 04:38, Arnt Gulbrandsen wrote:

> [G]roup syntax has been so little used in the past years that 
> IMO it's better to kill the syntax than to try to support it.

Bad premise; group syntax is used quite a bit.

> As a  
> feature, group syntax simply isn't carrying its load.

The capabilities have been somewhat limited since RFC 822
discontinued the provision for nested lists which existed
in RFCs 724 & 733.
 
> IMO, the best option would be to issue a short RFC saying:
> 
> 1. Mail senders SHOULD NOT generate group syntax.

I would not support that.

> 2. Mail receivers are urged to test that they can parse empty groups, 
> since several current senders generate that, e.g. "To: 
> unknown-recipients:;" and "To: ietf-announce:;".

Of course implementers should test their parser
implementations against legal content. RFCs 724, 733, 822,
and 2822 have all spoon-fed prospective implementers with
examples containing list syntax among other things.
 
> Comments?

If you want to write a "Testing Considered Useful" draft,
go ahead; but I doubt that any implementers who have
ignored the examples contained within the specification
RFCs will wake up and take notice.

Date: Fri, 14 Jan 2005 18:25:26 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Experiment #2 with multiple Reference headers (was References with multipl
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

Charles Lindsey writes:
> Well, as you know, the first attempt failed because of excessing 
> checking before it left my machine.

It didn't fail - it just showed that a transit agent will "clean" the 
message in ways that would modify the meaning of the message, should 
multiple references fields be allowed.

Which transit agent was it, if I may ask?

Arnt

To: [email-masked]
Xref: clerew local.mime:4348
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: General considerations for new message field specifications
Date: Fri, 14 Jan 2005 16:32:43 GMT
Lines: 56
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>I've thrown together an Internet Draft based on our discussion
>a few months ago.  It should appear soon at the usual places.

Actually, it showed up at the "usual places" before I saw it here.

>This mailing lists is as good a place as any for discussion;
>private comments are of course welcome.

OK. Essentially it is fine as far as it goes, but there are a few further
things that could usefully be said.

The most prominent of these would be to mention Graham Klyne's RFC 3864,
which introduces the IANA header registry. Writers of drafts for new
headers should be encouraged to take out a provisional registration of the
header-name at the same time as the initial draft, and also to include
instructions for the header to be added to the permanent registry in an
IANA Considerations section.

The next point concerns the use of RFC 2822-style syntax, which you
rightly seek to encourage. A problem arises when you need to take or
modify something out of an earlier standard which was produced back in the
pre-2822 days. A good working example would be the MIME-style <parameter>.
There are now at least two other standards which have taken over the basic
idea first introduced in RFC 2045 (strictly, in its predecessor). One is
the Content-Distribution header and the other is the auto-submitted field
(RFC 3834). I have also seen a recent draft for an OpenPGP header.

The problems all such proposal have in common is the necessity to:

1. Make it clear where [CFWS] can be inserted (currently, it is hard work
to deduce this from RFC 2045 cum RFC 822).
2. Make it clear that more parameters will appear in future extensions,
which MUST NOT be generated yet, but MUST be accepted and ignored now.
Except that paramaters with x-tokens MAY be generated now.
3. Make it clear when the value part of any parameters that are defined
need to be quoted.
4. Make it clear thqt RFC 2231 applies.

RFC 3834 makes a reasonable job of all these things, except that it avoids
#3 by not actually defining any parameters :-) . But the OpenPGP draft is
a bot weak, hence the need for you to cover thses issues.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 14 Jan 2005 16:37:30 GMT
To: [email-masked]
Subject: Experiment #2 with multiple Reference headers (was References with multipl
From: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

Well, as you know, the first attempt failed because of excessing checking
before it left my machine. This one will be piped directly into sendmail,
and I will examine the outgoing queue to make sure it is intact.

This message contains two Reference headers. Please report whether it is
delivered intact and whether it causes any other problems, such as
peculiar threading.

For this first experiment, both the Reference headers are identical. If
this causes no obvious problems, then I shall try further examples
representing various DAGs.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Fri, 14 Jan 2005 08:46:49 -0800
To: Keith Moore <[email-masked]>, Arnt Gulbrandsen <[email-masked]>
From: Laurence Lundblade <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Cc: [email-masked]
Sender: [email-masked]

At 05:05 AM 1/14/2005, Keith Moore wrote:

>>>>1. Mail senders SHOULD NOT generate group syntax.
>>>
>>>maybe, maybe not.  groups are useful.
>>
>>They're not used...
>
>they are used, just not terribly often.  but the utility of a feature is 
>not directly related to the frequency of its use.

Eudora generates it for a work group feature. I believe it is important for 
the work group feature to work well (between Eudora users).

Pine used to generate it and still does if nothing has changed, but I 
haven't used Pine for a few years.

LL

Date: Fri, 14 Jan 2005 15:52:25 +0000
From: Tony Finch <[email-masked]>
To: [email-masked]
Subject: Re: Making group syntax do-not-generate
Sender: [email-masked]

On Fri, 14 Jan 2005, Charles Lindsey wrote:
>
> What is really wanted is an allowance to have a genuine empty list, as in
>    To: <>
> or else to permit the To header to be omitted entirely.

The latter is already permitted.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LANDS END TO ST DAVIDS HEAD INCLUDING THE BRISTOL CHANNEL: SOUTH OR SOUTHEAST
4 OR 5, OCCASIONALLY 6 IN THE WEST. PATCHY RAIN. MODERATE OR GOOD. MODERATE OR
ROUGH, BUT SLIGHT IN SHELTER.

To: [email-masked]
Xref: clerew local.mime:4346
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 14:51:48 GMT
Lines: 52
Sender: [email-masked]

In <[email-masked]> Arnt Gulbrandsen <[email-masked]> writes:

>Bart Schaefer writes:
>> On Thu, 13 Jan 2005 09:47:24 -0500, Keith Moore <[email-masked]> wrote:
>>>  furthermore, you don't seem willing to accept or act on negative 
>>>  results, as evidenced by your continued insistence on using LIST: 
>>>  despite having been informed that it confuses one widely used mail 
>>>  user agent.

I do not recall any such confusion having been reported in the past, but I
see we have one now:
>>
>> It confuses Gmail, too.  Gmail displays the recipients (and addresses 
>> the replies) as LIST, ietf-822
>>
>> I suppose as a well-behaved beta-tester I should report that as a bug.

Yes, clearly a bug. Anyway, the discussion of my LIST idea have finished
way way back, so I have now removed that bit from my script.

>Perhaps. But group syntax has been so little used in the past years that 
>IMO it's better to kill the syntax than to try to support it. As a 
>feature, group syntax simply isn't carrying its load.

Indeed. It leads to confusion as to which headers are <mailbox-list> and
which are <address-list>, and I have never seen it used for its intended
purpose of addressing groups. Only for enabling an empty mailbox-list.

>IMO, the best option would be to issue a short RFC saying:

>1. Mail senders SHOULD NOT generate group syntax.
>2. Mail receivers are urged to test that they can parse empty groups, 
>since several current senders generate that, e.g. "To: 
>unknown-recipients:;" and "To: ietf-announce:;".

>Comments?

What is really wanted is an allowance to have a genuine empty list, as in
   To: <>
or else to permit the To header to be omitted entirely.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:4347
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Date: Fri, 14 Jan 2005 15:04:30 GMT
Lines: 55
Sender: [email-masked]

In <[email-masked]> Richard Clayton <[email-masked]> writes:

>>This message contains two Reference headers. 

>not here it didn't

>>Please report whether it is
>>delivered intact 

>exactly how might one know ?

Well evidently you did :-)

Of course, it should now be obvious from all the replies received that
there was a common point of failure, and that the finger should be pointed
at my machine. And indeed, that is where it happened (thanks to Henry
Spencer). SO I shall try again, and make sure it gets out this time.

>it always strikes me that the sort of people who combine two threads
>into one and answer bits of both at random (and who therefore demand two
>reference headers so that their valuable contributions can be correctly
>placed into the responses) are exactly the sort of people I find have
>contributed very little to the conversation :(

Maybe, but people keep raising this matter, both on this list and
elsewhere, so the demand is there. The chief problem is how the MUA would
display the thread, assuming all the necessary information could be
provided to it. Clearly, no article should be displayed earlier than any
of its ancestors, but there is no convenient way to display a DAG by
simple indentation, which works so nicely for trees. But that is really a
matter of waiting for some bright MUA implentor to dream up an acceptable
method.

>also, I endorse Keith Moore's point about the likely value of the
>experiments given the huge number of clients out there -- are you
>expecting me to feed this through a few dozen of my Perl scripts as
>well, to see what they do ? If they die("multiple References") then
>should we just criticise my coding style ?

No, I think this initial experiment can do no more than expose the
grossest problems that might arise, using a moderate collection of agents.
If that shows up no serious problem, then it would be worth while examing
on a larger scale what might possibly go wrong.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Cc: Arnt Gulbrandsen <[email-masked]>, [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 09:02:41 -0500
To: Tony Finch <[email-masked]>
Sender: [email-masked]

> The original idea for groups seems to have been as a way to note the
> address book lookup key that was used to retrieve the addresses,

I don't think so.  I've been under the impression that groups came from 
a convention used in paper business correspondence (like interoffice 
memos) to indicate to recipients "why" they were copied on the 
correspondence. ("why" as in - you're being copied on this 
correspondence because you are a member of the Foo board, rather than 
because of some other hat you wear)

Keith

Date: Fri, 14 Jan 2005 13:51:29 +0000
From: Tony Finch <[email-masked]>
To: Keith Moore <[email-masked]>
cc: Arnt Gulbrandsen <[email-masked]>, [email-masked]
Subject: Re: Making group syntax do-not-generate
Sender: [email-masked]

> > > > 1. Mail senders SHOULD NOT generate group syntax.
> > >
> > > maybe, maybe not.  groups are useful.
> >
> > They're not used...
>
> they are used, just not terribly often.  but the utility of a feature is not
> directly related to the frequency of its use.

In practice I've only ever seen groups used in the form To: undisclosed-
recipients:; and at least one MUA from Redmond gets the syntax laughably
wrong, like To: <undisclosed-recipients:;> [sic].

The original idea for groups seems to have been as a way to note the
address book lookup key that was used to retrieve the addresses, but since
that's generally only useful to the sender one can see why it hasn't been
used much. This could be changed with better directory technology
integrated into MUAs, but in the absence of that people have implemented
groups on the server side rather than the client side.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTIES CROMARTY FORTH TYNE DOGGER: SOUTH 5 TO 7, BUT 4 AT FIRST IN EAST
FORTIES AND EAST DOGGER. RAIN LATER. GOOD BECOMING MODERATE.

Date: Fri, 14 Jan 2005 13:42:05 +0000
From: Tony Finch <[email-masked]>
To: Richard Clayton <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Sender: [email-masked]

On Thu, 13 Jan 2005, Richard Clayton wrote:
>
> viz: It might be elitist, but I find that the directed tree graphs we
> have at the moment work remarkably well --- and I don't see why anyone
> wants to encourage random connections between multiple discussions :(

It probably arises from experience with web forums and blogs, where
references are maintained manually and may form arbitrary graphs.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL: SOUTH VEERING SOUTHWEST 5 TO 7, BECOMING CYCLONIC FOR A TIME IN
SOUTHEAST. OCCASIONAL RAIN. MODERATE OR GOOD.

Date: Fri, 14 Jan 2005 14:40:11 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Making group syntax do-not-generate
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
>>>> 1. Mail senders SHOULD NOT generate group syntax.
>>>
>>> maybe, maybe not.  groups are useful.
>>
>> They're not used...
>
> they are used, just not terribly often.  but the utility of a feature 
> is not directly related to the frequency of its use.

I agree with the latter sentence.

>> (Btw, I can think of at least three other reasons why they don't 
>> parse groups, so I don't think that presumption is warranted.)
>
> which are?

Some good reasons, some bad:

1. A judgment that it's better for the user to optimize the UI for the 
simple case, ie. no groups visible.

2. Some people choose to support only what a set of major MUAs 
generates. (Many people develop with limited time and money, and test 
all that they support. Sometimes that entails making choices noone 
likes.)

3. "It's ugly". You know, the RFC2047/2231 argument.

4. Some people test only with a corpus of real-world mail. Me, I have a 
corpus of 1.6 million messages where groups don't occur even once. In 
such a setting, it's easy for group-related bugs to slip through.

>>> surely it's harder to read and adhere to the combination of both 
>>> RFCs X and Y? particularly when they give conflicting advice?
>>
>> Specious comparison. People aren't really adhering to RFC X in the 
>> first place.
>
> but are we going to help the situation by giving them more to read?  
> or to put it another way, where is the barrier?

The primary barrier is IMO in the large number of deployed mail 
parsers/readers that haven't been tested with groups, and cannot be 
assumed to work.

> or maybe the barrier is in trying to get so many different 
> implementations of a complex protocol (which doesn't support enough 
> features as it is) to act consistently and compatibly with one 
> another?

Yes...

Arnt

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 08:05:22 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

>>> 1. Mail senders SHOULD NOT generate group syntax.
>>
>> maybe, maybe not.  groups are useful.
>
> They're not used...

they are used, just not terribly often.  but the utility of a feature 
is not directly related to the frequency of its use.

>> if we deprecate the group syntax we will wind up reinventing it.
>
> Perhaps whatever we then invent will be used.
>
> I don't know why groups aren't used. Perhaps it's because people have 
> little use for the concept, or perhaps there's some detail about their 
> syntax. In the latter case, a reinvention might succeed where groups 
> haven't.

many people don't know about them.  some MUAs make it difficult to 
support groups because they're trying to make it easy to input one 
address at a time.

(822 address field syntax may be a culprit here.  what with phrases, 
angle brackets, quoting rules, comments, etc. (and encoded-words which 
were added later) it's too baroque to expect ordinary people to type it 
in correctly.  and yet when MUAs try to insulate that syntax from the 
users, it robs senders of the power to express things like the _roles_ 
of recipients or why they're being sent the message.)

>> this proposal strikes me as one of the form: since people won't 
>> adhere to RFC X, presumably because they don't bother to read it 
>> fully, let's write RFC Y that tells them to do something else.
>
> More like "... let's write RFC Y telling other people that they 
> shouldn't expect anyone to adhere to RFC X".
>
> (Btw, I can think of at least three other reasons why they don't parse 
> groups, so I don't think that presumption is warranted.)

which are?  (or maybe I learned so long ago that address fields need to 
be lexically analyzed left-to-right and parsed right-to-left that I've 
almost forgotten that phrases and group syntax complicate parsing for 
those who haven't yet realized that.)

>> surely it's harder to read and adhere to the combination of both RFCs 
>> X and Y? particularly when they give conflicting advice?
>
> Specious comparison. People aren't really adhering to RFC X in the 
> first place.

but are we going to help the situation by giving them more to read?  or 
to put it another way, where is the barrier?  is it in reading and 
understanding the RFCs or in implementing them?  or maybe the barrier 
is in trying to get so many different implementations of a complex 
protocol (which doesn't support enough features as it is) to act 
consistently and compatibly with one another?

Date: Fri, 14 Jan 2005 13:37:43 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Re: Making group syntax do-not-generate
Cc: Keith Moore <[email-masked]>
Sender: [email-masked]

Keith Moore writes:
>> 1. Mail senders SHOULD NOT generate group syntax.
>
> maybe, maybe not.  groups are useful.

They're not used...

> if we deprecate the group syntax we will wind up reinventing it.

Perhaps whatever we then invent will be used.

I don't know why groups aren't used. Perhaps it's because people have 
little use for the concept, or perhaps there's some detail about their 
syntax. In the latter case, a reinvention might succeed where groups 
haven't.

> mail receivers should be able to parse groups anyway, since they want 
> to be able to read legacy messages.

I didn't say the opposite ;)

> this proposal strikes me as one of the form: since people won't adhere 
> to RFC X, presumably because they don't bother to read it fully, 
> let's write RFC Y that tells them to do something else.

More like "... let's write RFC Y telling other people that they 
shouldn't expect anyone to adhere to RFC X".

(Btw, I can think of at least three other reasons why they don't parse 
groups, so I don't think that presumption is warranted.)

> surely it's harder to read and adhere to the combination of both RFCs 
> X and Y? particularly when they give conflicting advice?

Specious comparison. People aren't really adhering to RFC X in the first place.

Arnt

Cc: [email-masked]
From: Keith Moore <[email-masked]>
Subject: Re: Making group syntax do-not-generate
Date: Fri, 14 Jan 2005 07:04:26 -0500
To: Arnt Gulbrandsen <[email-masked]>
Sender: [email-masked]

> 1. Mail senders SHOULD NOT generate group syntax.

maybe, maybe not.  groups are useful.  if we deprecate the group syntax 
we will wind up reinventing it.

in general it seems like internet mail and MUAs are suffering from too 
small a feature set - our tools aren't powerful enough to make email a 
really effective medium, especially given the volume at which many of 
us use it.  which is not to say that we should keep useless features, 
but groups are useful.  IMHO more people would use them if MUAs 
supported them better - not just parsing them but providing good UAs 
for them.

at least both Apple Mail and gmail are being actively maintained, so 
there's hope that they can be fixed soon.  (though if I could figure 
out how to get apple to listen to bug reports, I'd be giving them an 
earful...)

> 2. Mail receivers are urged to test that they can parse empty groups, 
> since several current senders generate that, e.g. "To: 
> unknown-recipients:;" and "To: ietf-announce:;".

mail receivers should be able to parse groups anyway, since they want 
to be able to read legacy messages.

this proposal strikes me as one of the form: since people won't adhere 
to RFC X, presumably because they don't bother to read it fully, let's 
write RFC Y that tells them to do something else.  surely it's harder 
to read and adhere to the combination of both RFCs X and Y?  
particularly when they give conflicting advice?

and as long as we're trying to fix mailers, why not fix them to have 
more utility?  unless we believe that it's the humans who can't deal 
with groups.  (but as far as I can tell, humans seem to understand the 
concept better than mailers).

Date: Fri, 14 Jan 2005 10:38:21 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: [email-masked]
Subject: Making group syntax do-not-generate
Sender: [email-masked]

Bart Schaefer writes:
> On Thu, 13 Jan 2005 09:47:24 -0500, Keith Moore <[email-masked]> wrote:
>>  furthermore, you don't seem willing to accept or act on negative 
>>  results, as evidenced by your continued insistence on using LIST: 
>>  despite having been informed that it confuses one widely used mail 
>>  user agent.
>
> It confuses Gmail, too.  Gmail displays the recipients (and addresses 
> the replies) as LIST, ietf-822
>
> I suppose as a well-behaved beta-tester I should report that as a bug.

Perhaps. But group syntax has been so little used in the past years that 
IMO it's better to kill the syntax than to try to support it. As a 
feature, group syntax simply isn't carrying its load.

IMO, the best option would be to issue a short RFC saying:

1. Mail senders SHOULD NOT generate group syntax.
2. Mail receivers are urged to test that they can parse empty groups, 
since several current senders generate that, e.g. "To: 
unknown-recipients:;" and "To: ietf-announce:;".

Comments?

Arnt

Date: Thu, 13 Jan 2005 21:44:02 -0800
From: Bart Schaefer <[email-masked]>
Reply-To: Bart Schaefer <[email-masked]>
To: [email-masked]
Subject: Re: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Sender: [email-masked]

On Thu, 13 Jan 2005 09:47:24 -0500, Keith Moore <[email-masked]> wrote:
> 
> furthermore, you don't seem willing to accept or act on negative results,
> as evidenced by your continued insistence on using LIST: despite having
> been informed that it confuses one widely used mail user agent.

It confuses Gmail, too.  Gmail displays the recipients (and addresses
the replies) as
LIST, ietf-822

I suppose as a well-behaved beta-tester I should report that as a bug.

Date: Fri, 14 Jan 2005 11:10:18 +1000
From: Laird Breyer <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: LIST: [email-masked];
Subject: Re: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Reply-To: [email-masked]
Mail-Followup-To: Charles Lindsey <[email-masked]>, LIST: [email-masked];
Sender: [email-masked]

On Jan 13 2005, Charles Lindsey wrote:
> 
> This message contains two Reference headers. Please report whether it is
> delivered intact and whether it causes any other problems, such as
> peculiar threading.

Not for me, I only see one Reference header. This is after
POP3/fetchmail retrieval, I don't know what goes on earlier in the
transport.
 
I also think Keith Moore is right about this list being an
unrepresentative sample for positive results, but the converse is also
true: the few negative results so far already indicate this won't work
reliably with current software.

-- 
Laird Breyer.

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: General considerations for new message field specifications
Date: Thu, 13 Jan 2005 19:27:48 -0500
Sender: [email-masked]

--Boundary-00=_FIx5B7WDux+4Y03
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I've thrown together an Internet Draft based on our discussion
a few months ago.  It should appear soon at the usual places.
A copy of the text is attached.

This mailing lists is as good a place as any for discussion;
private comments are of course welcome.

--Boundary-00=_FIx5B7WDux+4Y03
Content-Type: text/plain;
  charset="us-ascii";
  name="draft-lilly-field-specification-00.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename="draft-lilly-field-specification-00.txt"

Network Working Group                                           B. Lilly
Internet Draft                                              January 2005
Expires: July 14, 2005

      Implementer-friendly Specification of Message and MIME-Part
                   Header Fields and Field Components
                   draft-lilly-field-specification-00

Status of this Memo

   This document is an Internet-Draft and is subject to all provisions
   of section 3 of RFC 3667 [BCP78].  By submitting this Internet-Draft,
   the author represents that any applicable patent or other IPR claims
   of which he is aware have been or will be disclosed, and any of which
   he become aware will be disclosed, in accordance with RFC 3668.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   <http://www.ietf.org/ietf/1id-abstracts.txt>.

   The list of Internet-Draft Shadow Directories can be accessed at
   <http://www.ietf.org/shadow.html>.

Copyright Notice

   Copyright(C)The Internet Society (2005).

Abstract

   Implementation of generators and parsers of header fields requires
   certain information about those fields.  Interoperability is most
   likely when all such information is explicitly provided by the
   technical specification of the fields.  Lacking such explicit
   information, implementers may guess, and interoperability may suffer.
   This memo identifies information useful to implementers of header
   field generators and parsers.

1.  Introduction

   Internet messages consist of a message header and a body [STD11],
   [RFC2822].  MIME content begins with a MIME-part header [RFC2045],
   [RFC2046].  Message headers and MIME-part headers consist of fields.
   While the Message Format and MIME specifications define their
   respective overall formats and some specific fields, they also have
   provision for extension fields.  A number of extension fields have
   been specified, some more or less completely than others.  Incomplete
   or imprecise specification has led to interoperability problems as

Lilly                     Expires July 14, 2005                 [Page 1]

Internet Draft       Specification of Header Fields         January 2005

   implementers make assumptions in the absence of specifications.  This
   memo identifies items of potential interest to implementers and
   section 4 of this memo may serve as a checklist for specifications of
   extension fields and field components.

2.  Requirement Levels

   The key words "SHOULD", and "SHOULD NOT", in this document are to be
   interpreted as described in [BCP14].

3.  Scope

   This memo is intended to supplement various specifications,
   guidelines, and procedures for specification of header fields
   [STD11], [BCP9], [RFC2045], [RFC2046], [RFC2822], [BCP90].  It does
   not absolve authors of header field specifications from compliance
   with any provisions of those or other specifications, guidelines, and
   procedures.  It offers clarification and supplementary suggestions
   that will promote interoperability and may spare specification
   authors many questions regarding incomplete header field
   specifications.

4.  Specification Items

4.1.  Established Conventions

   A number of conventions exist for naming and specifying header
   fields.  It would be unwise to specify a field which conflicts with
   those conventions.

4.1.1.  Naming Conventions

   Several conventions have been established for naming of header
   fields.

4.1.1.1.  Resent- prefix

   Field names beginning with "Resent-" have particular semantics as
   given in [STD11] and [RFC2822].  If a Resent- version of a field is
   applicable, an author SHOULD say so explicitly, and SHOULD provide a
   comprehensive specification of any differences between the plain
   field and the Resent- version.

4.1.1.2.  Content- prefix

   Field names beginning with "Content-" are MIME extension fields
   [RFC2045].  This prefix SHOULD be used for all MIME extension fields
   and SHOULD NOT be used for fields which are not MIME extension
   fields.

4.2.  Common Specification Items

   Several items are specified for standard header fields; these items
   should also be specified for extension fields.

Lilly                     Expires July 14, 2005                 [Page 2]

Internet Draft       Specification of Header Fields         January 2005

4.2.1.  ABNF

   [STD11] is vague about where whitespace is permitted or required in
   header field syntax.  [RFC2822] addresses that issue by defining
   grammar productions such as FWS and CFWS.  Extension field ABNF
   SHOULD clearly specify where comments, line folding, and whitespace
   are prohibited and permitted, and SHOULD use the RFC 2822 grammar
   productions in ABNF for that purpose.

   All ABNF should be carefully checked for ambiguities and to ensure
   that all productions resolve to some combination of terminal
   productions provided by a normative reference.  [RFC2234] provides
   several productions that may be useful.  While use of suitable
   productions defined and in use is encouraged, specification authors
   are cautioned that some such productions have been amended by
   subsequently issued RFCs and/or by formal errata [Errata].

   It is sometimes necessary or desirable to define keywords as protocol
   elements in structured fields.  Protocol elements SHOULD be
   case-insensitive per the Internet Architecture [RFC1958].  Keywords
   are typically registered by IANA; a specification using registered
   keywords SHOULD include an IANA considerations section, and SHOULD
   indicate to readers of the specification precisely where IANA has set
   up the registry (authors will need to coordinate this with IANA prior
   to publication as an RFC).  In many cases, it will be desirable to
   make provision for extending the set of keywords; that may be done by
   specifying that the set may be extended by publication of an RFC, or
   a formal review and registration procedure may be specified
   (typically as a BCP RFC).

   Provision may be made for experimental or private-use keywords.
   These typically begin with a case-insensitive "x-" prefix.  Note that
   [BCP82] has specific considerations for use of experimental keywords.

   If some field content is to be considered human-readable text, there
   should be provision for specifying language in accordance with
   [BCP18].  Header fields typically use the mechanism specified in
   [RFC2047] as amended by [RFC2231] and [Errata] for that purpose.
   Note, however, that that mechanism applies only to three specific
   cases; unstructured fields, an RFC 822 "word" in an RFC 822 "phrase",
   and comments in structured fields.  Any internationalization
   considerations should be detailed in an Internationalization
   Considerations section of the specification as specified in [BCP18].

4.2.2.  Minimum and Maximum Instances per Header

   Some fields are mandatory, others are optional.  It may make sense to
   permit multiple instances of a field in a given header; in other
   cases at most a single instance is sensible.  RFC 2822 specifies a
   minimum and maximum count per header for each standard field in a
   message; specification authors SHOULD likewise specify minimum and
   maximum counts for extension fields.

Lilly                     Expires July 14, 2005                 [Page 3]

Internet Draft       Specification of Header Fields         January 2005

4.2.3.  Categorization

   RFC 2822 defines categories of header fields (e.g. trace fields,
   address fields).  Such categories have implications for processing
   and handling of fields.  A specification author SHOULD indicate any
   applicable categories.

4.3.  Semantics

   In addition to specifying syntax of a field, a specification document
   should indicate the semantics of each field.  Such semantics are
   comprised of several aspects:

4.3.1.  Producers, Modifiers, and Consumers

   Some fields are intended for end-to-end communication between author
   or sender and recipient; such fields should not be generated or
   altered by intermediaries in the transmission chain [Crocker04].
   Other fields comprise trace information which is added during
   transport.  Authors SHOULD clearly specify who may generate a field,
   who may modify it in transit, who should interpret such a field, and
   who is prohibited from interpreting or modifying the field.

4.3.2.  What's it All About?

   When introducing a new field or modifying an existing field, an
   author should present a clear description of what problem or
   situation is being addressed by the extension or change.

4.3.3.  Context

   The permitted types of headers in which the field may appear should
   be specified.  Some fields might only be appropriate in a message
   header, some might appear in MIME-part headers as well as message
   headers, still others might appear in specialized MIME media types.

4.4.  Overall Considerations

   Several factors should be considered regarding how a field interacts
   with the Internet at large, with the applications for which it is
   intended, and in interacting with other applications.

4.4.1.  Security

   Every specification is supposed to include a carefully-considered
   Security Considerations section [RFC2223].

4.4.2.  Backwards Compatibility

   There is a large deployed base of applications which use header
   fields.  Implementations that comprise that deployed base may change
   very slowly.  It is therefore critically important to consider the
   impact of a new or revised field or field component on that deployed
   base.  A new field, or extensions to the syntax of an existing field

Lilly                     Expires July 14, 2005                 [Page 4]

Internet Draft       Specification of Header Fields         January 2005

   or field component, might not be recognizable to deployed
   implementations.  Depending on the care with which the authors of an
   extension have considered such backwards compatibility, such an
   extension might, for example:

   a. Cause a deployed implementation to simply ignore the field in its
      entirety.  That is not a problem provided that it is a new field
      and that there is no assumption that such deployed implementations
      will do otherwise.

   b. Cause a deployed implementation to behave differently from how it
      would behave in the absence of the proposed change, in ways that
      are not intended by the proposal.  That is a failure of the
      proposal to remain backwards compatible with the deployed base of
      implementations.

   There are many subtleties and variations that may come into play.
   Authors SHOULD very carefully consider backwards compatibility when
   devising extensions, and SHOULD clearly describe all known
   compatibility issues.

4.4.3.  Compatibility With Legacy Content

   Content is sometimes archived for various reasons.  It is sometimes
   necessary or desirable to access archived content, with the semantics
   of that archived content unchanged.  It is therefore important that
   lack of presence of an extension field or field component should not
   be construed (by an extension specification) as conferring new
   semantics on a message or piece of MIME content which lacks that
   field or field component.

4.4.4.  Interaction With Established Mechanisms

   Header fields are handled specially by gateways under various
   circumstances.  Message fragmentation and reassembly [RFC2046] is one
   example.  If special treatment is required for a header field under
   such circumstances, it SHOULD be clearly specified by the author of
   the specification.  [RFC3798] is an example of how this might be
   handled (however, because that specification requires deployed RFC
   2046-conforming implementations to be modified, it is not strictly
   backwards compatible).

5.  Acknowledgments

   The author would like to acknowledge the helpful comments provided by
   members of the ietf-822 mailing list.  In particular, Peter Koch and
   Keith Moore have made useful comments.

6.  Security Considerations

   No new security considerations are addressed by this memo.  The memo
   reinforces the need for careful consideration of security issues.

Lilly                     Expires July 14, 2005                 [Page 5]

Internet Draft       Specification of Header Fields         January 2005

7.  Internationalization Considerations

   This memo does not directly have internationalization considerations,
   however it reminds specification authors of the need to consider
   internationalization of textual field components.

8.  IANA Considerations

   While no specific action is required of IANA in regard to this memo,
   it does note that some coordination between IANA and specification
   authors who do require IANA to set up registries is at least
   desirable, if not a necessity.  IANA should also closely coordinate
   with the RFC Editor so that registries are set up and properly
   referenced at the time of publication of an RFC which refers to such
   a registry.  IANA is also encouraged to work closely with authors and
   the RFC Editor to ensure that descriptions of registries maintained
   by IANA are accurate and meaningful.

Normative References

   [BCP9]      Bradner, S., "The Internet Standards Process -- Revision
               3", BCP 9, RFC 2026, October 1996.

   [BCP14]     Bradner, S., "Key words for use in RFCs to Indicate
               Requirement Levels", BCP 14, RFC 2119, March 1997.

   [BCP18]     Alvestrand, H., "IETF Policy on Character Sets and
               Languages", BCP 18, RFC 2277, January 1998.

   [BCP82]     Narten, T., "Assigning Experimental and Testing Numbers
               Considered Useful", BCP 82, RFC 3692, January 2004.

   [BCP90]     Klyne, G., Nottingham, M., and J. Mogul, "Registration
               Procedures for Message Header Fields", BCP 90, RFC 3864,
               September 2004.

   [Errata]    RFC-Editor errata page,
               http://www.rfc-editor.org/errata.html

   [RFC2045]   Freed, N. and N. Borenstein, "Multipurpose Internet Mail
               Extensions (MIME) Part One: Format of Internet Message
               Bodies", RFC 2045, November 1996.

   [RFC2046]   Freed, N. and N. Borenstein, "Multipurpose Internet Mail
               Extensions (MIME) Part Two: Media Types", RFC 2046,
               November 1996.

   [RFC2223]   Postel, J. and J. Reynolds, "Instructions to RFC
               Authors", RFC 2223, October 1997.

   [RFC2234]   Crocker, D. and P. Overell, "Augmented BNF for Syntax
               Specifications: ABNF", RFC 2234, November 1997.

Lilly                     Expires July 14, 2005                 [Page 6]

Internet Draft       Specification of Header Fields         January 2005

   [RFC2822]   Resnick, P., "Internet Message Format", RFC 2822, April
               2001.

   [STD11]     Crocker, D., "Standard for the format of ARPA Internet
               text messages", STD 11, RFC 822, August 1982.

Informative References

   [BCP78]     Bradner, S., "IETF Rights in Contributions", BCP 78,
               RFC 3667, February 2004.

   [BCP79]     Bradner, S., "Intellectual Property Rights in IETF
               Technology", BCP 79, RFC 3668, February 2004.

   [Crocker04] Crocker, D., "Internet Mail Architecture", Work in
               progress

   [RFC1958]   Carpenter, B., "Architectural Principles of the
               Internet", RFC 1958, June 1996.

   [RFC3798]   Hansen, T. and G. Vaudreuil, "Message Disposition
               Notification", RFC 3798, May 2004.

Author's Address

   Bruce Lilly

   Email: [email-masked]

Full Copyright Statement

   Copyright(C)The Internet Society (2005).  This document is subject to
   the rights, licenses and restrictions contained in BCP 78 [BCP78],
   and except as set forth therein, the author retains all his rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE REPRESENTS OR
   IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 [BCP78] and BCP 79 [BCP79].

Lilly                     Expires July 14, 2005                 [Page 7]

Internet Draft       Specification of Header Fields         January 2005

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   [email-masked].

Acknowledgment

   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Lilly                     Expires July 14, 2005                 [Page 8]


--Boundary-00=_FIx5B7WDux+4Y03--

Date: Thu, 13 Jan 2005 19:03:48 +0000
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
From: Richard Clayton <[email-masked]>
Subject: Re: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Sender: [email-masked]

>Return-path: <[email-masked]>
>To: LIST: [email-masked];
>Xref: clerew local.mime:4338
>Newsgroups: local.mime
>Path: clerew!chl
>From: "Charles Lindsey" <[email-masked]>
>Subject: Experiment with multiple Reference headers
> (was References with multiple precursore [sic])
>Message-ID: <[email-masked]>
>X-Newsreader: NN version 6.5.2 (NOV)
>References: <[email-masked]> <200501100920.17864.blilly@ero
>ls.com> <[email-masked]> <[email-masked]>
>Date: Thu, 13 Jan 2005 10:52:52 GMT
>Lines: 18
>Sender: [email-masked]
>Precedence: bulk
>List-Archive: <http://www.imc.org/ietf-822/mail-archive/>
>List-ID: <ietf-822.imc.org>
>List-Unsubscribe: <mailto:[email-masked]?body=unsubscribe>
>
>
>This message contains two Reference headers. 

not here it didn't

>Please report whether it is
>delivered intact 

exactly how might one know ?

I've snipped a lot of transport stuff already ... but I wouldn't like to
guess how much of what remains was added by that process -- or whether
other things have disappeared on the way

>and whether it causes any other problems, such as
>peculiar threading.

it always strikes me that the sort of people who combine two threads
into one and answer bits of both at random (and who therefore demand two
reference headers so that their valuable contributions can be correctly
placed into the responses) are exactly the sort of people I find have
contributed very little to the conversation :(

viz: It might be elitist, but I find that the directed tree graphs we
have at the moment work remarkably well --- and I don't see why anyone
wants to encourage random connections between multiple discussions :(

>For this first experiment, both the Reference headers are identical. If
>this causes no obvious problems, then I shall try further examples
>representing various DAGs.

also, I endorse Keith Moore's point about the likely value of the
experiments given the huge number of clients out there -- are you
expecting me to feed this through a few dozen of my Perl scripts as
well, to see what they do ? If they die("multiple References") then
should we just criticise my coding style ?

- -- 
richard @ highwayman . com                       "Nothing seems the same
                          Still you never see the change from day to day
                                And no-one notices the customs slip away"

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject: Re: Experiment with multiple Reference headers
Date: Thu, 13 Jan 2005 19:26:04 +0100
Organization: <URL:http://purl.net/xyzzy>
Lines: 17
Cc: Charles Lindsey <[email-masked]>
Sender: [email-masked]

Charles Lindsey wrote:
 
> This message contains two Reference headers.

Not "here" = on GMaNe,

> Please report whether it is delivered intact

It's not "intact", one of your References didn't make it.  
If GMaNe would be my gateway I'd be tempted to reject it
instead of removing any extraneous References.

                    Bye, Frank

P.S.: courtesy copy because I'm not subscribed and maybe
      my reply via GMaNe doesn't make it to the list.

Date: Thu, 13 Jan 2005 09:47:24 -0500
From: Keith Moore <[email-masked]>
To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked], LIST: [email-masked];
Subject: Re: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Sender: [email-masked]

> 
> This message contains two Reference headers. Please report whether it is
> delivered intact and whether it causes any other problems, such as
> peculiar threading.
> 
> For this first experiment, both the Reference headers are identical. If
> this causes no obvious problems, then I shall try further examples
> representing various DAGs.

such experiments can do nothing to demonstrate whether this works in the
Internet as a whole, as the recipients of this list are not even close to
being a representative sample of Internet users.

furthermore, you don't seem willing to accept or act on negative results,
as evidenced by your continued insistence on using LIST: despite having 
been informed that it confuses one widely used mail user agent.

if you're not going to heed negative results, and positive results are
meaningless, why should anyone bother to help you with these experiments?

To: LIST: [email-masked];
Xref: clerew local.mime:4337
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: References with multiple precursors (was Attempts at establishing harmful conventions)
Date: Thu, 13 Jan 2005 10:47:59 GMT
Lines: 135
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Tue January 11 2005 06:46, Charles Lindsey wrote:

>> So here is a suggestion to get around the problem in a way that will
>> probaby allow existing MUAs to do a reasonable job:
>> 
>> Simply allow there to be multiple occurrences of the References header in
>> each message - one for each path from the root to the tip in question.

>No. Expressly forbidden by RFC 2822 section 3.6.2, which specifies
>a maximum of one References field per header.

Yawn!

Has it not occurred to you that what I am suggesting would require an
update of RFC 2822, and of several other documents as well. The whole
purpose of this discussion list is to examine suggested enhancements to
the mail protocols to see whether they are worth pursuing. So it is hardly
sensible to say "you can't consider that change to the protocols because
it would change the protocols".
> 
>> Then an updated MUA can look at them all, and has full information
>> available to give the best possible presentation. Meanwhile, an unupdated
>> MUA will probably just take the first, or the last, or a randomly chosen
>> one, and will at least produce a plausible thread.

>That is not backwards compatible. We don't foist illegal content
>(in this case multiple References fields) on widely deployed
>implementations and shrug off the incompatibility with an off-hand
>remark hypothesizing that those implementations will "probably"
>cope in some way.

No, it is not necessarily backwards incompatible, for reasons which I
tried to explain, but which you refuse even to listen to.

It may or may not be the case that existing implementations would break in
disastrous ways if more than one References header was found in a message.
I rather expect they would not, for the simple reason that implementors
would have had to go out of their way to cause such nasties, and
implementors are not noted for doing unnecessary work.

It is indeed the case that no present MUA would generate such a thing, and
might even thwart a user who tried to create it manually, but that is not
the point.

Would any transport fail to deliver it? I doubt it, because I doubt any
transport even bothers to look at the References header. Would any storage
agent fail to store it? I doubt it, because no-one would want a storage
agent so illiberal as to drop malformed messages on the floor (there are
too many malformed messages around for that). It might truncate one of the
'superfluous' headers, but that is not a cause of serious harm.

Would any MUA fail to display the message altogether? No, for the same
reason. Would an MUA fail to display it in a sensible place in a thread?
Yes, that is indeed the problem that might arise. But my guess is that it
would either fail to include the message in any thread at all, or it would
thread it based upon examination of one of the References headers only,
and my reason for supposing that is that I cannot imagine any way of
writing an agent that was expecting to thread based on examination of one
header that would have any effect other than one of those. But the only
way to find out would be to try it on various existing agents, and see.

Yes, the behaviour of such an agent wouold be sub-optimal, but if you
introduce new functionality into a protocol, then you expect sub-optimal
behaviour from agents that have not been upgraded. The same would be true
if you introduced some brand new header to achieve the new functionality,
but that would not amount to backwards incompatibility. There is no gain
without pain.

Anyway, I shall now try to post messages with multiple Reference headers
to this list, so that we can gauge the extent of any possible problems.
Would anyone who completely fails to see them please report here.
Otherwise, please reply to them.

> 
>> And it is not actually necessary for the whole of each of the various 
>> paths to appear in the various References headers. Just the segment that 
>> differs from the others should suffice, though careful thought would have
>> to be given to defining exactly what the rule should be.

>Having failed to give careful thought, you still claim that deployed
>MUAs "will at least produce a plausible thread".  Incredible.

>You also failed to take note of the summary of conclusions of the
>earlier discussion; your scheme, in addition to failing to be
>backwards compatible:

>a. ignores that "expecting UAs to coordinate updates to multiple fields
>   is probably over-optimistic"; you fail to even mention how the
>   multiple References fields which you propose would be updated when
>   such a message is responded to.

I am making a suggestion for a method that may be worth looking at further,
and you complain that I have not described every last nuance of a new
protocol. It I had described a whole new protocol at that level, you would
have complained that it was premature to describe the gory details when
the general principle had not been established.

>b. ignores the fact that "several existing UAs botch References and
>   In-Reply-To fields, destroying their utility"; illegally adding
>   more instances of the field can only exacerbate that problem.

So you are saying that the current existence of broken implementations
should be a bar to all future protocol development?

>c. does not address repetition of message-ids and the bulk associated
>   with such repetition; indeed, it involves such repetition

Any solution will involve additional bulk (because there is additional
information to be conveyed). Exactly how much bulk is one of those gory
details which would have to be sorted out if this idea became a serious
endeavour.

>d. ignores the conclusion that "rather than overload additional
>   semantics on the References field, a new field or set of fields
>   should be developed"

No, the previous discussion reached no such conclusion. Indeed it reached
no consensus at all, except that there were difficulties with all methods
proposed up to that time. There may well be difficulties with what I now
suggest, but progress will never be made unless people are prepared to
evaluate such difficulties to see whether they are outweighed by the
advantages to be gained.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: LIST: [email-masked];
Xref: clerew local.mime:4338
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Experiment with multiple Reference headers (was References with multiple precursore [sic])
Date: Thu, 13 Jan 2005 10:52:52 GMT
Lines: 18
Sender: [email-masked]

This message contains two Reference headers. Please report whether it is
delivered intact and whether it causes any other problems, such as
peculiar threading.

For this first experiment, both the Reference headers are identical. If
this causes no obvious problems, then I shall try further examples
representing various DAGs.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: References with multiple precursore [sic] (was Attempts at establishing harmful conventions)
Date: Wed, 12 Jan 2005 09:09:05 -0500
Cc: "Charles Lindsey" <[email-masked]>
Sender: [email-masked]

On Tue January 11 2005 06:46, Charles Lindsey wrote:

> Currently, where a response has only one precursor, a discussion thread
> consists of a tree, and the References header consists of the path from
> the root of the tree to the message at the tip of the tree which contains
> that References header.
[...]
> If you want to extend that to responses to multiple precursors, then the
> tree becomes a DAG (Directed Acyclic Graph), and there are many possible
> paths from the root to each tip

Yes, as noted (more succinctly) by Adam M. Costello a year and a half
ago.

> So here is a suggestion to get around the problem in a way that will
> probaby allow existing MUAs to do a reasonable job:
> 
> Simply allow there to be multiple occurrences of the References header in
> each message - one for each path from the root to the tip in question.

No. Expressly forbidden by RFC 2822 section 3.6.2, which specifies
a maximum of one References field per header.
 
> Then an updated MUA can look at them all, and has full information
> available to give the best possible presentation. Meanwhile, an unupdated
> MUA will probably just take the first, or the last, or a randomly chosen
> one, and will at least produce a plausible thread.

That is not backwards compatible. We don't foist illegal content
(in this case multiple References fields) on widely deployed
implementations and shrug off the incompatibility with an off-hand
remark hypothesizing that those implementations will "probably"
cope in some way.
 
> And it is not actually necessary for the whole of each of the various
> paths to appear in the various References headers. Just the segment that
> differs from the others should suffice, though careful thought would have
> to be given to defining exactly what the rule should be.

Having failed to give careful thought, you still claim that deployed
MUAs "will at least produce a plausible thread".  Incredible.

You also failed to take note of the summary of conclusions of the
earlier discussion; your scheme, in addition to failing to be
backwards compatible:

a. ignores that "expecting UAs to coordinate updates to multiple fields
   is probably over-optimistic"; you fail to even mention how the
   multiple References fields which you propose would be updated when
   such a message is responded to.

b. ignores the fact that "several existing UAs botch References and
   In-Reply-To fields, destroying their utility"; illegally adding
   more instances of the field can only exacerbate that problem.

c. does not address repetition of message-ids and the bulk associated
   with such repetition; indeed, it involves such repetition

d. ignores the conclusion that "rather than overload additional
   semantics on the References field, a new field or set of fields
   should be developed"

To: LIST: [email-masked];
Xref: clerew local.mime:4335
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: References with multiple precursore (was Attempts at establishing harmful conventions)
Date: Tue, 11 Jan 2005 11:46:02 GMT
Lines: 71
Sender: [email-masked]

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>On Thu December 2 2004 18:58, Steve Dorner wrote:
>>  
>> At 8:27 PM +0200 12/2/04, Kai Henningsen wrote:
>> >  > For that matter, how many permit a user to select multiple
>> >>  messages to compose a response citing all of the selected
>> >>  messages (including appropriate setting of In-Reply-To and
>> >>  References fields)?
>> >
>> >Hopefully few, as there *is no* appropriate setting for those fields in 
>> >that situation.
>> 
>> We actually hashed out a way that seemed right enough on this list a 
>> year or so ago.

>There was a discussion in June 2003, however if it can be
>said to have reached any conclusions, I believe those
>conclusions include:

[snipped list of problems]

>So far, I haven't seen any detailed proposal to address the
>issue in accordance with those conclusions (indeed, several
>points are somewhat contradictory).

Yes, that is a reasonable summary of the problems that we found. Here is
another suggestion:

Currently, where a response has only one precursor, a discussion thread
consists of a tree, and the References header consists of the path from
the root of the tree to the message at the tip of the tree which contains
that References header.

Sometimes the path may get pruned, because its length has become
excessive. And sometimes implementations botch it in various ways, or even
omit it entirely. But by and large it is well understood, and MUAs manage
to display threads more or less OK.

If you want to extend that to responses to multiple precursors, then the
tree becomes a DAG (Directed Acyclic Graph), and there are many possible
paths from the root to each tip, and there is no way for the References
header to represent that situation. Moreover, inventing new syntax for the
References header leads to compatibility problems.

So here is a suggestion to get around the problem in a way that will
probaby allow existing MUAs to do a reasonable job:

Simply allow there to be multiple occurrences of the References header in
each message - one for each path from the root to the tip in question.

Then an updated MUA can look at them all, and has full information
available to give the best possible presentation. Meanwhile, an unupdated
MUA will probably just take the first, or the last, or a randomly chosen
one, and will at least produce a plausible thread.

And it is not actually necessary for the whole of each of the various
paths to appear in the various References headers. Just the segment that
differs from the others should suffice, though careful thought would have
to be given to defining exactly what the rule should be.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

From: Bruce Lilly <[email-masked]>
Reply-To: ietf-822 <[email-masked]>
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Attempts at establishing harmful conventions
Date: Mon, 10 Jan 2005 09:20:17 -0500
Cc: Steve Dorner <[email-masked]>
Sender: [email-masked]

On Thu December 2 2004 18:58, Steve Dorner wrote:
> Re: Attempts at establishing harmful conventions
>  Date: 2004-12-02 18:58
>  From: Steve Dorner <[email-masked]>
>  To: Kai Henningsen <[email-masked]>
>  CC: [email-masked]
>  
> At 8:27 PM +0200 12/2/04, Kai Henningsen wrote:
> >  > For that matter, how many permit a user to select multiple
> >>  messages to compose a response citing all of the selected
> >>  messages (including appropriate setting of In-Reply-To and
> >>  References fields)?
> >
> >Hopefully few, as there *is no* appropriate setting for those fields in 
> >that situation.
> 
> We actually hashed out a way that seemed right enough on this list a 
> year or so ago.

There was a discussion in June 2003, however if it can be
said to have reached any conclusions, I believe those
conclusions include:
1. imposing new syntax on the References field is not
   backwards compatible
2. expecting UAs to coordinate updates to multiple fields
   is probably over-optimistic
3. several existing UAs botch References and In-Reply-To
   fields, destroying their utility
4. repeating message-ids may make a field "too long" (there
   is of course no limit to the length of a logical (continued)
   field, however a msg-id doesn't look like "1", it looks
   more like
   "<[email-masked]>",
   (that's a short example -- the domain name on the RHS can be
   up to 255 octets, and the local-part can also be lengthy)
   and one wouldn't want to have to repeat that)
5. rather than overload additional semantics on the References
   field, a new field or set of fields should be developed

So far, I haven't seen any detailed proposal to address the
issue in accordance with those conclusions (indeed, several
points are somewhat contradictory).

From: Hg Doreswamy-A21177 <[email-masked]>
To: [email-masked]
Subject: vMessage 1.1 Specification  needed
Date: Thu, 6 Jan 2005 17:35:56 +0530 
Sender: [email-masked]

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01C4F3E8.13AB5CBA
Content-Type: text/plain

Hello,
    Can I know the place where I can get the vMessage 1.1 specification
Thnx
Dore

------_=_NextPart_001_01C4F3E8.13AB5CBA
Content-Type: text/html
Content-Transfer-Encoding: base64

------_=_NextPart_001_01C4F3E8.13AB5CBA--

To: [email-masked]
From: wayne <[email-masked]>
Subject: SPF I-D for review: draft-schlitt-spf-classic-00.txt
Date: Mon, 03 Jan 2005 20:28:38 -0600
Lines: 45
Cancel-Lock: sha1:xTNqCNiho489iqFb93I3YSlrs6k=
Sender: [email-masked]

fyi;

There is a new I-D for the SPF email anti-forgery system available for
review.  This draft tries to document the current practices of the
~1,000,000[1] published SPF records and ~10,000[1] deployed SPF
systems that are checking 20-100million emails per day.

See: http://www.ietf.org/internet-drafts/draft-schlitt-spf-classic-00.txt

Discussions about this, and previous, drafts have been taking place on
the [email-masked] mailing list.  To subscribe or read
the archives, see http://spf.pobox.com/mailinglist.html

While I have been reading this mailing list for many months and will
note any comments posted here, sending email to the spf-discuss list
or to me personally is probably better than flooding this list.

I realize that the whole subject of SPF (and similar systems) has a
certain amount of controversy to it, but for the purposes of this
draft, I am very reluctant to try debate these issues.  The goal is to
document a de-facto standard.  Debates about better techniques, why
SPF is evil, etc. are probably best discussed on things like the IRTF
ASRG list, SPAM-L, the NANAE newsgroup, or on spf-discuss on a
separate thread/subject.

While SPF mostly deals with the [2]821 SMTP level, there are some
[2]822 issues, in particular, the Received-SPF header.  To be quite
honest, this header has kind of been neglected and could use more
refinement than most other parts of the draft.  One thing I just
tracked down (after this I-D was already submitted) was the RFC on
having the IANA register email headers.  I need to add stuff for this
in the IANA Considerations section.

-wayne

[1] These numbers are estimates, but pretty solid ones.  I have data
to back them up and they are almost certainly underestimating the real
numbers.